Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Authors: Hoigi Seo
Wongi Jeong
Jae-sun Seo
Se Young Chun
Venue: ICML 2025 poster (2025)
Keywords: Generative model
Efficiency
Pruning
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: In this paper, we target to reduce the size of text encoder in T2I diffusion mode with Skip and Re-use.

Abstract: Large-scale text encoders in text-to-image (T2I) diffusion models have demonstrated exceptional performance in generating high-quality images from textual prompts. Unlike denoising modules that rely on multiple iterative steps, text encoders require only a single forward pass to produce text embeddings. However, despite their minimal contribution to total inference time and floating-point operations (FLOPs), text encoders demand significantly higher memory usage, up to eight times more than denoising modules. To address this inefficiency, we propose Skip and Re-use layers (Skrr), a simple yet effective pruning strategy specifically designed for text encoders in T2I diffusion models. Skrr exploits the inherent redundancy in transformer blocks by selectively skipping or reusing certain layers in a manner tailored for T2I tasks, thereby reducing memory consumption without compromising performance. Extensive experiments demonstrate that Skrr maintains image quality comparable to the original model even under high sparsity levels, outperforming existing blockwise pruning methods. Furthermore, Skrr achieves state-of-the-art memory efficiency while preserving performance across multiple evaluation metrics, including the FID, CLIP, DreamSim, and GenEval scores.

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-lmage Generation
Hoigi Seo
Wongi Jeong"
Jae-sun Seo
Se Young Chun 13
Abstract
2.8 TRIS
Percnage (%)
Percentage P%
Varitional Autoencader
Large-scale text encoders in text-to-image (T2I) diffusion models have demonstrated exceptional performance in generating high-quality images from textual prompts. Unlike denoising modules that rely on multiple iterative steps, text encoders require only a single forward pass to produce text embeddings. However, despite their minimal con- tribution to total inference time and floating-por poini operations (FLOPs), text encoders demand signif- icantly higher memory usage, up eight times more than denoising modules. To address this inefficiency, we propose Skip and Re-use lay- ers (Skrr), simple yet effective pruning strat- egy specifically designed for text encoders in T21 diffusion models. Skrr exploits the inherent re- dundancy in transformer blocks by selectively skipping reusing certain layers in manner tailored for T2I tasks, thereby reducing mem- ory consumption without compromising perfor- mance. Extensive experiments demonstrate that Skrr maintains image quality comparable to the original model even under high sparsity levels, outperforming existing blockwise pruning meth- ods. Furthermore, Skrr achieves state-of-the-art memory efficiency while preserving performance across multiple evaluation metrics, including the FID, CLIP, DreamSim, and GenEval scores.
62.2
Text Encurder 3 (TS XXL)
Text Encuoder 2 (CLIP-G)
Varistional Autoencader
(CLIP-L)
96.7
X5-XXL)
Denoising Transformer
Text Enuseder 2 (CLIP-G)
Encoeder (CLIP-L)
Danoising Transformet
(a) FLOPs ratio.
(b) Parameter ratio.
Figure I. (a) FLOPs distribution during image generation Stable Diffusion (SD3) (Esser et al., 2024). (b) Parameter distribution across modules in SD3. The text encoders contributes less than 0.5% to the overall FLOPs but account for 70% of the total model parameters. For VAE, only the decoder was considered.
et al., 2023; Cao al., 2023; Kawar al., 2023), video generation (Liu et al., 2024; Polyak et al., 2024), and 3D creation (Poole et al., 2022; Cao et al., 2023; Seo et al., 2023; Wang et al., 2024c). With modern architecture and large-scale text encoders, they produce high-quality images that closely match text prompts. Despite these successes, they require significant computational resources, especially memory, making deployment and scalability challenging.
To address these issues, research suggests enhancing the ef- ficiency of the T2I diffusion model through strategies such as knowledge distillation (KD) (Castells et al., 2024b; Li et al., 2024b; Song et al., 2024b; Kim et al., 2024; Zhao et al., 2024), which transfers knowledge from larger models to smaller ones; pruning (Castells al., 2024a; Ganjdanesh et al., 2024; Lee et al., 2024; Wang al., 2024b), which eliminates superfluous weights; and quantization (Li et al., 2023; He et al., 2024; Ryu et al., 2025), which reduces preci- sion by utilizing fewer bits. While effective, these methods target the denoising module. As shown in Fig. 1, the text encoders account for over 70% of the total parameters, only 0.5% of floating- poiit operations (FLOPs), causing dis- proportionate memory usage. Despite this imbalance, efforts to reduce the text encoder size have been limited.
1. Introduction
Diffusion generative models excel at tasks such as text-to- image (T2I) synthesis (Rombach et al., 2022; Podell et al., 2023; Esser et al., 2024; Chen et al., 2024), editing (Brooks
'Equal contribution 'Dept. of Electrical and Computer Engi- neering, Scoul National University, Republic of Korea "School of Electrical and Computer Engineering, Cornell Tech, USA 'INMC IPAL, Scoul National University, Republic Korca, Correspon- dence to: Se Young Chun <sychun arr>
Large language models (LLMs) also face similar challenges, where model sizes result significant computational and memory overhead, hindering their practical use. To address this problem, studies such as KD (Hsieh al., 2023; Huang
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
al., 2023a; Ko al., 2024), quantization (Xiao et al., 2023; Ashkboos et al., 2024b; Lin et al., 2024), and pruning (Sun et al., 2023; Ashkboos et al., 2024a; Men et al., 2024; Song et al., 2024a; Yang et al., 2024; Zhang et al., 2024) have been proposed. While KD reduces the size of the model, it requires costly training. Quantization reduces memory us- age by reducing precision, but it requires specific hardware support. In contrast, pruning offers parameter reduction with minimal performance loss, making it an efficient solution.
for the text encoder in T2I diffusion models.
We present Skip, a pruning approach for lightweight T2I diffusion models, and Re-use, method to restore
T2I performance by leveraging the remaining layers, supported by theoretical analysis.
Skır achieves ofate-of-the-ar blockwise pruning for T21 synthesis, improving GenEval scores by up to 20.4% at high sparsity over 40%.
Among pruning techniques, structured pruning has been ac- tively studied to remove rows and columns (van der Ouderaa et al., 2023; Ashkboos et al., 2024a) of the model weights, or entire layers or blocks (Gromov et al., 2024; Men et al., 2024; Yang et al., 2024; Zhang et al., 2024) of the model to reduce the size of the model and improve the inference speed. However, these methods are designed for autore- gressive LLMs and face challenges when applied to T21 diffusion models, limiting their effectiveness in this context.
2. Related Works
2.1. Efficient diffusion model
As diffusion generative models scale, T2I synthesis has achieved impressive results in generating high-fidelity, text- aligned images. However, this advancement comes with sig- nificant computational and memory overhead. Previous re- search has primarily focused on optimizing the efficiency of the denoising module through methods such as knowledge distillation (Li et al., 2024b; Song et al., 2024b; Castells et al., 2024b; Zhao al., 2024; Kim et al., 2024), pruning model weights (Fang et al., 2023; Ganjdanesh al., 2024; Wang et al., 2024b; Castells et al., 2024a; Lee et al., 2024), and quantization of weights to lower precision bits (Li et al., 2023; He et al., 2024; Li et al., 2024a; Wang et al., 2024a; Ryu et al., 2025). While these approaches effectively reduce the computational and memory costs of the pipeline, they overlook the substantial memory burden imposed by the text encoder, which remains largely underexplored. To address this gap, we propose targeted pruning strategy for the text encoder of a T2I diffusion model, which allows more mem- ory efficient T2I synthesis with comparable performance.
We propose Skip and reuse the layers (Skrr), a blockwise pruming technique for T2I diffusion models. Skrr effectively reduces text encoder size, alleviating memory overhead while preserving image quality and text alignment. Skrr involves two primary stages: the Skip to detect layers for pruning, followed by the Re-use to recycle the remaining layers to mitigate performance degradation. To the best of our knowledge, this is the first work to tackle the challenge of constructing a lightweight text encoder for T2I tasks.
In the Skip phase, sub-blocks of the text encoder transformer are pruned using T21 diffusion-tailored discrepancy metric to align dense and pruned models. Prior methods greed- ily remove blocks to reduce computational costs, but often overlook block interactions, leading to suboptimal prun- ing. To mitigate this, we propose a beam search (Freitag & Al-Onaizan, 2017)-based approach that explores multiple pruming paths simultaneously, balancing the performance of exhaustive and efficiency of greedy strategies. In the Re-use phase, we assess the discrepancy from reusing adjacent un- skipped blocks to identify those that can be effectively reuti- lized. Additionally, we provide theoretical support showing that Re-use can enhance performance beyond mere skipping. To improve discrepancy measurement in both phases, we employ a projection module identical to the one used for conditioning text embeddings in the denoising module.
2.2. Blockwise pruning for LLMs
Although LLMs show promising performance in various tasks, their size often limits practical deployment. To address this problem, model compression techniques have been de- veloped, with pruning emerging as a promising solution due to its ability to reduce the parameter count with minimal re- training. Notably, blockwise pruning of transformers (Men et al., 2024; Yang et al., 2024; Zhang et al., 2024) effectively reduces parameters while preserving performance.
ShortGPT (Men et al., 2024) proposed a method to prune blocks to the desired sparsity by removing them one by one and assigning Block Influence (BI) score based on changes in cosine similarity and pruning those with lower BI first. LaCo (Yang et al., 2024) introduced a strategy to reduce the size of LLMs by merging the weights of adjacent transformer blocks, effectively compressing the model. Fin- erCut (Zhang et al., 2024) refined this approach by pruning sub-blocks, composed of Multi-Head Attention (MHA) and Fred-Forward Network (FFN) layers with normalization, in a fine-grained manner. It sequentially pruned sub-blocks
We conducted comprehensive experiments across various metrics, sparsity levels, and diffusion models to thoroughly evaluate the T2I performance of compressed text encoders. The results indicate that Skrr surpasses current autoregres- sive LLM- M-MMgaggteed pruning techniques on image fidelity and text-image alignment in high sparsity (> 40%).
Our contributions can be summarized as follows.
We propose Skrr, an effective layer pruning method
2

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texa-to-Image Generation
Trandoemer blook
Socnser
04
(a) Original image. (b) 7th 22th removed. (c) 3r 5th removed.
Tranefomer Block Index
Transfommr Block Indox
(a) Hidden states similarity.
(b) Fixed input similarity.
Figure 3. (a) An image created by the PixArt-E. dense en- coder using the prompt "A car made out of vegetables." with |I |a 0.03. For image (b), the 7 and 22th sub-blocks are excluded, resulting in Metriei = 0.15, Metrics = 0.002, and f lla = 0.19. Image (c) generated by removing 5th sub-blocks, producing Metric; = 0.11, Metric, 0.04, and Ilfe 3.34. Despite Metric, being higher in (c). the large Iife ll value compared to (b) lcads abnormal image. Notably, Metricz more accurately indicates differences in image quality.
Figure 2. (a) The cosine similarity of hidden states in T5 trans- former blocks demonstrates progressive variations. The result indi- cates specific layers could be omitted without serious performance degradation. (b) The cosine similarity of block outputs using fixed imputs. The similarity map reveals redundant role across blocks, suggesting that certain blocks could be replaced by adjacent blocks.
3 and
while partially considering interactions between blocks, re- ducing the performance gap with the dense model.
blocks. This strong similarity underscores the redundancy in the model and confirms the potential to prune blocks without significantly compromising performance.
Despite these advances, encoder compression in diffu- sion models remains underexplored. To address this discrep- ancy, we propose Skrr, blockwise text encoder pruning ap- proach tailored for T2I diffusion models. We evaluated Skrr against existing LLM--assd pruning techniques, demon- strating its effectiveness in reducing the size of the model while maintaining the T2I performance of its dense model.
Discrepancy metric. For effective block removal, it is crucial to evaluate impact of pruning each block on output. A strong metric must be defined to measure text embedding changes of post-removal. Given that text embeddings affect image quality in text-to-image models, choosing the right metric is vital. Current T2I diffusion models predominantly employ transformer frameworks to synthesize images from input noise and text embeddings. However, text embeddings from the output of the text encoder are not used as is. They undergo alignment through single linear layer or multi- layer perceptron (MLP), which is represented as:
3. Method
Skrr is built around two main parts: Skip identifies layers to prune, and Re-use selects layers to reuse from the re- mained ones. During the Skip phase, each multi-head at- tention (MHA) and feed-forward network (FFN) sub-block individually evaluated for its importance using T2I ieffusion-tailored metric. The sub-blocks are then ranked based on their significance. To optimize the pruning pro- cess, blocks with low importance are removed sequentially while exploring multiple possible combinations using a beam search-basee algorithm. The Re-use phase evaluates each layer to reuse a layer based on the metric leveraged in Skip phase the original output, ensuring that important information is conserved, thus reducing performance loss. The overall framework of Skrr is depicted in Fig. 4.
f = proj(E((e) frex..) ); Odcnoise.).
(I)
where is the input prompt, E(-; Owea.) is the text encoder parameterized with 0 proj(:; fdeeoise.) is the projection layer in the denoising module for condition vector from the text encoder. Using the features extracted in this manner, the importance of a block can evaluated by comparing the similarity between the feature faeus of the dense model and the feature f Saip of the model that skips (prunes) block. A commonly used metric in prior studies (Men et al., 2024; Yang et al., 2024; Zhang et al., 2024) for measuring discrep- ancy is cosine similarity. which is formulated as:
3.1. Skip Algorithm
Feasibility of skipping blocks. Prior work (Men et al., 2024; Yang et al., 2024; Zhang et al., 2024) shows trans- former blocks can be pruned based output similarity in LLMs. We extend this analysis to text encoders in diffusion models, especially T5-XXL (Raffel et al., 2020), widely used in T2I models, as shown in Fig. 2a. We observed a high degree of similarity between the hidden states of adjacent
Sdense Sskip Metric (faense fakap) [l weweellllfaaaaeeee lassppplll
(2)
Another metric worth considering is the mean- equared error (MSE) between two vectors, which differs from angular metrics by accounting for both the direction (angle) and the
3

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Skip algorithm (Algorithm 1)
Re-use algorithm (Algorithm 2)
Original Text Encoder
Original Text Encoder
N2
VIN
VIn
vIn
NM
Kid
NE
Ki
22
3
Text embod
Calib, data
Proj.
Skipped Text Encoder
Disc.
Disc
N.23
VIEN
VII
VIEV
Nad
VIJI
FEN
Search for every
Search every skipped block
Skipped Text Encoder
Re-used Text Encoder
N20
VIN
VIN
View
122
VIN
3
(a) Illustration Skip phase.
(b) Illustration Re-use phase.
Figure 4. The visualization of overall framework of Skrr. (a) shows the Skip phase, which repeatedly assesses each sub-block by determining the output discrepancy (Disc.) between the dense and skipped models using a calibration dataset (Calib. data). To account for block interactions, keeps the top & options with the smallest discrepancies and beam search for refined selection. (b) presents the Re-use phase, evaluating if recycling remaining block instead of skipped sub-blocks results in smaller output discrepancy. If so, hidden states fed back into the chosen layers. This two-phase approach efficiently reduces model size with minimal T2I performance loss.
magnitude of the vectors. The MSE is formulated as:
leading abnormal images. As illustrated in Fig. 3, prun- ing two blocks significantly increases || causing abnormalities shown in Fig. 3c. Thus, discrepancies in soo must be considered. Notably, in Fig. 3c, Metric; failed to assess image quality, while Metricz reported smaller values for Fig. 3b, confirming its reliability and effectiveness.
Metricz(Jacnes fakip) - E (áées Sákapp
where d is the dimension of the output feature vectors faeus and faip- and represents the i-th component of vector f. Metric, considers only the angle between vectors, while Metrica integrates both the angle and magnitude, offering a more comprehensive evaluation of output discrepancies between dense and skipped models (Zhang et al., 2024). Therefore, chose Metrica assess the discrepancies.
Beam search-based strategy. Most blockwise pruning methods perform (1) rank the importance of the block per layer and pruning in order (Men et al., 2024), or (2) perform sequential pruning while re-evaluating blocks (Yang et al., 2024; Zhang al., 2024). The first method is efficient, but may ignore block interactions and lead to suboptimal re- sults. The second method acknowledges interactions among blocks, yet its greedy approach can still lead to suboptimal results. To address these issues, we propose a novel method akin to beam search, which evaluates multiple pruning paths concurrently better account for block interactions, achiev- ing more effective pruning without degrading performance.
Null condition discrepancy. In diffusion generative mod- els, guidance is essential to produce high-quality images. A prevalent approach is classifier-fre frre guidance (CFG) (Ho & Salimans, 2021), )), which improves conditional synthesis by utilizing unconditional scores. This technique derives an unconditional score from null condition, extrapolates it with the conditional score, and is formulated as follows:
E( =) = + w)e fo) we(e,, f(x,,
Algorithm. To synthesize the proposed approach, we in- troduce the Skip algorithm in Algorithm 1. Before delv- define two key discrepancy met-
where e(.,.) the denoising score network, H, noisy sample at timestep fe denotes the condition vector from Eq. (1), fa represents the null condition vector, and w is the guidance scale. While guidance improves image quality, excessive scaling may cause over-saturation or artifacts. We observed that pruning certain text encoder blocks ampli- fies its norm by over x00x compared to the dense version,
ing into the specifics, rics: Dj.- derived from Metric2, and D representing Metric, with null inputs. The algorithm is inspired by beam search (Freitag & Al-Onaizan, 2017), iterating over each unskipped sub-block while maintaining the beams with the smallest sum of D), and Ds-- This process is re-

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Algorithm 1 Skip Algorithm
Algorithm 2 Re-use Algorithm
Require: Calibration dataset C, dense model M, null imput Cz. number of layers L. beam size k
Require: Calibration dataset C, dense model M, null input
Cg. skip indices list S, re-use indices dictionary R
Ensure: Skip index list S*
Ensure: Re-use indices dictionary R
1: S
1: R 4 @.M +
Prune M, M S)
2: B+ [(, S
2: DM t- -etDiscrepanccy (M, M, C,c)))
3: while (D) E B such that (S] < do
for each E S do
4: Bnew +0
4: I 4 max < $ S) rt min[j > 1i $11$5) 5
5: for each (D,S) B do
5: M + Update(MM. R (i:
6: for each layer index i & S do
M, f Updato R fi:r))
7:
S' + Append(S, i)
DM + GetDiscrepancy (M, MM,,,C,es)
8:
M + Prune(M,S')
8:
DI + GetDiscrepancy(M MM, c2)
9:
D' < GetDissrepancy M,,
9:
D. < BeuDiserepancy(MMM.. C,
10:
Bnew f Bne U ((D', 8")}
10:
Dr DM A Di < D, then
11:
R+ R RU:
11:
end for
12:
else if D, < DM A D. < Di then
12:
end for
R + (-:U(t 14: end if
13:
13:
Update Brew to smallest & candidates with D
14:
(D", (D*,S*)+= + ss) D' in B
15: M Update(M,, R.
15:
end while
16:
end for
16: return S"
17: return R
Function Definitions:
1: GetDiscrepaney(M (MM, M,
Lemma 3.1 (Error bound of two transformers). Let M (x, 0) + Rd be an T--block transformer with input x Rd and parameter set 0 (01....,02) OL). defined as:
2: Dy. < M(C,
3: Diz +- MSB(M((ss)
4: return Di++Dr. 1
M = M=((Ft + +)+ (Fr.1 + (Fi o (5)
peated from the & beams, iteratively updating them to ensure smaller D. Upon traversing all blocks, the algorithm pro- duces list of Skip indices S*, effectively capturing inter- block interactions. Finally, the blocks are pruned sequen- tially according the S to achieve the desired sparsity.
where F; (25,8,) b.) Rdd is the i-th block with parameters
O2, and 2i E Rd. Assume that F; is Li-Lipschitz in z; and Mi-Lipschitz in 0,. Then, for any two parameter sets 0 =
(0, (...... ..)....)))..... and 0 (0...... 0r), the following holds:
[|M{;;)) M(x;;)) 0)1l
3.2. Re-use Algorithm
(6)
Σ II Lx) M:l10: -0.1 0.1l== =- k=it1
Feasibility of reusing blocks. Despite extensive research on methods such as recurrent networks (Sherstinsky, 2020; Gu & Dao, 2023) which loop the output of the network back into the input for efficiency, the practice of reusing specific layers in neural networks by reintegrating hidden states into internal layers remains understudied. We conducted a feasi- bility study to investigate whether the internal components of the T21 diffusion text encoder serve analogous functions (see Fig. 2b). We randomly sampled tokens from the embed- ding, passed them through each transformer block as fixed input, and measured the similarity of their output. The re- sults indicate significant similarity between adjacent blocks, suggesting that performance can be restored by reintroduc- ing non-omitted layers into adjacent skipped layers. We also verified the existence of condition for Re-use that achieves a tighter error bound compared to Skip alone. The existence is formalized in Theorem 3.2, which theoretically demon- strates the advantages of incorporating the Re-use phase. To establish this result, we first introduce the following lemma.
The proof for Lemma 3.1 is in the Appendix Sec. A.I. With the lemma, we can prove the following theorem.
Theorem 3.2 (Tighter error bound of Re-use). Under the assumptions of Lemma 3.1. let 6 be the parameters of the reused F. Define Ussip as the error bound for the com- pressed model with Skip alone and Usiip. we as the error bound for the compressed model with Skip and Re-use. If |00 - 8:1l < [10,1], then the following holds:
Uskip, Re-use < Ustip-
This theorem establishes the theoretical feasibility of Re- use by showing a existence of condition under which the error bound becomes tight with its application. The proof for Theorem 3.2 is shown in the Appendix Sec. A.2.
5

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Table Quantitative comparisons of Skrr with baselines. We compared Skmr with the haselines of ShortGPT, LaCo, and FinerCut under three different sparsity scenarios on PixArt-22. The results show that SkIr reliably maintains image fidelity and performs comparable to the dense model across all given sparsity levels. Unlike ShortGPT and LaCo, FinerCut and Skır use sub-block pruning. hindering direct sparsity level alignment. Sparsity levels were matched as closely as possible for fair evaluation, reflecting how much the compressed model's parameters differ from the dense encoder. (t) / denotes that higher/ lower metric is favorable.)
Sparsity
GenEval f
Method
FID L
CLIP
DreamSim 1
(%)
Single
Two
Count.
Colors
Pos.
Color attr.
Overall
Dense
0.0
22.89
0.314
1.0
0.988
0.616
0.475
0.795
0.108
0.255
0.539
24.3
24.96
0.309
0.753
0.944
0.381
0.431
0.715
0.033
0.083
0.431
ShorGGT
32.4
27.28
0.294
0.651
0.834
0.197
0.291
0,537
0.048
0.038
0,324
40.5
55.26
0.215
0.357
0.306
0.025
0.090
0.100
0.0
0.0
0.087
24.3
19.45
0.311
0.726
0,909
0.336
0.394
0,713
0.065
0.128
0.424
LaCo
32.4
24.70
0.303
0.677
0.781
0.227
0.250
0.606
0.043
0.040
0.325
40,5
21.60
0.291
0.620
0.784
0.162
0.150
0.489
0.030
0.033
0.275
26.3
20.66
0.313
0.798
0.947
0.465
0.394
0.737
0.103
0.105
0.458
FinerCut
32.2
20.49
0.313
0.771
0.903
0.409
0.344
0.697
0.078
0.128
0,426
41.7
20.36
0.308
0.731
0.841
0.306
0.306
0,628
0.050
0.073
0.367
27.0
20.15
0.315
0,800
0.956
0.434
0,425
0.763
0.095
0.145
0.471
Skrr (Ours)
32.4
20.19
0.313
0.775
0.928
0.397
0.413
0.774
0.100
0.118
0.455
41.9
19.93
0.312
0.741
0.913
0.410
0.450
0.755
0.055
0.068
0.442
Algorithm. The Re-use algorithm provided in Algo- rithm enhances performance of pruned models by rein- troducing adjacent layers for skipped layers. Starting with pruning based on skip indices S, it evaluates each skipped block for reuse with discrepancy score D between pruned and dense model outputs under three configurations: current state, reuse of the previous sub-block, and reuse of the sub- sequent sub-block. The configuration with the smallest D is selected, ensuring aligmment with the dense model. MHA and FFN sub-blocks are reused as their respective types, maintaining consistency and functionality. The sub-block with the lowest discrepancy is reintroduced, iteratively up- dating the dictionary for Re-use R until all skipped layers are evaluated. This results in refined dictionary R, allow- ing efficient compression while maintaining performance.
specifically tailored for T2I synthesis. The detailed configu- ration and example are provided in Appendix Sec. B.3.
4.1. Quantitative results
Metrics. We evaluated the performance of the Skrr and baseline models with four metrics: Fréchet Inception Dis- tance (FID) (Heusel et al., 2017), CLIP (Radford et al., 2021) score and DreamSim (Fu et al., 2023) score with MS- MS-COCO (Lin al., 2014) 30k validation set, along- side GenEval (Ghosh et al., 2024). FID measures the real versus generated image similarity using Inception-v3 fea- tures (Szegedy et al., 2016). The CLIP score measures the semantic alignment of the image-text. DreamSim assesses the composition and color similarity in pruned text encoders versus the original. GenEval measures pruned text encoders' effects on T2I synthesis across six dimensions: single object, two objects, counting, color accuracy, positional alignment, and color attribution. See Appendix Sec. B.4 for details.
4. Experiments
Baselines. To evaluate the performance of Skrr, we com- pared multiple LLM-based blockwise pruning techniques for T2I synthesis, including ShortGPT (Men et al., 2024), LaCo (Yang et al., 2024) and FinerCut (Zhang et al., 2024). Pruning in diffusion pipelines specifically targets the T5-
T2I synthesis performance comparison. We evaluated the T2I synthesis performance of SkIT against various base- lines and metrics, as shown in Table 1. ShonGPT maintains performance at low sparsity, but its performance deteriorates rapidly as sparsity increases. A similar pattern is observed for LaCo. FinerCut exhibits more gradual decline in per- formance compared to other basclines, but the generated im- ages still show a significant drop in GenEval scores. In con- trast, Skrr demonstrates performance comparable to dense models in all the sparsity levels, achieving high fidelity while preserving strong text alignment metrics especially in high-sparsity settings. Interestingly, in some cases, the FID score for compressed models improves, while DreamSim,
XXL (Raffel et al., 2020), which constitutes most of the parameter size. Detailed configurations and implementation specifics are available in Appendix Sec. B.I and Sec. B.2.
Most blockwise pruning methods rely on a cali- bration dataset to identify and remove less influential blocks. To this end, we constructed a calibration set by sampling 1k text prompts from the CC12M (Changpinyo et al., 2021),
Dataset.
6

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Dense
ShortGPT
LaCo
FinerCut
Skrr (Ours)
(gpuniity level })
PixArt-21
"A vast galaxy with swirling nebulae in vivid reds, purples, and golds, and a radiant, glowing planet with rings reflecting light like a prism."
(epursily level 2]
PixArt-
"A lively fountain square where people play musical instruments, with rainbows refracting the mist."
5y
PixAit-E
Lavey Aa (Syin
"A child holding lontern with sofily glowing golden light, standing field tall grass under deep indigo sky filled with sparkling
SD3
"A clock with intricate gears exposed, frozen in time."
FLUX. I-evv -dev
5 Peac Sontrne
young with ciit--spaatered hands sitting cross-legged the floor of brightly studio. She's 's wearing round glasses and colorful apron, swrounded by unfinished convases and jars of paintbrushes. Her focused expression at a masterplece in progress.
Figure Comparison images generated with baseline and Skrr-compressed text encoders across Pix E, Stable Diffusion (SD3), and FLUX. 1 1-dev. At low sparsity (level 1-24.3% 5 for ShortGP and Laco, 26.3% for FinerCut, and 27.0% for Skrr), both methods perform comparably to dense models. but Skrr outperforms at higher sparsity(level 2-32.4% for ShortGPT and Laco, 32.2% for FinerCut, and 32.4% for Skm. level 3 4-45%5% for ShortGPT and Laco, 41.7% for FinerCut, and 41.9% for Skrr), maintaining alignment to dense model and preserving details in the prompt such as "glasses", "colorful apron", ", and "paint-splattered hands", where baseline methods fail.
CLIP, and GenEval scores degrade. This indicates that com- pressing text encoders preserves or boosts image quality, while text alignment declines. We provide further detailed analysis in Sec. Experimental results for FLUX. -schelll with higher sparsity levels (50%) and additional benchmarks provided in Appendix Sec. C.6.
the model precision standardized to Bfloat16. As shown in Table 2, Skrr significantly reduces the parameters and memory usage similar to other baselines compared to dense model. While its FLOPs are slightly higher than the other baselines, this negligible, since the text encoder accounts for only small portion (0.6%) of the pipeline's total FLOPS.
4.2. Qualitative results
Computational cost. To evaluate the efficiency of Skrr, we compared the computational cost of dense and pruned models. We measured the number of parameters, memory usage, and total FLOPs within the PixArt-E, pipeline, with
We present qualitative results that demonstrate the perfor- mance of T2I synthesis using Skrr-compressed text en- coders, extending the experiments to SD3 and FLUX, state-

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Table 2. Number of parameters (Param.), memory usage (Mem.), and FLOPs were evaluated on PixArt-\' across pruning methods, considering the entire pipeline to analyze each strategy's impact on computational cost. All metrics were measured at the maximum sparsity achieved by each pruning method.
Table 3. T21 performance with various beam size k. While larger k values incur higher computational costs for the candidate search, they enhance T2I performance. (k = 1 a greedy approach
Spursity
GenEval
k
CLIP
DreamSim
(%)
Single.
Count.
Colors.
Method
Sparsity (%)
Param. (B)
Mem. (GB)
TFLOPs
41.9
1
0.310
0.737
0,900
0.372
0.739
41.9
2
0.312
0.757
0,912
0.450
0.731
Dense
0.0
5.42
10.18
91.94
41.9
3
0.313
0.746
0,912
0.450
0.755
ShorGGP
40.5
3.49
6.59
91.79
40.7
4
0.310
0.739
0.925
0.328
0.707
FinerCut
41.7
3.43
6.48
91.74
Skrr (Ours)
41.9
3.43
6.46
91.90
The Re-use phase addresses perfor- mance degradation from the Skip phase, optimizing within memory constraints. Its impact was assessed by comparing model performance with and without Re-use. As shown in Fig. 6, Skip alone preserves high fidelity but may fail to accurately reflect the text prompt. In contrast, incorporating Re-use produces images closely resembling those from the dense model, ensuring better alignment with the prompt.
Influence of Re-use.
Dense
Skip only
Skip Re-use (Skrr)
3)
PixcAr--2
"Two rod balloons fied mazibox sunzy zuburban street."
FLUX. I-dey
Leguriie lovel
Influence of beam search. We performed an ablation study that evaluated the effect of the beam search at dif- ferent values of k, as shown in Table 3. As & increases, performance initially improves and then decreases. This trend is consistent with previous findings (Cohen & Beck, 2019; He et al., 2023), which observed similar behavior in LLM decoding in which performance increases and then deteriorates as the size of the beam increases. Based this observation, we selected the optimal beam size k = 3.
"A with puicisplateeel honde uning croeriigee the of brightly atudin She's wearing
and colorfil upres, surrannded by usffaishusl andjars of paintrnukees Her forused expression at masterpiccr in progrese"
Figure 6. Ablation study on Re-use. Without Re-use, Skip alone leads to images that often misalign with the prompt, while Re-use ensures more faithful adherence to the prompt.
of-the-art diffusion models. As shown in Fig. 5, ShortGPT achieves satisfactory image quality at low sparsity, but di- verges significantly at higher sparsity levels, failing to align with input text at sparsity over 40% (level 3). LaCo and FinerCut maintain image fidelity across sparsity levels but show reduced alignment with the dense model. In contrast, Skrr consistently preserves both image quality and align- ment, closely resembling the outputs of dense models. For SD3 and FLUX, which use multiple text encoders, image fidelity remains intact at higher sparsity levels, but similarity to dense encoder output decreases. These results highlight Skir's robustness in preserving image quality and alignment to original image while exhibiting consistent behavior across models under varying sparsity conditions.
5. Discussion
Previously, we noted better FID results with T2I synthe- sis when using text encoder with skipped layers. Beyond CFG, other guidance methods in diffusion models include perturbed attention guidance (PAG) (Ahn et al., 2024) and autoguidance (Karras et al., 2024), both of which approxi- mate the unconditional score by modifications of denoising networks. We propose that layer skipping or merging affects the null text embedding similar to these methods. To verify this hypothesis, we perturbed fa, as described below.
4.3. A blation study
Sa - ~~N(0, I),
(8)
We conducted an ablation study to evaluate the contribution of each component in the Skrr framework. Specifically, we analyzed the effectiienees of Re-use, highlighting its role in minimizing performance degradation from Skip. We also examined the size of the beam, demonstrating its effective- ness in block selection. The experiments were carried out on PixAnt- and a subset of the MS-COCO (Lin et al., 2014) validation dataset with the highest sparsity. We provide ad- ditional ablation study in Appendix Sec. C.5.
where is a random vector sampled from normal distribu- tion and À is small scalar value. The FID and CLIP scores of the original model and the unconditional output with small perturbations applied to So were measured on the MS- COCO 30k dataset. The results show that the FID decreases, indicating that the fidelity improved when perturbations are introduced to the unconditional feature fa. Detailed results and configurations are provided in Appendix Sec. C.3.
8

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
6. Conclusion
Education and Research Program for Future ICT Pioneers, Seoul National University.
In this paper, we introduce Skip and Re-use Layers (Skrr), an effective compression method for the text encoder in text-to-image (T2I) diffusion models. Skrr integrates three key components: (1) a pruning metric based on the Skrr dot product to identify redundant sub-blocks, (2) beam search- based algorithm to account for interactions between trans- former blocks during pruning, and (3) a re-use mechanism that mitigates performance degradation by leveraging the re- maining layers to recover lost capacity from skipped blocks with theoretical supports. Extensive experiments demon- strate that Skrr consistently outperforms existing blockwise pruning techniques for text encoder compression in image synthesis tasks, achieving qualitatively and quantitatively su- perior results. Additionally, our analysis reveals that pruning or merging layers not only reduces model complexity but can also enhance certain aspects of performance. We further analyze these improvements from the perspective of model guidance, offering insights into how structural adjustments contribute more effective T2I generation.
References
Ahn, D., Cho, H., Min, J., Jang, W., Kim, J., Kim, S., Park, H. H., Jin, K. H., and Kim, S. Self-rectifying diffusion sampling with perturbed-attention guidance. In ECCV, pp. 1-17. Springer, 2024.
Ashkboos, S., Croci, M. I.., Nascimento, M. G. d., Hoefler, T., and Hensman, J. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401. 15024, 2024a.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li. B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier- 4-bit inference in ro- tated llms. arXiv preprini arXiv:2404. 00456, 2024b.
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pixx Learning to follow image editing instructions. In CVPR, pp. 18392-18400 2023.
Impact Statement
Cao, M., Wang, X., Qi, Shan, Y., Qie, X., and Zheng, Y. Masactrl: Tuning-free free mutual celf-attention control for consistent image synthesis and editing. In CVPR, pp. 22560-22570, 2023.
Text-to-image (T2I) synthesis models are powerful tools that can enable creative expression, education, and accessi- bility-but they also raise important societal concerns, such as potential misuse in misinformation, surveillance, deep- fake generation. While our work does not directly advance the fidelity or expressiveness of T2I models, it focuses in- stead on improving their memory efficiency by introducing Skrr, a method for pruning text encoders. This makes exist- ing models more accessible in settings with limited hard- ware, such as mobile or edge devices, and could support broader, more equitable access to generative tools. However, we recognize that making these models easier to deploy may also lower the threshold for harmful or irresponsible use. These concerns are not the primary focus of our work, but they are important to consider as the field progresses. We encourage complementary research and governance ef- forts aimed at ensuring that the benefits of more efficient generative models are realized in socially responsible ways.
Castells, T. Song, H.-K., Kim, B.-K., and Choi, S. Ld- pruner: Efficient pruning of latent diffusion models using task-agnostic insights. In CVPR, pp. 821
2024a. 20244.
Castells, T., Song, H.-K., Piao, T., Choi, S., Kim, B.-K., Yim, H., Lee, C., Kim, J. G., and Kim, T.-H. Edgefu- sion: On-device text-to-image generation. arXiv preprint arXiv:2404.11925, 2024b.
Changpinyo, S., Sharma, P., Ding, N., and Soricut, R. Con- ceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, pp. pp. 3558- 3568, 2021.
Chen, J., Ge, C., Xie, E., Wu, Y., Yao, L., Ren, X., Wang, Z., Luo, P., Lu, H., and Li, Z. Pixart-sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In ECCV, pp. 74 Springer, 2024.
Acknowledgements
This work was supported in part by Institute of In- formation & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea govern- Artificial Intelligence Graduate School Program (Seoul National University)] and the National Research Foundation of Korea(NRI)) grant funded by the Korea goverment(MSITT (No. NRF- 2022M3C1 A30920221 1). Also, the authors acknowledged the financial support from the BK21 FOUR program of the
Cho, J., Hu, Y., Baldridge, J. M., Garg, R., Anderson, P., Krishna, R., Bansal, M., Pont-Tuser J., and Wang, Davidsonian scene graph: Improving reliability in fine- grained evaluation for text-to-image generation. In ICLR, 2024.
ment(MSIT) 210.RS-2021-121133
Cohen, E. and Beck, C. Empirical analysis of beam search performance degradation in neural sequence models. In ICMI., pp- 1290-1299. PMIR, 2019.

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML. PMLR, 2024.
Huang, K., Sun, K., Xie, E., Li, Z., and Liu, X. T2i- compbench: A comprehensive benchmark for open-world compositional text-to-image generation. NeurIPS, 36: 78723-78747, 2023b.
Fang, G., Ma, X., and Wang. X. Structural pruning for diffusion models. NeurIPS, 2023.
Karras, T., Aittala, M., Kynkäänniemi. T. Lehtinen, J., Aila, T., and Laine, S. Guiding diffusion model with a bad version of itself. arXiv preprint urXiv:2406.02507, 2024.
Freitag, M. and Al-Onaizan, Y. Beam search strategies for neural machine translation. ACL, pp. 56, 2017.
Kawar, B., Zada, S., Lang, O.. Tov, O., Chang. H., Dekel, T., Mosseri, I., and Irani, M. Imagic: Text-based real image editing with diffusion models. In CVPR, pp. 6007-6017, 2023.
Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang. R., Dekel, T., and Isola, P. Dreamsim: Learning new dimensions of human visual similarity using synthetic data, arXiv preprint ar X3v:22336009344 2023.
Kim, B.-K., Song, H.-K., Castells, T., and Choi, S. Bk-sdm: A lightweight, fast, and cheap version stable diffusion. In ECCV, pp. 381-399. Springer, 2024.
Ganjdanesh, A., Shirkavand, R., Gao, S., and Huang, H. Not all prompts are made equal: Prompt-base based prun- ing text-to-image diffusion models. arXiv preprint arXiv:2406. 12042, 2024.
Ko, J., Kim, S., Chen, T., and Yun, S.-Y. Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03889, 2024.
Ghosh, D., Hajishirzi, H., and Schmidt, I. Geneval: An object-focused framework for evaluating text-to-image alignment. NeurlPS, 36, 2024.
Lee, Y., Lee, Y.JJ., and Hwang. Dit-pruner: Pruning diffusion transformer models for text-to-image synthesis using human preference scores. In ECCVW, pp. 1-9, 2024.
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers, arXiv preprint arXiv:2403.1178% 2024.
Li, M., Lin, Y., Zhang, Z.,, Cai, T., Li, X., Guo, J., Xie, E., Meng, C., Zhu, J.-Y., and Han, S. Svdqunat: Absorb- ing outliers by low-rank components for b-t diffusion models. arXiv preprint arXiv:24/ 2:Xi::2411000000 1.05007, 2024a.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arX0v:23/2.00752, 2023.
He, J., Sun, S., Jia, X., and Li, W. Empirical analysis of beam search curse and search emoos with model errors in neural machine translation. In EAMT, pp. 91-101, 2023.
Li, X., Liu, Y., Lian, L., Yang, H., Dong, Z., Kang, D., Zhang, S., and Keutzer, K. Q-diffusion: Quantizing diffu- sion models. In CVPR, pp. 17535-17545, 2023.
He, Y., Liu, L., Liu, J., Wu, W., Zhou, H., and Zhuang, B. Ptqd: Accurate post-training quantization for diffusion models. NeurlPS, 36, 2024.
Li, Y., Wang, H., Jin, Q., Hu, J., Chemerys, P., Fu, Y., Wang, Y., Tulyakov, S., and Ren, Snapfusion: text-to-image diffusion model on mobile devices within two seconds. NewrIPS, 36, 2024b.
Heusel, M., Ramsauer. H., Unterthiner, T., Nessler, B., and Hochreiter, Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS, 30, 2017.
Lin, J., Tang. J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. MLSys, 6:87-100, 2024.
Ho, J. and Salimans, T. Classifier-free free diffusion guidance. In NewrIPSW, 2021.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In ECCV, pp. 740-755. Springer, 2014.
Hsich, C.-Y., Li, C.-L., Ych, C.-k., Nakhost, H., Fujii, Y., Ratner, A., Krishna, R., Lee, C.-Y., and Pfister, T. Distill- ing step-by-step! outperforming larger language models with less training data and smaller model sizes. In ACL, pp. 8003-8017, 2023.
Liu, Y., Zhang, K., Li, Y., Yan, Z., Gao, C., Chen, R., Yuan, Z., Huang, Y., Sun, H., Gao, J., et al. Sora: A review on background, technology, limitations, and op- portunities of large vision models. URL https://arxiv. org/abs/2400. 17 2024.
Huang, K., Guo, X., and Wang, M. Towards efficient pre- trained language model via feature correlation distillation. NeurlPS, 36:16 161 14-16128, 2023a.
10

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Men, X., Xu, M., Zhang, Q. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403.03885. 2024.
Song, Y., Lorraine, J., Nie, W., Kreis, K., and Lucas, J. Multi-student diffusion distillation for better one-step generators. arXiv preprint arXiv:2410.23274 2024b.
Sun, M., Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models, 2016.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In ICCV, pp. 4195-4205, 2023.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Rethinking the inception architecture for computer vision. In CVPR, pp. 2818-2826, 2016.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, Sdxl: Im- proving latent diffusion models for hishluesol image synthesis. arXiv preprint arXiv:2307.01952. 2023.
van der Ouderaa, T. F., Nagel, M., Van Baalen, M., and Blankevoort, T. The Ilm surgeon. ICLR, 2023.
Wang, H., Shang, Y., Yuan, Z., Wu, J., Yan, J., and Yan, Y. Quest: Low-bit diffusion model quantization via efficient selective finetuning, arXiv preprint arXiv:2402.03666, 2024a.
Polyak, A., Zohar, A., Brown, A., Tjandra, A., Sinha, A., Lee, A., Vyas, A., Shi, B., Ma, C.-Y., Chuang, C.-Y., al. Movie gen: A cast of media foundation models. arXiv preprint arXiv:2410. 0et.:2240..3777 13720, 2024.
Wang, Z., Jiang, Y., Zheng, H., Wang, P., He, P., Wang, Z., Chen, W., Zhou, M., et al. Patch diffusion: Faster and more data-efficient training of diffusion models. NeurIPS, 36, 2024b.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-33 using 2d diffusion. arXiv preprint arXiv:2209. 14988, 2022.
Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural PMLR,
Wang. Z., Lu, C., Wang, Y., Bao, F., Li, C.,, Su, H., and Zhu, J. Prolifiedreamer: High-fidelityy and diverse text-to-3d generation with variational score distillation. NeurIPS, 36, 2024c.
language supervision. In ICML, pp. 2021.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In ICML, pp. 38087-38099, PMLR, 2023.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 21((1)): 1-67, 2020.
Yang. Y., Cao, Z., and Zhao, H. Laco: Large language model pruning via layer collapse. arXiv preprint 1AX1:2442.11187 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, and Ommer, High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Zhang. Y., Li, Y., Wang. X., Shen, Q., Plank, B., Bischl, B., Rezaei, M., and Kawaguchi, K. Finercut: Finer-grained interpretable layer pruning for large language models. arXiv preprint arXiv:2405. 18218, 2024.
Ryu, H., Park, N., and Shim, H. Dgq: Distribution-awee aware group quantization for text-to-image diffusion models. arXiv preprint 2xXXx:5511..4044., 2025.
Seo, H., Kim, H., Kim, G., and Chun, S. Y. Ditto-nerf: Diffusion-basse iterative text to omni-directional model. arXiv preprini arXiv:2304.02822 2023.
Zhao, Y., Xu, Y., Xiao, Z., Jia. H., and Hou, Mobilediffu- sion: Instant text-to-image generation mobile devices. In ECCV, pp. 225-242. Springer, 2024.
Sherstinsky, A. Fundamentals of recurrent neural network (rnn) and long short-term memory (Istm) network. Phys- ica D: Nonlinear Phenomena, 404:132306, 2020.
Soboleva, D., Al-Khateeb, F., Myers, R., Steeves, J. R., Hestness, J., and Dey, N. SlimPajama: 627B token cleaned and deduplicated version of RedPajama, 2023.
Song. J., Oh, K., Kim, T., Kim, H., Kim, Y., and Kim, J.JJ. Sleb: Streamlining llms through redundancy verification and elimination of transformer blocks. ICML. 2024a.

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
A. Proofs
A.1. Proof of Lemma 3.1
Lemma 3.1 (Error bound of two transformers). Let M: (x, + I2 be an L-block transformer with input x E R and parameters (f1....,fL) defined as:
M M (Ft-1+1)
(FI +1),
(A1)
where Fj (2 B.) + 2441 the i-th block with parameters 0i and 2i41 E TR...
With each block F; being Li-Lipschitz in the input that satisfies follows.:
[P(( 0.) - Fx(2 S Lillz - z'1l
(A2)
And Mi-Lipschitz in the parameters which follows.
[[ 0,) Pi(z; 0,11 S M,110 0'11
(A3)
Then, for any two parameter sets = (OI...., 0z) and 0 - (0i,..., Ôr.). the following holds:
[IM(x;)) M(a;; 011
II (1+ L&) M.l10: &:1l
(A4)
Ami+1
Proof.
With Eq. A1, we can formulate the difference two hidden states between dense model and modified model as follows.:
|| - f141/l = [|[ F(z; 0:)] - [ + ((:;;0)))
(A5)
By the triangle inequality:
||++11 - &1+1/l S || 2,ll + (2) 01)
(A6)
(A)
We now split term (A) with the assumptions in Eq. A2 and Eq. A3:
(A) = || [(2;; 0,) F2;;; 0:11
(A7)
< [Z;; 0.) (2; 0,)1 + (1::((2;))) (2(2;;0))
(A8)
- 1i 2:11 + M, | 0.11
(A9)
inpur Lipschitz
parameter Lipschitz
So combining,
|
&+1/l
I ||: - -2sl + Lgll2 - &ll +M - 8ell
(A10)
Hence,
|| 2141ll < (1 + T+)l - &ll + Ms110 0.1l
(A11)
Define the error at block as
E, || - -2:l
(A12)
Eq. A1l becomes
E & (1+ L,), + M1 - 0.1l
(A13)
We start from Ei = ||||| 2111 || x|| = 0 (since both netowrks see the same input x)) Thus:
Eg < (1 + E + 10 - 6 = M, 110 0,1l
(A14)
Eg < (1 + L2) + M2| 0.11 s (1 +
- 0.11 M2 - 0211]
(A15)
If we do recursive telescoping over all blocks, we get
L
E1+1 = || - 21+1/l 5
Σ
II
(1+ Ln) Mallo, 0,11
(A16)
==1
kmi+1
12

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Since MMx;0) =L+1 and M(x; I have shown:
-M(x;)) = || 21+1l1 - Σ
II k=i+1
(1+ La) M.110 0.1l
(A17)
i=
A.2. Proof of Theorem 3.2
Theorem 3.2 (Tighter error bound of Re-use). Under the same assumptions in Lemma 3.1, let 02 denote the i-th block of a transformer that is skipped, f represent the corresponding Re-used block, Uskip is a error bound for compressed model with Skip alone, and Usup, Re-use a error bound for compressed model with skip and reuse. If following condition is satisfied,
110; 0511 < 110:11
(A18)
then, the following inequality holds:
U Skip, Re-use
Skip
(A19)
Proof. With Lemma 3.1, we can formulate the error bound of compressed model as follows:
[[M{z;)) M(x; 0)1l S Σ cilee 0.1
(A20)
where C2 is a constant for each i-th block. For the unskipped block, 0, = 0;, 11:;-10,000 0,11 = So, we define a parameter set
0skip that exclude 0 in the parameter set 0. And we can rewrite the Lemma 3.1 as follows:
[|M 0) M(x;; 0shipp)
C.l1 6:ll
(A21)
VES
where S is a set of skipped (pruned) block indices. And skipped block can be represented as follows:
0,,skip 0
(A22)
Then, we can manipulate the error bound Eq. A21 with following.
[| 0) - MMx;; Oskip) S
Σ
C-110,11
Uskip
(A23)
:ES
Re-use substitute the skipped weight to the weight adjacent block:
O2.Ro-use 0r
(A24)
we make set i that satisfies Eq. A18 and denote as R then apply Re-use,
[|M{;;) - -M(x; Askip. wsw <
Σ
C110.111 +
Eslle, - 871 = Ustip,
(A25)
LESAiER
jER
Since the consists of block indices that satisfies Eq. A18, we have shown:
U Skip. Re-use
Σ
C.Il8.11 + E 0711 < Σ Call0k| = Usuip
(A26)
JESAIER
JER
RES
13

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
B. Detailed Experimental Setup
B.1. Baseline configurations
Table AI. Block indices ordered by the Block Influence (BI) score obtained from ShortGPT with our calibration dataset. Blocks with lower BI scores are ranked carlier, while those with higher BI scores are ranked later. During block pruning, blocks with the lowest BI scores are pruned first, according to the specified pruning ratio.
Order of block index
10, 11, 8, 9, 12.1122212 13, 5, 14, 6, 4, 3, 7, 15, 17, 16, 18, 2. 19, 20, 21, 22, 23, 0
ShortGPT (Men et al., 2024) ShortGPT calculates the Block Influence (BI) score by measuring cosine similarity between intermediate features of each layer, extracted from the calibration data, and then taking its complement. We calculated the BI scores using the Ik calibration subset of the CC12M dataset that we constructed and sorted the blocks in ascending order based on their BI scores. The ordered block indices are presented in Table Al.
LaCo (Yang et al., 2024) LaCo presents an algorithm that merges adjacent transformer layers on the basis of the cosine similarity between their output features and those of the original model. If similarity exceeds predefined threshold, the layers are merged to reduce the size of the model. However, our experiments revealed that LaCo's performance is highly sensitive to its hyperparameter settings. In some cases, the algorithm failed to effectively reduce the number of parameters.
To ensure a fair comparison, we conducted thorough and extensive experiments to identify the optimal hyperparameters that maximize LaCo's 's performance in the Text-to- Imaee (T2I) diffusion model. The selected hyperparameters are as follows: Layer collapse interval I 2, Number of layers to merge M 2, First layer to merge C = 1, Last layer to merge H 24, cosine similarity threshold T = 0.7.
We applied LaCo by continuously merging layers until the compressed model achieved the desired target sparsity.
Table A2. Sub- blockllck indices ordered with MSE metrics from FinerCut with our calibration dataset. For the efficiency, we extracted top 10p-24 sub-block indices for pruning. During the pruning process, blocks with the lowest MSE are pruned first, according to specified pruning ratio.
Order of sub-block index
46, 5, 11, 6, 47, 44, 20, 19, 18, 45, 40, 41, 1, 32, 31, 17, 24, 23, 3, 42, 43, 30, 38, 37
FinerCut (Zhang et al., 2024) FinerCut employs a finer-grained blockwise pruning strategy based the structural decomposition of transformer blocks into two distinct sub-blocks: Multi-Heat Attention (MHA) and Feed-Forward Network (FFN) sub-blocks. To assess the importance of each sub-block, FinerCut originally proposed various evaluation metrics: 1) cosine similarity, 2) mean-squared error (MSE), and 3) Jensen-Shanme -haenonn divergence (JSD). In its original implementation for regoereegeesive LLMs, FinerCut adopted JSD after evaluating the effectiveness of these metries. However, since the text encoder in the T2I diffusion model lacks language modeling head, metries based on perplexity and JSD could not be applied. For the fair comparison, we implemented FinerCut using MSE the sole metric to evaluate sub-block importance, which is the closest metric with our discrepancy metric. We also conducted experiments of FinerCut with cosine similarity in Sec. C.6.
The sorted sub-block indices determined by FinerCut are presented Table A2. The even indices represents the MHA blocks and the odd indices denote the FFN blocks. Due to FinerCut's sub-block-level pruning granularity, it was not possible achieve the exact sparsity ratios as ShortGPT and LaCo. Therefore, pruning was carried out up to the sub-block index that most closely matched the target sparsity for a fair comparison.
Skrr (Ours) This section details the order of indices determined during the Skip phase of Skrr. Because the projection module is incorporated into the denoising module of each diffusion model, the sub-block indices vary across different diffusion models. We executed the Skip phase for all models, and the resulting order indices for each model is presented in the Table A3. This provides insight into how the indices are prioritized during the Skip phase for various diffusion architectures.
14

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Additionally, the re-use indices of Skrr in PixArt-E are presented in Table A4, Table A5, and Table A6. The re-use indices independently computed using the skip indices for each sparsity level, and overall, the later blocks exhibited tendency not to be reused.
Table A3. Sub-block indices ordered with discrepancy metric D from Skrr with our calibration dataset in PixArt-E, Stable Diffusion 3 (SD3), and ILUx..-ddv. For the efficiency, we extracted top-24 sub-block indices for pruning. During the pruning process, blocks with the lowest discrepancy pruned first, according the specified pruning ratio.
Model
Order of sub-blocks
PixArt-E
46, 6, 6, 44,
24, 23, 17, 18, 45, 10, 30, 29, 40, 39, 38, 37, 42, 25, 22, 43, 36 22, 23, 17, 18, 30, 29, 43, 44, 20, 1, 41, 42, 40, 39, 38, 37, 21, 16
SD3
46, 45, 6, 11, 2,
FLUX.I-dev
46, 45, 6, 11, 20, 19, 22, 23, 30, 29, 47, 44, 32, 31, 18, 17, 37, 36, 43, 40, 39, 42
Skipped Block
Re-used Block
Skipped Block
Re-used Block
Skipped Block
Re-used Block
4
2
0
7
5
7
6
4
6
4
11
13
10
10
12
17
19
11
13
11
9
18
16
17
19
17
19
23
21
18
20
18
16
23
21
23
21
44
24
24
45
29
25
27
46
30
29
47
39
30
40
37
Table A4. Skm Re-use indices of sparsity level in PixArt-E.
44
38
45
39
46
40
47
42
44
Table A5. Skrr Re-use indices of sparsity
45
level in PixArt-E.
46
47
Table A6. Skır Re-use indices of sparsity level 3 in PixArt-
B.2. Diffusion model and text encoder
For the quantitative comparison, we performed experiments on two diffusion transformer (Peebles & Xie, 2023) (DiT) text encoders: PixArt-E (Chen et al., 2024). P employs the T5-XXL model (Raffel et al., 2020) as its text encoder. We have also conducted experiments on compressing text encoders that leverages several text encoders. For example, Stable Diffusion 3 (SD3) leverages CLIP-L, CLIP-G, and TS-XXL. The results compressing multiple text encoders are presented in the Sec. C.5. For quantitative and qualitative evaluations, we measured discrepancy and generated images using B precision with the pretrained weights P PixArt EixxArt-aphhat wwwpha/Pixxaa.
Sigma Miina-XX---- 2-1024-- MSS obtained from the Hugging Face Diffusers library. For PixArt-E, all images were generated in the resolution of 512 x 512. Furthermore, the FLUX model and SD3, included in the qualitative results, was evaluated using the stabi tyaa sstaallllleeeee fuusi on 3-med um and black- forest labs /FLUX. d-dev weights from the Hugging Face Diffusers library. The text encoder configura- tion and inference precision for both models were consistent with the aforementioned setup, using the Bfloat16 precision and applying the same pruning strategy to ensure a fair and consistent evaluation. Additionally, we fixed the number of function evaluations (NFE) to 20 across all diffusion models. All other hyperparameters, such as the classifier-fre guidance (Ho
15

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Salimans, 2021) scale, were the default value. All experiments were performed single Nvidia A100 or RTX 4090 GPU.
B.3. Calibration dataset
We constructed a calibration set derived from the CC12M (Changpinyo al., 2021) dataset identify blocks for pruning. While existing LLM- methods typically utilize calibration sets such as the Common Crawl's web corpus (Raffel et al., 2020) (C4), SlimPajama (Soboleva et al., 2023) and WikiText (Merity et al., 2016), primarily focusing on perplexity- -bsed performance metrics, these approaches are not directly applicable to T2I models. To address this gap, we curated calibration set specifically tailored for the T2I text encoder by selecting only clean captions and semantically rich prompts ranging from 150 to 250 tokens from the CC12M image-text paired dataset. This selection ensures a more effective calibration for image synthesis tasks. Representative examples of prompts from the constructed dataset are provided in Table A7.
B.4. Metrics
Fréchet Inception Distance (FID) (Heusel et al., 2017) score The Fréchet Inception Distance (FID) is a widely used metric for evaluating the performance of image generative models by quantifying the similarity between the distributions of real and generated images. Specifically, FID measures the Fréchet distance between feature representations extracted from pre-trained image classification model, typically the Inception- 3 model. This approach leverages the model's rich intermediate features to capture high-level image statistics. The FID score is formally defined as follows:
dF
= I1 1 tr< "'=+=++2(EE))
(A27)
where and Σ are the mean and covariance of the feature representations of real images, Jí' and E' are the mean and covariance of the feature representations of generated images. A lower FID score indicates that the generated images are more similar to the real images both quality and diversity.
CLIP (Radford et al., 2021) score The CLIP score evaluates the semantic alignment between a given text prompt and the image generated from that prompt by measuring the cosine similarity between their CLIP embeddings. This metric leverages CLIP model trained on extensive image-text pairs to capture cross-modal relationships. The CLIP score can be formally defined as:
Eimago (I) (T)
CLIP(I,T) cos (Eimago (I), Erext (T)) Dimage (IIll Frexa ( (T)ll
(A28)
where / the generated image, T' is the text prompt, Eimage (-) represents the CLIP image encoder, and Eeex(-) denotes the CLIP text encoder. The cosine similarity captures how well the generated image aligns with the semantic content of the text prompt. For our experiments, we leveraged the weights of the opena i / cl //cllppppo// ip-vi ti be--bbassppteeee model from the Hugging Face library to calculate the CLIP score.
DreamSim (Fu et al., 2023) score The DreamSim score quantifies the semantic similarity at the mid-level between two images by evaluating their compositional, stylistic, and color characteristics. It quantifies how closely the overall structure and visual attributes of the images align. Formally, the DreamSim score can be expressed as Eq. A29:
DreamSim(Ter, Ipmu) distpreamSim ( (Tre, Ipou), distpreamSim (--.) E [0.
(A29)
where Let represents the image generated using the original text encoder, and Ipen denotes the image generated using the pruned text encoder. The function distpreamSii (+)*) computes the normalized semantic distance between the two images, as output by the DreamSim model. By subtracting this distance from 1, the DreamSim score reflects higher similarity with higher values, effectively capturing the semantic consistency between the original and pruned models' outputs.
GenEval (Ghosh et al., 2024) GenEval is comprehensive evaluation metric designed to assess the degree to which T21 generative model aligns generated images with input text prompts. In this study, GenEval was employed to evaluate whether the image synthesis results produced by the compressed text encoder accurately reflect the intended textual descriptions.
The GenEval metric comprises six sub-metrics:
Single Object Generation Assesses the model's 's ability to generate images from prompts containing a single object (e.g..
16

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Table A7. Examples of prompts from the calibration set across various lengths are presented. These text prompts sampled from the CC12M dataset, featuring rich and descriptive expressions with lengths optimized for calibration. This selection ensures the prompts are semantically meaningful and well-suited for effective text-to- image model calibration.
Example prompts in calibration dataset
1. A collection of photography equipment neatly arranged on a wooden surface. Items include a camera, smartphone, tablet, drone, portable power bank, tripod, cleaning kit, strap, case, and backpack. The warm wooden background contrasts with the modern gear.
2. A moment at a subway station with vintage train numbered "7" at the platform. The platform has safety barriers and yellow line, illuminated by fluorescent lights. Text reads "last stop for the 7 train" and credits the photographer. 3. A modern living room with minimalist design. A pendant light provides a warm glow. A wooden table holds a glass of water, a book, a smartphone, and notebook. A white cabinet and cityscape view complete the cozy atmosphere.
A person wearing a white t-shirt with the text "A Day to Remember" in pink and black lettering. The shirt features a black collar and short sleeves, displayed plainly for product showcasing. A vibrant digital artwork of stylized cityscape. Buildings vary in color and pattern, resembling a patchwork quilt, creating dense, lively urban environment.
6. A small modern bathroom with brick-patterned walls and tiled flooring. A white sink under a window, glass shower enclosure, and toilet create a rustic yet clean look.
7. A vintage light-colored train car with blue and white stripes is parked on a track under a metal canopy. Metal stairs lead to the entrance, possibly part of a museum exhibit.
A wall with a playful quote: "In this house we are real, we make mistakes, we say I'm sorry, we give hugs, we give second chances, we forgive, we laugh lot, we love each other, we are a family." A guitar leaning against the wall adds a cozy, homey touch.
9. A vibrant bouquet of flowers arranged in clear glass vase. the bouquet consists of various types of flowers, including hydrangea, calla lilies, roses, and gerbera daisies, with burgundy berries interspersed among them. the flowers are in shades of pink and purple, creating a striking contrast. the arrangement is set against a plain, light-colored background, which accentuates the colors and textures of the flowers. the style of the image is a close-up photograph that captures the details of the floral arrangement.
10. The memorial church stanford university, a large, ornate building with a prominent cross at the top. illuminated at night. the facade is adorned with intricate mosaics and sculptures, including central figure that appears to be religious figure, possibly saint deity. the church's 's architecture is reminiscent of gothic and romanesque styles, with pointed arches and large central archway that leads to the entrance. the surrounding area is dimly lit, with the church standing out as a beacon of light in the darkness.
11. A serene lakeside setting with a houseboat that resembles a private yacht. the boat is equipped with a dining area featuring a table set for four with blue tableware, and a bar area with a blender, wine glasses, and bottle of wine. the deck is furnished with multiple lounge chairs and dining table, all under a retractable awning. the houseboat docked near a rocky shoreline with clear blue sky and majestic red rock formation in the distance, suggesting a location like lake powell. the overall atmosphere is one of relaxation and leisure, ideal for vacation getaway. 12. A graphic design with a stylized representation of a face, possibly a deity, with a serene expression. the face is framed by a green border with a white outline and a blue background. above the face, there is a crescent moon and a symbol that resembles a peace sign. below the face, the word "chill" is prominently displayed in bold, white capital letters. the overall style of the image is modern and graphic, with a clear emphasis on the word "chill" suggesting a theme of relaxation or tranquility.
"a photo of a giraffe").
2. Two Objects Generation Evaluates the model's ability correctly generate images from prompts with two distinct objects (e.g., "a photo of knife and stop sign").
3. Counting - Measures whether the model can accurately represent the specified number of objects (e.g. "a photo of three
apples"), ").
4. Colors Verifies whether the generated image correctly reflects the color specified in the prompt (e.g., "a photo of a pink car").
Position Tests the model's understanding of spatial relationships described in the prompt (e.g., "a photo of a sofa under
17

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Table A8. Sparsity ratio of the encoder, parameter count (Param.), memory usage (Mem.), FL.OPs ratio of text encoder (T5-XXI.) with respect to the total pipline, and total TFLOPs of the pipline were evaluated on SD3 across pruning methods, considering the entire pipeline analyze each strategy's impact on computational cost. metrics were measured the maximum sparsity achieved by each pruning method and NEE with 28.
Method
Sparsity (%)
Param. (B)
Mem. (GB)
FLOPs (%)
TFLOPs
Dense
0.0
7.66
14.49
0.41
174.2
ShortGPT
40.5
5.73
10.93
0.32
174.1
LaCo
40.5
5.73
10.82
0.32
174.1
FinerCut
41.7
5.67
10.82
0.30
174.0
Skrr (Ours)
41.9
5.73
10.93
0.35
174.1
cup ").
6. Color Attribution Assesses the correct assignment of specified colors multiple objects (e.g.. "a photo of a black car and green parking meter").
For evaluation, we generated images using fixed random seed, producing 553 distinct prompts with four images per prompt, resulting in total of 2,212 images. This setup ensured consistent and reproducible evaluation across all sub-metrics.
B.5. A Ablation study setup
Due to constraints in our experimental environment, it was not feasible to evaluate the entire MS-COCO 30k image dataset for the ablation study. Instead, we performed the study using a Ik subset. Given that both the CLIP score and the DreamSim score effectively capture the impact of text encoder pruning, we focus our experiments on evaluating various configurations of these two metrics within the ablation study. Furthermore, for GenEval, experiments were conducted on the entire set. The experiments discussed in the Sec. C.5 were conducted using the same experimental setup as described above.
C. Additional Experiments
C.1. Interaction between blocks
In addition to the block pairs shown in Fig. 3, we identified additional pairs that exhibit interactions. These block pairs consistently maintain cosine similarity; however, their norms change significantly. We provide several qualitative results that highlight these interactions in Fig. A1 which is experimented with PixArt-E. For cases where 2, the number of combinations increases to &1(48-A)) making exhaustive experimentation computationally infeasible. As result, we limit our analysis to a subset two-block interactions.
C.2. Computational cost on different diffusion models
In the main paper, we report the computational cost only for PixArt-E. In this section, we extend the analysis to include the computational cost of dense, baselines, and Skrr-compressed models on Stable Diffusion (SD3). Since SD3 contains more parameters and a more computationally intensive denoising module compared to Pix the text encoder consumes fewer FLOPs ratio in the total pipeline. However, the text encoder still accounts for significant memory usage. The detailed results provided in Table A8, At the highest sparsity level, FinerCut achieves the lowest parameter count, memory usage, and FLOPs due to its slightly higher sparsity. Skrr exhibits the same parameter count and memory consumption ShortGPT or LaCo at the same sparsity level (40.7%). While Skrr incurs slightly higher FLOOP count than other baselines due to the re-use mechanism, the additional computational cost is minimal, accounting for less than 0.05% of the entire pipeline or under 0.1 TFLOPS. Despite this, Skrr still demands less computation than the dense model. These results underscore ability of Skrr to perform T2I synthesis in a computationally efficient manner, even when applied to models with varying complexities.
C.3. Perturbation to the null condition feature
We conducted an experiment to evaluate the impact of perturbations on fo and their effect on the FID and CLIP score. Using the PixArt-E model, we applied small scalar parameter 10-2 and generated images for the MS-COCO
18

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
3rd skip
5th skip
3d4,5t skip
13th skip
15 skip
13th 15th skip
"Color photo of corgi made of transparent glass, standing on the riverside in Yosemite National Park."
"Game-Art- - An island with different geographical properties and multiple small cities floating in space. "
"Photorealistic closeup video of two pirate ships baitling each other as they sail inside cup of coffee." "
"Happy dreamy owl monster silting on tree bench, colorful glittering particles, foress background, detailed feathers.
Figure AI. Examples illustrate various block interactions. When individual blocks are skipped, the generated images remain highly similar to those produced by the dense model, showing minimal impact. However, when two blocks are skipped simultaneously, the image severely degraded, demonstrating the presence significant interactions between blocks.
30k validation set. The results demonstrated that the perturbed fa produced a lower FID score (22.89 - 20.65) and a comparable CLIP score (0.314 - 0.314) to the original model. Furthermore, we provide qualitative comparison between the images generated with the original null condition fz and those created using the perturbed feature vector fo Fig. A2.
C.4. Applying Re-use baselines
To further validate the effectiveness Re-use, we conducted experiments evaluate its compatibility with other block-wise pruning methods plug-and-pla, manner. Among the baselines considered for these experiments, ShortGPT and FinerCut employed block-prume prrunig techniques, allowing us to apply Re-use and carry the evaluations. ShortGPT prunes layer normalization, MHA, and FFN single unit, enabling to perform Re-use adjacent whole blocks. Similarly, FinerCut, which operates at the sub-block level like Skrr, also allowed the application of Re-use. The results of these experiments are presented in Fig. A3 and Fig. A4. The blocks re-used by shortGPT and Finercut are presented in Table A9 and Table A10, respectively. As shown in the results, Re-use demonstrated superior performance, further substantiating its effectiveness in alignment with both the experimental and thcoretical findings presented carlier.
C.5. Additional ablation study
Effectiveness of Re-use. this section, we provide additional experimental results to demonstrate the effectiveness of Re-use. This mechanism effectively mitigates the performance degradation caused by Skip by utilizing the remaining blocks in the model. To evaluate its impact, we performed extensive quantitative and qualitative experiments. Quantitatively, we
19

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Original
Perturbed
Original
Perturbed
Original
Perturbed
"Young min wearing glasses lounging on with three white laptop computers his
watery glass jar blooming flowers"
woman carrying couple
poles."
"Apple in and over-flowing bowl windowsill with a rose vase."
"Dinosaur statue park with various kiles flying through the air."
"A family is sitting around dinner table."
"A narrow hotel room with two made up beds,"
"A man holding brown dog and a video camera.""
"A small plane flying through a cloudy blue sky.""
"There many preparing
golden clock rhino sculpture sitting top of fireplace." "
"A sitting the back of couch man wearing glasses."
ribbon"
"Small bird sitting skateboard posed front of dark blue background cloth."
"A stops himself from falling off of his skateboard."
"A hotdog is sitting with fries in a car plate."
Figure A2. Images generated from same seed under the different null condition (original vs. perturbed) reveal notable differences. While the original model occasionally exhibits poor fidelity omits objects specified in the prompt, the images generated using the perturbed condition feature demonstrate higher fidelity, more accurate representation of the given prompt, and improved conditional image synthesis performance. All image pairs were generated with the same seed.
20

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
ShorGPT
ShortGPT + Re-use
Dense
ShorlGPT
ShortGPT + Re-use
"Three girafles standing amongat copse of frees."
"Seveaal slting around a table in small room with loptiog, boots and papers.
"A hnife and a pizza end into slices on the plate"
"a cluse of foam box of food"
"A crowd of young children sitting next to each other."
large green and yellow parked green yellow
PixArt-E
young boy weoring colorful umbrella
plate includes both meat vegetable
O
"A man standing another wearing headphones.
"large mustord yellow yellow
commercial airplane parked the airport."
"A curled up, asleep, chair."
"A rad motoreycle with back parked the side of road."
"A doablle real bus purked on street.
"A muloreyele parked ontiride in parking lot near the beach.
"A motorcycle is parked in busshyfildd
"A blue double decker bus that says Garage on it."
Figure A3. Qualitative results applying Re-use to ShorGGP.. high sparsity levels, ShorGGP struggles to generate high-fideltty images. Incorporating Re-us restores fidelity, text adherence, and alignment with the dense model.
21

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
FinerCut
FinerCul Re-use
Dense
FinerCut
FinerCut Re-use
"A child stops ofter toking a hite of slice of pizza."
"Somoone picks up slice of fresh pizza with one hand."
"A person on a motor bike on road."
"a close up of woman wearing jevelry hoat background
"The airplane is laking off the runvayy at the airport."
"A couple older men siting on top of a bench."
PixArt-E
young woman riding slope holding ski poles.
"Cauliflower, carrots, broccoli are arnanged vegetable bowl.
"A close up with many flowers.
"A riding snowboard top covered slope:"
"A view trufic street mutorcycles."
filled with clatter and laptop computer.
"A siice of pizza sitting plate box."
"A double red bus porked street.
"A close up of rug ground." "
"A yellow duck boat floating top of lake."
Figure A4. Qualitative results of applying Re-use to FinerCut. While FinerCut performs well compares other baselines in maintaining fidelity and alignment, its alignment with the dense model decreases. Incorporating Re-use effectively restores this aligmment.
22

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Skipped Block
Re-used Block
Skipped Block
Re-used Block
4
7
5
7
6
7
8
7
9
11
10
7
17
21
11
7
18
16
12
7
19
15
13
7
20
16
14
7
23
21
Table A9. ShonGPT indices
29
32
40
41
42
44
46
47
Table A10. Finercut re-use indices
measured the CLIP, DreamSim, and GenEval scores of the PixArt- model with and without Re-use in the maximum sparsity. The results, presented in Table A11, show that Re-use enhances the CLIP, Dreamsim and GenEval scores. Although minor performance degradation was observed in other GenEval scores, the high performance achieved in challenging metrics like counting underscores the effectiveness of Re-use, especially in scenarios where the text encoder's capability plays a critical role.
In addition to the quantitative findings and qualitative examples presented in the main paper, we further confirmed that Re-use outperforms Skip alone in a variety of settings. These results are illustrated in Fig. A5 for PixArt-E and Fig. A6 for FLUX.. 1-dev. These visualizations reinforce the effectiveness of Re-use in generating images that align closely with text prompts, while maintaining model-agnostic behavior. The results demonstrate that Re-use not only enhances adherence to textual descriptions but also improves the performance of dense models across diverse scenarios.
Table AII. Quantitative ablation study for Re-use with PixArt-E. T2I synthesis performance with Re-use demonstrates that it effectively restores performance degraded by pruning. achieving excellent recovery without incurring additional memory overhead.
GenEval
Re-use
CLIP
DreamSim
Single
Two.
Count.
Colors
Pos.
Color attr.
Overall
x
0.311
0.745
0.928
0.379
0.409
0.758
0.065
0.088
0.438
/
0.312
0.746
0.913
0.409
0.450
0.755
0.055
0.068
0.442
Compressing multiple text encoders. Modern text-to-image (T2I) diffusion generative models frequently employ multiple text encoders to enhance their performance. notable example is Stable Diffusion 3 (SD3), which incorporates CLIP-L, CLIP-G, and T5-XXL text encoders. In the experiments in the main article, we focused on compressing the T5-XXL text encoder, which is the largest and has the highest parameter count. Extending this approach, we evaluated the performance of the model when compressing all text encoders simultaneously. Specifically, for SD3, we applied compression to CL.PP-L (30.6% sparsity), CLIP-G (30.3% sparsity) and T5-XXL (41.9% sparsity).
The guantitative results, summarized in Table A12, reveal that although the CLIP and GenEval scores experience slight reductions, they remain sufficiently comparable to those of the dense model. Furthermore, qualitative results, illustrated
23

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
Skip only
Skip Re-use (Skrr)
Dense
Skip only
Skip Re-use (SkrT)
"Three books stacked neatly on desk, thetr spines displaving bold, colorful
"Two striped socks honging on a clothesline in the wind.
titles."
Pixart-
"Four vintage leucups stacked precurionsly matching
"Two puper cranes folded folded, from patterned origumi paper, sitting windowsill.
"Fwo dolphins leaping ant of the ocean under bright blue sky.""
"Thiree books stocked neatly on a desk, their spines displaying bold, colorfud titles."
"Three oranges placed blue ceramic bowl rustic wooden table.
"A sevene countryside with rolling flelds, sheep grazing, and small stone bridge.
Figure A5. Qualitative results Re-use. Images generated with PixAnt-), and encoder (T5-XXI.) compressed by the full Skrr framework (dense, Skip only, and Skip with Re-use) are compared. With Skip only, performance noticeably degrades, particularly for detailed prompts or tasks requiring strong text encoder capabilities, such counting, resulting deviations from dense model outputs.
24

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
Skip only
Skip Re-use (Skrr)
Dense
Skip only
Skip (Skrr)
sculptor chiseling vtatue open studio, fruguenni snotteing with each procise strike. Sunlight reflects off the smooth, polished neffcce of fintched nearby."
"Floating green cliffs cascadlngg waterfalls rurset, by soft and vibrant colors the sky.
d-ev
"An anciont cliffside tomple with ornate pillars, warn sunset lighting, and overgrowt greewayy
"A sclentist in iaboratory surrounded by glowing vials and advanced equipment consully piperting blue liguist Her coat slightly wrinkied, and sufety goggles rent on her forehead."
yiung stting reading sipping on a cappaccino The nutlight streame throngh the window casting worm glow her face she reflect the
"A gardoner planting socailing community gursion, survounded by sona wegetoblez and flowerz watering con wote beside her as she geatly pats the soil around
the plants."
"A beckeeper white mit ronding liee on sonny form, Bees buzz around or he consfilly iaspects boweycumb, liquid glistening light." "
"A young woman painting mural on the wall of on orphanages. hor Butt rieaf back louse Chiiden guther and paint af the growing artwork.
helping
piggle
Figure A6. Qualitative results on Re-use, Images generated with FLUX.I-dev and text encoder (TS-XXL) compressed by the full Skrr framework (dense, Skip only, and Skip with Re-use) are compared. When only Skip is used, some prompts may not be accurately reflected, leading to images that deviate from those generated by the dense model or even shift to animated style rather than a realistic one.
25

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
in Fig. A7, demonstrate that images generated by the compressed model using Skrr on multiple text encoders exhibit remarkable similarity those generated by the dense model. These findings indicate that compressing multiple text encoders simultaneously does not significantly compromise the model's is ability to generate high-quality, text-aligned images, thus validating the robustness of the compression approach.
Table A12. Quantitative ablation study for compressing multiple text encoders with Stable Diffussion The second row presents the evaluation results of the model where only the T5 text encoder is compressed, while the third row corresponds to the model in which all three text encoders TS, CLIP-I,, and CLIP--Gare compressed.
GenEval
Compressed
CLIP
DreamSim
Single
Two.
Count.
Colors
Pos.
Color attr.
Overall
Dense
0.318
1.0
0.994
0.869
0.600
0.856
0.280
0.538
0.689
T5 0.317
0.811
0,959
0.773
0.500
0.803
0.210
0.423
0,611
All
0.313
0.717
0.991
0.654
0.534
0.835
0.105
0.353
0.579
Skrr without projection module. To measure the discrepancy in Skrr, we tailored the evaluation for the T2I diffusion model by analyzing features extracted from the text encoder after projection through the projection module used in the denoising process. In this section, we present both quantitative and qualitative results to validate the effectiveness of incorporating the projection module. We evaluated the performance of the PixArt-E model compressed with Skrr, excluding the projection module. The results, summarized in Table A13, reveal that, while performance is largely preserved without the projection module, a slight degradation occurs due to loss of information. This degradation occurs because the omission of the projection module ignores crucial components that are influential in the image generation process.
Table A13. Quantitative results of the ablation study on the projection module. Applying the projection improves CLIP score, DreamSim, various GenEval tasks, demonstrating its significant impact on T21 generation performance when extracting features discrepancy measurement.
GenEval
Projection
CLIP
DreamSim
Single
Two.
Count.
Colors
Pos.
Color attr.
Overall
x
0.305
0.708
0.884
0,366
0.266
0.620
0.070
0.078
0.381
0.312
0.746
0.913
0.409
0.450
0.755
0.055
0.068
0.442
This trend is also reflected in the qualitative results shown in Fig. A8. The images generated with the projection module exhibit a higher degree of similarity to those produced by the dense model compared to the images generated without it. These findings underscore the importance of the projection module preserving key features necessary for accurate text-to-image generation, thereby proving its effectiveness maintaining model performance and fidelity.
Qualitative results of beam size In the main manuscript, we quantitatively analyzed the performance variation with respect to the beam size k. Here, we provide qualitative results to further illustrate its impact on image generation. As shown in Fig. A9, increasing the beam size leads to better alignment between the generated images and the dense model, demonstrating the effectiveness of proper beam sizes improving generation quality, but slight decline in performance as the beam size grows.
C.6. Additional quantitative results
Baseline with other metric. FinerCut employs cosine similarity and Sensen-Shannon Divergence (JSD) in addition the MSE used in our implementation. However, since the text encoder lacks language head, JSD is not applicable. Moreover, FinerCut's reported results indicate that MSE outperforms cosine similarity, leading us to adopt it in our implementation. To ensure fair comparison, we reimplemented FinerCut using cosine similarity and evaluated its performance using FID, CLIP, DreamSim, and GenEval, as presented in Table A14. Consistent with FinerCut's 's findings, cosine similarity yielded better performance compared to MSE. Consequently, we used the MSE-based FinerCut implementation as a baseline for an equitable comparison that shows higher performance.
26

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
T5
All
Dense
T5
All
"Two zebras mizle together in wooded area.
Two brown bears sitting on top of black and white checkered bed.
"A clock sitting on top of wooden desk."
"The old, adult elephant stands near wire fence.
"An individual pizza with cheese, pineapple and sauce."
"Three brown cowe grazing muddy grass field
Stable Diffusion 3
"A brown and white dog sitting black leather chair."
"The infamous Big Ben clock tower sitting under cloudy blue
sky.
"A herd sheep standing grass covered hillside."
"An adult polar bear swimming in the water."
R
couple of giroffe standing each other," "
"Twa brown swimming together the
"A shops table filled with apples oranges other fruits."
"This pizza square and has pepperont and mushrooms.
"A brown teddy hear sitting in the middle of road."
"A man in welsuit on surfboard in the ocean."
Figure A7. Qualitative results Skrr-compressed text encoders Stable Diffusion 3. We compare compressing only the Dense model's output and T5-XXL (T5) versus compressing all encoders (T5, CLIP-L, and CLIP-G), Skrr maintains high image fidelity and text- extt-image alignment across both cases.
27

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
No proj.
Proj.
Dense
No proj.
Proj.
"The airplane is taking off the runway at the airport."
"A mon wearing goggles and standing in the snow.""
"A female green top and laptop."
"A lit up counter fancy sinks "
"Tree is female tennis player that playing on the court."
"A person standing in fromt of a plate of food."
PixArt-2 2
"A little girl in public buthroom for kids."
"A batter standing in the batter's box waiting to hit the baseball.
using pair of pizza" "
"A balding with glasses, standing bridge.
"A fancy dish on a white plate at restaurant.
"A bunch of sheep are standing in a snowy field."
"A man performing skateboard ramp Junap."
"Two yellowish green sitting counter:
"A clock on side of building with skyline in background."
"A man holding a smart phone in his right hand.
Figure A8. Qualitative comparison of using projection layer versus no projection PixAnt-E, Without projection Proj.), text-imaat -mtaiimae alignment remains similar, but the generated image deviates from the dense model's ss output. With projection (Proj.). both text-image aligument and image fidelity closely match the dense model.
28

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
k=l
k=2
k=3
k=4
woman helping child put on skis."
"A traffic sign has been vandalized to look like scary face." "
PixArt-E
"An airport filled with planes sitting tarmacs.
"Tree female tennis player that playing the court."
"A suitcase that is on the ground with some shoes."
"People hold flynng kites in the air by the water."
Figure A9. Qualitative analysis of the impact of beam size image generation quality. As shown, inereasing the beam size enhances the alignment between generated images and the dense model, resulting in improved visual coherence and fidelity. Larger beam sizes allow for more exhaustive exploration of the search space, leading more refined and higher-quality outputs. These observations further support our quantitative findings presented in the main paper, demonstrating the effectiveness of using larger beam sizes.
29

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Table A14. Quantitative comparison of FinerCut similarity metries, including cosine similarity and MSF, evaluated using FID,, CLIII, DreamSim, and GenEval. The results confirm that MSE outperforms cosine similarity, which is consistent with the finding of ours.(t) denotes that a higher lower metric is favorable).
Sparsity
GenEval f
Method
FID 4
CLIP
DreamSim f
(%)
Single
Two
Count.
Colors
Pos.
Color attr.
Overall
Dense
0.0
22.89
0.314
1.0
0.988
0.616
0.475
0.795
0.108
0.255
0.539
26.3
19.64
0.311
0.745
0.925
0.394
0.363
0.657
0.060
0.085
0.414
FinerCut (cos.)
32.2
21.16
0,306
0.689
0,875
0.346
0,322
0,620
0.043
0.058
0.377
41.7
21.84
0.306
0.671
0.847
0.275
0.256
0.625
0.053
0.033
0.348
26.3
20.15
0.315
0.800
0.947
0.465
0.394
0.737
0.103
0.105
0.458
FinerCut (MSE)
32.2
20.19
0,313
0.775
0.903
0.409
0.344
0,697
0.078
0.128
0.426
41.7
19.93
0.312
0.741
0.841
0.306
0.306
0.628
0.050
0.073
0.367
27.0
20.15
0.315
0.800
0.956
0.434
0.425
0.763
0.095
0.145
0.471
Skrr (Ours)
32.4
20.19
0.313
0.775
0.928
0.397
0.413
0.774
0.100
0.118
0.455
41.9
19.93
0,312
0.741
0.913
0.410
0.450
0.755
0.055
0.068
0.442
Comparison with additional benchmark. In the main paper, we provided comprehensive comparison using metries such as FID, CLIP score, DreamSim, and GenEval. To further investigate the effectiveness of SKII, conducted additional experiments using benchmarks including 121-CompBench (Huang et al., 2023b) and DSG (Cho et al., 2024) scores, as presented in Table A15. T2I-CompBench consists of various subsets, where images are generated from task-specific prompts and evaluated using a vision- rision-language model. Since some subtasks in 121-CompBench overlap with those in GenEval, we focused on the Complex, Shape, and Texture subsets, which are not included in GenEval. The DSG benchmark enables more accurate and extensive performance evaluation by measuring T2I performance through graph- based representations. is composed of eight subsets. To assess the performance of Skrr, we evaluated it across all subsets on the DSG- Ik dataset. The results demonstrate that Skrr significantly preserves performance compared other text encoder compression baselines and are consistent with the quantitative comparisons presented in the main paper.
Comparison with additional base model. In the main script, we conducted a quantitative comparison using Pix one of the best open-sourced models. We extended this experiment to a larger model, FLUX. I-shnell,,, and the results are presented in Table A16. BLUX.I-shnell is derived from FLUX. by applying timestep distillation, enabling faster sampling and making more sensitive to the quality of the text encoder. Our results show that Skrr consistently outperforms other text encoder compression baselines in preserving performance, consistent with the observations made using PixArt-E..
Comparison with extreme sparsity. In the main paper, we evaluated performance across three sparsity levels, with the maximum sparsity set to the low 40% range. We extended these experiments to investigate how each baseline behaves under more extreme sparsity conditions, exceeding 50%. The results, presented in Table A17, show that while other baselines suffer from significant degradation in image fidelity sparsity, Skrr maintains high image quality and exhibits only a gradual performance drop. This further demonstrates the effectiveness of the Re-use phase in preserving generation quality even under severe compression.
higher FID and lower T2I cerformance-under extreme
C.7. Additional qualitative results
In this section, we present additional qualitative results to further illustrate the performance of our approach. We define the model compressed with 20%-30% sparsity as sparsity level 1, 30%-40% as sparsity level 2, and 40%-50% as sparsity level 3. The qualitative comparison for sparsity level is shown in Fig. A10, while comparisons for sparsity level 2 are presented in Fig. All and Fig. A12. Finally, Fig. A13 and Fig. A14 illustrate the comparisons for sparsity level 3. The results demonstrate that Skrr generates outputs that more closely aligned with the prompts and more consistent with the outputs of the dense model compared to other baselines. All presented results were generated using PixArt-E and are part of dataset comprising 30,000 images, which were produced for FID measurements.
Additionally, we provide a qualitative comparison of the text encoders compressed by each compression method across all sparsity levels, juxtaposed with the images generated by the dense model. By comparing images generated with the same seed at sparsity levels 1, 2, and 3, as defined above, we can assess the degree of deviation from the original image as the
30

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Table A15. Quantitative comparisons of Skrr with baselines with additional benchmark. We compared Skrr with the baselines of LaCo, and FinerCut under sparsity level on PixArt-E. We conducted evaluations on three subsets Complex (Comp.), Shape, and Texture from T2I CompBench, which are orthogonal GenEval. Additionally, performed experiments all sets from the DSG benchmark, including TIFA, Paragraph (Para.), Relation (Rel.), Count, Real user (Real.), Pose, Defying (Def.), and Text. (t denotes that higher
lower metric is favorable.)
Sparsity
T2I-CompBench
DSG-Ik
Method
(%)
Comp.
Shape
Texture
TIFA
Para.
Rel.
Count
Real.
Pose
Def.
Text
Overall
Dense
0.0
0.5137
0.4595
0.5715
0.879
0.886
0.836
0.740
0.600
0.626
0.808
0.683
0.761
LaCo
40.5
0.3127
0.3155
0,3370
0.763
0.812
0.630
0.602
0.518
0,594
0.595
0.582
0.649
FinerCut
41.7
0.4162
0.3146
0.4155
0.715
0.661
0.716
0.569
0.470
0.585
0.549
0.502
0.597
Skrr (Ours)
41.9
0.4438
0.3756
0.4721
0.797
0.791
0.757
0.636
0.537
0.553
0.691
0.605
0.667
Table A16. Quantitative comparisons of Skır with baselines with FLUX.I 1-schnell, evaluated using FID, CLIP, DreamSim, and GenEval. The results confirm that our method Skrr consistently outperforms other baselines even with state-of-the-art T2I diffusion models.(1 / denotes that higher/ lower metric is favorable).
Sparsity
GenEval t
Method
FID L
CLIP f
DreamSim
(%)
Single
Two
Count.
Colors
Pos.
Color attr.
Overall
Dense
0.0
20.45
0.312
1.0
0.994
0.879
0.603
0.738
0.280
0.500
0.666
LaCo
40.5
28.67
0.292
0.631
0.756
0.275
0.275
0.372
0.025
0.053
0.293
FinerCut
41.7
38.87
0.268
0.536
0,522
0.169
0.072
0.298
0.008
0.008
0.179
Skrr (Ours)
41.9
24.28
0.300
0.698
0.925
0.439
0.300
0.617
0.058
0.053
0.399
Table A17. Quantitative comparisons of Skır with baselines with PixArt- under extreme sparsity, evaluated using FID, CLIP, DreamSim. and GenEval. The results confirm that our method Skır consistently outperforms other baselines even with sparsity over 50%. / denotes that higher/ lower metric is favorable).
Sparsity
GenEval f
Method
FID 4
CLIP
DreamSim
(%)
Single
Two
Count.
Colors
Pos.
Color attr.
Overall
Dense
0.0
22.89
0.314
1.0
0.988
0.616
0.475
0.795
0.108
0.255
0.539
LaCo
48.6
280.3
0.194
0.170
0,003
0.0
0.0
0.0
0.0
0.0
0.001
FinerCut
51.3
154.7
0.191
0.176
0.078
0.008
0.0
0.027
0.0
0.0
0.019
Skr (Ours)
51.4
20.04
0.307
0.699
0.888
0.333
0.366
0.686
0.038
0.055
0.394
31

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
sparsity increases. These comparisons are illustrated in Fig. A15 and Fig. A16. The results reveal that, while the baseline methods occasionally generate images similar those from the dense model low sparsity, the differences become more pronounced sparsity levels increase. In contrast, the model compressed using Skrr consistently maintains high degree of similarity to the original images across all sparsity levels, demonstrating its robustness.
D. Limitations
In this section, we address the limitations of Skrr. While Skir effectively preserves Text-to-Image (T2I) performance during pruning, its performance deteriorates noticeably at extreme sparsity levels (> 60%). This degradation is likely due to significant reduction in the representational capacity of the text encoder as the number of parameters becomes excessively limited. However, such sparsity could still provide additional memory efficiency through complementary techniques such as weight quantization. Another limitation is that Skrr does not achieve performance improvements beyond that of the original dense model. While pruning the text encoder can enhance certain image quality metrics, such as improving FID scores, we observed consistent decline in performance on benchmarks like CLIP score and GenEval as sparsity increased. Finally, the Seam--earch-Bee other methods. Let L denote the number of transformer blocks and k the beam size, methods such as ShortGPT and LaCo have time complexity of (L), while FinerCut has a complexity of O(L²). SkIr exhibits complexity of O(kL2), but since & < T. in typical settings, the additional computational overhead remains relatively low. Addressing these limitations would be an interesting future work.
approach introduced in the skip phase of Skrr incurs additional computation compared to
32

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Texl-to-Image Generation
Dense
ShortGPT
LaCo
FinerCut
Skrr (Ours)
"A bronze statue with neck tie around its neck."
"A stuffed animal tied to something with chain on the ground."
"A man running towards kick ball a field." "
"A brown and white cow lying in green pasture."
"A person with a tie and a suit".
"A pizza sitting on top of plate on table."
Figure A 10. Qualitative comparison of images generated by models compressed to sparsity range 20%-300 (sparsity level 1) using ShonGPT, LaCo, FinerCut, Skrr, alongside images generated by dense models. Skır demonstrates remarkable ability to produce images closely resembling those from the dense model in the majority of cases.
33

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Dense
ShortGPT
LaCo
FinerCut
Skrr (Ours)
"A couple of doughnuts wrapped in paper."
"A group of people sitting at tables next to each other."
"A cat is lying on a white bedsheet."
A
"Two brown bears embracing each other on top of a dirt ground."
brown dog laying on a pillow on the the floor." "
jumbo jet airplane parked with a person checking it. "
Figure A11. Qualitative comparison of images generated by models compressed to sparsity range 30%-40% (sparsity level 2) using ShonGPT, LaCo, FinerCut, Skrr, alongside images generated by dense models. Skrr demonstrates remarkable ability to produce images closely resembling those from the dense model in the majority of cases.
34

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Dense
ShortGPT
LaCo
FinerCut
Skrr (Ours)
"A man holding a tennis racquet into tennis court."
"Elephants stand in large expanse of grass."
"A small kitten walking on computer keyboard."
"The two traffic lights placed pole the ground."
"A bowl with some noodles inside of it.
"A train that is a large rail way."
Figure A A12. Qualitative comparison of images generated by models compressed to sparsity range 30%-40% (sparsity level 2) using ShonGPT, LaCo, FinerCut, Skrr, alongside images generated by dense models. Skır demonstrates remarkable ability to produce images closely resembling those from the dense model in the majority of cases.
35

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Dense
ShortGPT
LaCo
FinerCut
Skrr (Ours)
"A zebra standing on top of dirt field."
"A little girl smiling widely for the camera" "
"A lot of blue and yellow umbrellas sitting under a clock."
"A black cat is standing small desk."
"A smiling man sitting in a fancy restaurant. "
"Two zebras nuzzle together in a wooded area."
Figure A 13. Qualitative comparison of images generated by models compressed to sparsity range 40%-500 (sparsity level 3) using ShonGPT, LaCo, FinerCut, Skrr, alongside images generated by dense models. Skır demonstrates remarkable ability to produce images closely resembling those from the dense model in the majority of cases.
36

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Dense
ShortGPT
LaCo
FinerCut
Skrr (Ours)
"a stuffed teddy bear in pink dress holding stuffed houquet of roses" "
"A bunch of sheep are standing in a snow field."
siow.f "
"A box of apples sits marketplace."
"A plate food that is a table."
"There is a snow boarder doing a trick in the air." "
"A man standing in front of a flat screen TV."
Figure A 14. Qualitative comparison of images generated by models compressed to sparsity range 40%-500 (sparsity level 3) using ShonGPT, LaCo, FinerCut, Skrr, alongside images generated by dense models. Skrr demonstrates remarkable ability to produce images closely resembling those from the dense model in the majority of cases.
37

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
ShortGPT
LaCo
FinerCut
Skrr (Ours)
Sparsity Lev.
Dense
Sparsity Lev. 2
Sparsity Lev.
"A cat is sitting on desk next to a computer."
ShortGPT
LaCo
FinerCut
Skrr (Ours)
Sparsity Lev. I
Dense
Sparsity Lev. 2
Sparsity Lev.
"A brown teddy bear sitting the middle of the road." "
Figure A15. Qualitative comparison of images generated by models compressed to various sparsity level using ShonGPT, LaCo, FinerCut,
and Skrr, alongside images generated by dense models. Skrr demonstrates remarkable ability to produce images closely resembling those from the dense model in all the sparsity levels. Notably, the lighting and composition of objects in the images retains even after harsh pruning the text encoder.
38

---

Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
ShortGPT
LaCo
FinerCut
Skrr (Ours)
Sparsity Lev.
Dense
Sparsity Lev. 2
Sparsity Lev. 3
"A domestic cat lounges on a quilt covering a bed." "
ShortGPT
LaCo
FinerCut
Skrr (Ours)
Sparsity Lev. I
Dense
Sparsity Lev. 2
Sparsity Lev. 3
"A cat laying on a laptop and looking like it has glowing eyes."
Figure A16. Qualitative comparison of images generated by models compressed to various sparsity level using ShonGPT, LaCo, FinerCut, and Skrr, alongside images generated by dense models. Skrr demonstrates remarkable ability to produce images closely resembling those from the dense model in all the sparsity levels. Notably, the lighting and composition of objects in the images retains even after harsh pruning the text encoder.
39


Reviews:
[{"review_id": "gTnnyuKvOp", "reviewer": "Reviewer_tVSB", "summary": "This paper introduces Skip and Re-use Layers (Skrr), a method for compressing text encoders in text-to-image (T2I) diffusion models to improve memory efficiency. The large-scale text encoders in T2I diffusion models consume a significant amount of memory despite contributing little to FLOPs. Skrr addresses this by selectively skipping and reusing layers in the text encoder. It uses a T2I diffusion-tailored discrepancy metric and beam search in the Skip phase to identify layers for pruning, and in the re-use phase, it reintroduces adjacent layers to skipped ones to mitigate performance degradation. Experiments show that Skrr outperforms existing blockwise pruning techniques, maintaining image quality and text alignment even at high sparsity levels.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful comments and encouraging feedback regarding our method's superior performance over existing blockwise pruning techniques, theoretical support, and extensive experiments across diverse metrics.\n\n---\n\n> **Question 1. Evaluation on additional benchmark.**\n\n**Response:** GenEval was selected in this work since it is a widely adopted benchmark among prior T2I works, thus can provide direct comparisons with our proposed method. To relieve your concern for additional evaluation, we now evaluate our Skrr on two more benchmarks (Tables 1 and 2 below), T2I-CompBench [1] (generating 4 images per prompt, thus 3,600 images total) and DSG-1K [2] (generating 4 images per prompt, thus 4,240 images total), in BLIP-VQA score and DSG score, respectively. These new results are consistent with our prior evaluation, demonstrating Skrr’s robustness across diverse and rigorous benchmarks.  \n\n**Table 1.** Quantitative results on BLIP-VQA score in T2I-CompBench dataset with PixArt-$\\Sigma$.\n|Method|Sparsity|Complex↑|Shape↑|Texture↑|\n|:-:|:-:|:-:|:-:|:-:|\n|Dense|0.0|0.5137|0.4595|0.5715|\n|LaCo|40.5|0.3127|0.3155|0.3370|\n|FinerCut|41.7|0.4162|0.3146|0.4155|\n|**Ours (Skrr)**|41.9|0.4438|0.3756|0.4721|\n\n**Table 2.** Quantitative results on DSG score in DSG-1k dataset with PixArt-$\\Sigma$.\n|Method| Sparsity | TIFA↑| Paragraph↑| Relation↑| Count↑| Real user↑| Pose↑| Defying↑| Text↑| Overall↑|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Dense|0.0|0.879|0.886|0.836|0.740|0.600|0.626|0.808|0.683|0.761|\n|LaCo|40.5|0.763|0.812|0.630|0.602|0.518|0.594|0.595|0.582|0.649|\n|FinerCut|41.7|0.715|0.661|0.716|0.569|0.470|0.585|0.549|0.502|0.597|\n|**Ours (Skrr)**|41.9|0.797|0.791|0.757|0.636|0.537|0.553|0.691|0.605|0.677|\n\n[1] Huang, et al. T2I-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. NeurIPS (2023).  \n[2] Cho, et al. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. ICLR (2024).\n\n---\n\n> **Question 2. Clarification on Skrr's practical applicability considering performance relative to dense models.**\n\n**Response:** As you may know, true practicality does not only needs excellent performance, but also requires efficient memory usage and we believe that our proposed Skrr can provide a way to controling this trade-off while minimizing performance degradation. As shown in all quantitative results, including Table 1 and Table 2 in question 1, Skrr consistently achieves better performance than all baseline pruning techniques, even at high sparsity. Furthermore, qualitative examples in both the main text (Fig. 5) and appendix (Fig. A10-A16) clearly show that Skrr maintains visual fidelity and semantic alignment with the dense model. These results underscore Skrr’s strong practical applicability despite compression. Additionally, Skrr is entirely training-free, which enhances its practicality and distinguishes it as a meaningful contribution to the field.\n\n---\n\n> **Question 3. Practical utility considering memory efficiency versus performance trade-off in text encoder pruning.**\n \n**Response:** We would like to clarify that our method is training-free, thus there is no claim regarding training; instead, our approach is explicitly designed for inference scenarios. In practical situations where it is challenging to anticipate incoming text prompts, offline feature extraction becomes impractical. Consequently, reducing the size of the text encoder is essential for effective model deployment. Reviewer v9A7 also acknowledged this motivation, explicitly stating, \"I agree with this claim as the size of the text encoder significantly affects the memory and size of the checkpoint in storage.\" Concerns regarding performance drops associated with memory-efficient designs have already been addressed above.\n\n---\n\n> **Question 4. Clarification on the representativeness and generalizability of the CC12M calibration dataset for text-to-image tasks.**\n\n**Response:** We would like to clarify that the CC12M dataset is widely recognized and extensively utilized for training and evaluating various T2I models, including Stable Diffusion 3. To address your concern about potential overfitting to our calibration dataset, we emphasize two key points: First, all evaluation metrics employed in our experiments—FID, CLIP, DreamSim, GenEval, T2I-CompBench, and DSG score—were computed on datasets entirely distinct and mutually exclusive from the calibration set. Consistent performance across these diverse benchmarks strongly suggests minimal risk of overfitting. Second, since all baseline methods evaluated in our experiments utilized the same calibration set for their respective pruning strategies, the fairness and comparability of our results remain intact.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. Skip Algorithm and Re-use Algorithm.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n- This paper proposes Skrr, which significantly reduces the memory usage of text encoders in T2I diffusion models.\n- It outperforms existing blockwise pruning techniques such as ShortGPT, LaCo, and FinerCut.\n- The paper provides theoretical support for the Re-use phase. Theorem 3.2 shows that incorporating the Re-use phase can lead to a tighter error bound compared to just skipping layers. \n- Extensive experiments across various metrics show the effectiveness of the proposed method. \n\nWeaknesses:\n- The Geneval score appears to be insufficient as a standalone metric. Are there alternative evaluation benchmarks, such as T2I-CompBench or DPG-Bench, that could provide a more comprehensive assessment of the proposed method?\n- Even when considering the Geneval score, a significant performance drop is observed compared to the dense model (where sparsity is 0), as shown in Table 1. This substantial decline severely limits the practical applicability of skrr in real-world scenarios.\n- Whether memory is a limiting factor in practice remains an open question. During training, it is feasible to extract features offline using the text encoder since backpropagation (BP) is not required. In such cases, optimizing memory usage may not be a critical concern. However, if the proposed algorithm sacrifices significant performance to achieve memory efficiency, its practical utility could be severely constrained, which still needs to be further discussed.\n- The authors state that the calibration dataset consists of 1k text prompts sourced from CC12M. However, it is unclear whether this dataset is representative of the text prompts used in text-to-image generation tasks. There is insufficient validation to ensure that the results derived from this calibration dataset generalize well to text-to-image tasks. For instance, the authors might have overfitted the calibration dataset when selecting re-used or skipped layers, which could lead to suboptimal layer choices for text-to-image generation. This issue requires thorough exploration and justification.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "Based on the above concerns, I suggest i) the authors should conduct more extensive validation of their method, including comparisons on additional datasets; ii) a rigorous analysis should be performed to verify the consistency between the calibration dataset and text-to-image datasets. Most importantly, the current method appears to sacrifice too much performance, which significantly limits its practical applicability. Addressing this performance degradation is critical for real-world deployment. Given these issues, I currently vote for weak reject.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PZQO4E3aN5", "reviewer": "Reviewer_hzEB", "summary": "This paper introduces Skrr, a pruning strategy for text encoders in text-to-image (T2I) diffusion models. Skrr reduces memory usage by selectively skipping and reusing transformer layers, leveraging the redundancy in T2I-specific text encoders. Experimental results on FID, CLIP, DreamSim, and GenEval are reported.\n\n## update after rebuttal\nThanks for the new evaluations. The Flux based results may be presented in the main text as well.\nThe field is changing rapidly -- I'm sure that the authors know the native image gen in gpt4o, which seems to be using special autoregressive tokens as conditioning for diffusion. Maybe in future versions some discussions on how this work fits into the 4o style structure will make the paper more interesting.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful comments and encouraging feedback.\n\n---\n\n> **Question 1. Novelty claiming on task.**\n\n**Response:** We acknowledge the reviewer's comment regarding the overstatement of novelty. We appreciate this important point and will carefully revise our manuscript to more accurately reflect the nature of our contribution. Specifically, we will clarify that our primary novelty lies in effectively addressing the particular challenges involved in pruning text encoders within text-to-image (T2I) diffusion models, rather than implying broader methodological novelty.\n\n---\n\n> **Question 2. Evaluation on additional benchmark.**\n\n**Response:** GenEval was selected in this work since it is a widely adopted benchmark among prior T2I works, thus can provide direct comparisons with our proposed method. To relieve your concern for additional evaluation, we now evaluate our Skrr on two more benchmarks (Tables 1 and 2 below), T2I-CompBench [1] (generating 4 images per prompt, thus 3,600 images total) and DSG-1K [2] (generating 4 images per prompt, thus 4,240 images total), in BLIP-VQA score and DSG score, respectively. These new results are consistent with our prior evaluation, demonstrating Skrr’s robustness across diverse and rigorous benchmarks.  \n**Table 1.** Quantitative results on BLIP-VQA score in T2I-CompBench dataset with PixArt-$\\Sigma$.\n|Method|Sparsity|Complex↑|Shape↑|Texture↑|\n|:-:|:-:|:-:|:-:|:-:|\n|Dense|0.0|0.5137|0.4595|0.5715|\n|LaCo|40.5|0.3127|0.3155|0.3370|\n|FinerCut|41.7|0.4162|0.3146|0.4155|\n|**Ours (Skrr)**|41.9|0.4438|0.3756|0.4721|\n\n**Table 2.** Quantitative results on DSG score in DSG-1k dataset with PixArt-$\\Sigma$.\n|Method| Sparsity | TIFA↑| Paragraph↑| Relation↑| Count↑| Real user↑| Pose↑| Defying↑| Text↑| Overall↑|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Dense|0.0|0.879|0.886|0.836|0.740|0.600|0.626|0.808|0.683|0.761|\n|LaCo|40.5|0.763|0.812|0.630|0.602|0.518|0.594|0.595|0.582|0.649|\n|FinerCut|41.7|0.715|0.661|0.716|0.569|0.470|0.585|0.549|0.502|0.597|\n|**Ours (Skrr)**|41.9|0.797|0.791|0.757|0.636|0.537|0.553|0.691|0.605|0.677|  \n\n[1] Huang, et al. T2I-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. NeurIPS (2023).  \n[2] Cho, et al. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. ICLR (2024).\n\n---\n\n> **Question 3. About Lipschitz continuity assumption.**\n\n**Response:** The assumption of Lipschitz continuity is widely used in deriving convergence rates and error bounds. Prior work has empirically shown that reasonably trained large transformer models satisfy this condition [3], and theoretical analyses have confirmed its validity in diffusion models [4]. Thus, our use of this assumption is well-justified, supporting the relevance of our theoretical framework to real-world T2I models.  \n[3] Khromov & Singh. Some Fundamental Aspects about Lipschitz Continuity of Neural Networks. ICLR (2024).  \n[4] Liang, et al. Unraveling the smoothness properties of diffusion models: A gaussian mixture perspective. arXiv (2024).\n\n---\n\n> **Question 4. Evaluation on FLUX.**\n\n**Response:** We provide extensive additional qualitative comparisons for stronger T2I models, such as SD3 and FLUX, in the appendix. Nevertheless, to directly address your concern regarding generalizability, we have also conducted quantitative evaluations on the FLUX.1-schnell model, and these results are presented in Table 3 below. The outcomes clearly demonstrate that our method preserves the performance of dense models in FLUX compares to other baselines, reinforcing our claim regarding the broad applicability and generalizability of our pruning strategy across multiple strong T2I diffusion models.\n\n**Table 3.** Quantitative results on FLUX.1-schnell.\n|Method|Sparsity|FID↓|CLIP↑|DreamSim↑|Single↑|Two↑|Count.↑|Colors↑|Pos.↑|Color attr.↑|Overall↑|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Dense|0.0|20.45|0.312|1.0|0.994|0.879|0.603|0.738|0.280|0.500|0.666|\n|LaCo|40.5|28.67|0.292|0.631|0.756|0.275|0.275|0.372|0.025|0.053|0.293|\n|FinerCut|41.7|38.87|0.268|0.536|0.522|0.169|0.072|0.298|0.008|0.008|0.179|\n|**Ours (Skrr)**|41.9|24.28|0.300|0.698|0.925|0.439|0.300|0.617|0.058|0.053|0.399|\n\n---\n\n> **Question 5. Relationship between Figure 3 and T2I models.**\n\n**Response:** Figure 3 presents experimental results specifically obtained from the T5-XXL encoder, which is widely adopted in T2I models. Note that T5-XXL is independently trained from T2I models, so it is reasonable to investigate this component separately while still having a strong relevance to the T2I context. To avoid potential confusion, we will revise the text to explicitly highlight the connection to T2I.", "claims_and_evidence": "On the Claim of Novelty:\nThe paper states, “To the best of our knowledge, this is the first work to tackle the challenge of constructing a lightweight text encoder for T2I tasks.” While technically correct, this phrasing may be misleading. Many text encoder pruning methods can be considered as contributing to lightweight text encoders. The unique aspect of this work lies in how it prunes the text encoder specifically for T2I tasks. However, the specific challenges in optimizing text encoders for T2I models are not clearly articulated, making it difficult to assess the true novelty of the approach.", "methods_and_evaluation_criteria": "The paper uses GenEval as a benchmark for evaluating text encoder performance. However, GenEval primarily consists of simple prompts that do not fully test the capabilities of powerful text encoders. More challenging benchmarks, such as the DSG score (Davidsonian Scene Graph: Improving Reliability in Fine-Grained Evaluation for Text-to-Image Generation, ICLR 2024), would provide a better assessment of how well the pruned text encoder captures fine-grained text-image alignment.", "theoretical_claims": "The paper presents Lemma 3.1 (Error Bound of Two Transformers) and Theorem 3.2 (Tighter Error Bound of Re-use) to support its pruning strategy. However, these theorems involve Lipschitz continuity, which the paper does not explicitly evaluate on real models. Without verifying the Lipschitz properties in practical settings, the theoretical claims may appear somewhat loose, and their applicability to actual T2I models remains uncertain.", "experimental_designs_or_analyses": "The primary baseline for evaluation is PixArt (in Tab 1), which is known to be a relatively weak T2I model. Furthermore, the paper tests its method on only this single model. As the proposed pruning strategy is intended to be a general solution for text encoder optimization, it should be validated on stronger T2I models—especially those that demonstrate superior text generation capabilities, such as Flux. Additionally, because PixArt’s performance on GenEval is highly variable, the reported results may lack reliability. Testing on multiple high-performing models would strengthen the claims of generalizability.", "relation_to_broader_scientific_literature": "not related", "essential_references_not_discussed": "n/a", "other_strengths_and_weaknesses": "no other comments", "other_comments_or_suggestions": "no other comments", "questions_for_authors": "1. Flux is tested, but why number evals are not reported?\n2. Fig 3 seems like a general text encoder study. Does it related to t2i? This is misleading.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3kymPiw8FR", "reviewer": "Reviewer_Prsj", "summary": "This paper propose skrr method, which can effectively reduce the memory consumption of the text encoder in the Text to Image (T2I) model while maintaining the quality of image generation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful comments and encouraging feedback regarding the clarity of our proposed two-stage pruning method (Skip and Re-use), as well as our comprehensive evaluation using multiple widely recognized metrics and state-of-the-art diffusion models.\n\n---\n\n> **Question 1. Computational cost of beam search-based algorithm.**\n\n**Response:** Thank you for the valuable suggestion. As you correctly pointed out, beam search introduces additional computational overhead, and we have clearly articulated its time complexity with respect to the number of transformer blocks $L$ and beam size $k$ in Table 1. However, since $k$ is set to a relatively small value (specifically, $k=3$ in Skrr) compared to $L$ (with $L=48$ for T5-XXL), the additional computational overhead remains minimal. Furthermore, it is crucial to emphasize that this overhead occurs only once during the pruning stage when blocks are selected and does not recur during the actual image synthesis phase. Thus, the computational cost from beam search has negligible practical impact on the applicability and efficiency of Skrr. We will include a detailed analysis addressing this aspect explicitly in our revised manuscript.\n\n**Table 1.** Time complexity of each block-wise pruning method for calibration. For Skrr, $k=3$.\n|ShortGPT|LaCo|FinerCut|Ours (Skrr)|\n|:-:|:-:|:-:|:-:|\n|$O(L)$|$O(L)$|$O(L^2)$|$O(kL^2)$|\n\n---\n\n> **Question 2. Clarification on statistical significance and stability of FID improvements observed under text encoder pruning.**\n\n**Response:** We emphasize that our evaluation of FID scores was conducted using the MS-COCO dataset with 30,000 prompts, a sufficiently large and widely used benchmark to ensure robustness and statistical reliability. The use of such an extensive dataset substantially reduces the likelihood that the observed improvements in FID scores at high sparsity are due to randomness or evaluation bias. Additionally, it should be noted that the calibration subset derived from CC12M and the MS-COCO dataset employed for FID evaluations are entirely disjoint, further diminishing concerns regarding dataset-induced bias.\n\nWe believed that it is worth reporting and discussing (at least in the Discussion section) this interesting phenomenon of decreasing FIDs since this was also observed in many different contexts such as fine-tuning for concept erasing [1–3], distillation [4–5], quantization [6], or compression [7–8], despite an intuitive expectation of potential performance degradation. A deeper exploration of this intriguing phenomenon and its underlying mechanisms would constitute compelling future work.\n\n[1] Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR (2024).  \n[2] Zhang, et al. Forget-me-not: Learning to forget in text-to-image diffusion models. CVPR (2024).  \n[3] Lee, et al. Concept pinpoint eraser for text-to-image diffusion models via residual attention gate. ICLR (2025).  \n[4] Zhao, et al. MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices. In ECCV (2024).  \n[5] Feng, et al. Relational diffusion distillation for efficient image generation. ACM MM (2025).  \n[6] Li, et al. SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models. ICLR (2025).  \n[7] Yuan, et al. Ditfastattn: Attention compression for diffusion transformer models. NeruIPS (2024).  \n[8] Chen, et al. $\\Delta$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. arXiv (2024).", "claims_and_evidence": "The motivation is reasonable.", "methods_and_evaluation_criteria": "The paper proposes a two-stage pruning method (Skip and Re use) , which is clear. However, Beam search method may introduce additional computational overhead, and the author did not quantitatively analyze the efficiency and cost of the pruning process itself.\nThe authors use multiple widely recognized evaluation metrics  and multiple existing state-of-the-art diffusion models as evaluation benchmarks.", "theoretical_claims": "It seems correct.", "experimental_designs_or_analyses": "When pruning with high sparsity, the FID score (image quality index) actually improves (i.e., the image quality improves). The author briefly discusses this in the Discussion section (from line 433 onwards), suggesting that this may be due to the disturbance of the null condition vector. However, without sufficient and clear quantitative experiments or theoretical support for this phenomenon, the improvement of FID may only be an accidental phenomenon or evaluation bias. More detailed experimental analysis should be conducted on this phenomenon (such as evaluating the stability of FID scores under different random seed or prompt distributions) to confirm whether FID improvement is truly statistically significant.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "see above", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JCX7x6g8Ha", "reviewer": "Reviewer_v9A7", "summary": "This paper suggests reducing the parameter in the Text-to-Image (T2I) generative models by pruning the transformer layers in the text encoder for conditioning. They highlight that most of the parameters in T2I models are available in the text encoder. Thus, they propose Skip and re-use layers (Skrr) that prune the layers in the text encoder with beam search and re-use the non-pruned layers as if they were pruned layers. In terms of similarity metric for pruning, authors use MSE as they observe cosine similarity leads to divergence of the feature norm of the unconditional embedding. Quantitative and qualitative experiments the improvement of generation quality compared to other LLM pruning methods, and ablations studies show the significance of re-use and beam search.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful comments and encouraging feedback regarding our motivation for pruning text encoders, our embedding-space-based similarity approach, the justification for using MSE over cosine similarity, and the clarity and validity of our experimental design and evaluations.\n\n---\n\n> **Question 1. Clarification on pruned model's unconditional score effectiveness and impact on FID when guiding original conditional scores.**\n\n**Response:** We believed that it is worth reporting and discussing (at least in the Discussion section) this interesting phenomenon of decreasing FIDs since this was also observed in many different contexts such as fine-tuning for concept erasing [1–3], distillation [4–5], quantization [6], or compression [7–8], despite an intuitive expectation of potential performance degradation. Furthermore, as you suggested, our experimental results, summarized in Table 1, indicate that guiding conditional embeddings from the Skrr-compressed model using unconditional embeddings from the dense model leads to degraded FID performance compared to employing both conditional and unconditional embeddings entirely from the Skrr-compressed model. A deeper exploration of this intriguing phenomenon and its underlying mechanisms would constitute compelling future work.\n\n**Table 1.** FID and CLIP score with combinations of conditional / unconditional scores from dense / compressed models in PixArt-$\\Sigma$.\n|Conditional|Unconditional|FID↓|CLIP↑|\n|:-:|:-:|:-:|:-:|\n|Dense|Dense|22.89|0.314|\n|Compress w/ Skrr|Compress w/ Skrr|19.93|0.312|\n|Compress w/ Skrr|Dense|24.31|0.307|\n\n[1] Lu, et al. Mace: Mass concept erasure in diffusion models. CVPR (2024).  \n[2] Zhang, et al. Forget-me-not: Learning to forget in text-to-image diffusion models. CVPR (2024).  \n[3] Lee, et al. Concept pinpoint eraser for text-to-image diffusion models via residual attention gate. ICLR (2025).  \n[4] Zhao, et al. MobileDiffusion: Instant Text-to-Image Generation on Mobile Devices. In ECCV (2024).  \n[5] Feng, et al. Relational diffusion distillation for efficient image generation. ACM MM (2025).  \n[6] Li, et al. SVDQuant: Absorbing Outliers by Low-Rank Component for 4-Bit Diffusion Models. ICLR (2025).  \n[7] Yuan, et al. Ditfastattn: Attention compression for diffusion transformer models. NeruIPS (2024).  \n[8] Chen, et al. $\\Delta$-DiT: A Training-Free Acceleration Method Tailored for Diffusion Transformers. arXiv (2024).\n\n---\n\n> **Question 2. Typo error.**\n\n**Response:** Your understanding is correct, and the point you mentioned is indeed a typo. Thank you for pointing this out. The values of $\\text{Metric}_1$ for (b) and \\(c\\) in Figure 4 should be swapped.\n\n---\n\n> **Question 3. Experiments on extreme sparsity.**\n\n**Response:** Thank you for proposing this insightful experiment. We conducted evaluations under high sparsity levels beyond 50%, and these results are presented in Table 2. While other pruning methods produce images with severely compromised fidelity under such extreme sparsity, Skrr, despite exhibiting some degree of performance degradation, continues to generate images with relatively high fidelity that remain reasonably aligned with the textual descriptions. We will include these detailed experimental results in the supplementary materials of our revised manuscript.  \n**Table 2.** Quantitative results on sparsity over 50% with PixArt-$\\Sigma$.  \n|Method| Sparsity | FID↓  | CLIP↑ | DreamSim↑ |Single↑|Two↑|Count.↑|Colors↑|Pos.↑|Color attr.↑|Overall↑|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Dense|0.0|22.89|0.314|1.0|0.988|0.616|0.475|0.795|0.108|0.255|0.539|\n|LaCo|48.6|280.3|0.194|0.170|0.003|0.0|0.0|0.0|0.0|0.0|0.001|\n|FinerCut|51.3|154.7|0.191|0.176|0.078|0.008|0.0|0.027|0.0|0.0|0.019|\n|**Ours (Skrr)**|51.4|20.04|0.307|0.699|0.888|0.333|0.366|0.686|0.038|0.055|0.394|\n\n---\n\n> **Question 4. FLOPs reduction in Skrr.**\n\n**Response:** Skrr not only reduces memory and parameters but also lowers FLOPs compared to the dense model, as shown in Table 2 of the main manuscript. While the Re-use mechanism slightly increase FLOPs relative to methods using pruning alone, the text encoder constitutes a minor fraction of the total T2I pipeline. As a result, Skrr reduces overall FLOPs by 0.04% compared to the dense model, with at most a 0.17% increase over other pruning approaches.", "claims_and_evidence": "- They claim that most of the parameters in the latest text-to-image generative models are the text encoder (e.g. T5-XXL encoder in SD3), so they need to focus on reducing the number of parameters of the text encoder to achieve a memory-efficient T2I generative model. I agree with this claim as the size of the text encoder significantly affects the memory and size of the checkpoint in storage.", "methods_and_evaluation_criteria": "- Pruning of the proposed method computes the similarity in embedding space of T2I generative model. It makes sense as this space would have the most information T2I model uses.\n\n- Authors justify that the usage of MSE as similarity by showing the problem of cosine similarity.\n\n- Authors show the similarity between features in adjacent layers, and it strengthens their argument about the re-use of non-pruned layers.\n\n- Evaluating the T2I generation on MS-COCO is a common metric as far as I know, and they show various evaluation metrics about T2I performance.", "theoretical_claims": "- It seems that their claim makes sense, but I didn't check the theoretical proof thoroughly.", "experimental_designs_or_analyses": "- Most of the experimental designs seem clear and valid. \n\n- In discussion, the authors discuss the enhancement in FID of the pruned model and relate it to recent guidance methods using perturbed or degraded models. Does this imply that the unconditional score in the pruned model would be a good guidance like the degraded counterpart in autoguidance? Then, what if we compute conditional score from the pruned model and guide it with the original model? Would it worsen the FID?\n\n- As I understand, the lower metric score is better for both metric_1 and metric_2, since it implies the pruned model works alike the original model. In such case, for every metric, (c) is better than (b) in Figure. 4. However, text in Figure. 4 claims that metric_2 is better than metric_1 and it makes some confusion for me.", "relation_to_broader_scientific_literature": "- Compressing the large-scale text-to-image generative model is a significant problem for its usage in the real-world scenario. This paper claims the importance of considering text encoder for memory-efficient T2I generative models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "One weakness of this paper is that it only reduces the memory and parameters, not actual computing cost such as FLOPS.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- The maximum sparsity the paper mentioned is 41.9, and I wonder if the authors have tested with a more highly compressed scenario.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes techniques to reduce the memory cost of language encoders in text2image models. The motivation is that language encoders account for most parameters of the model and thus reducing it would be desirable from a memory saving perspective. Experiments are conducted on PixArt, and demonstrated superior performance than competitive pruning methods. In the rebuttal, the authors show additional results on Flux which supports the same claims. Reviewers are overall positive, recognizing that it's a simple and effective method. Though concerns still remain, for example, the proposed method does cause significant drop in performance and also it could be better motivated in terms how the method is specific for text2image settings instead of LLMs in a general sense. Overall, I think this paper has sound ideas and results in a small but valid scope, and I lean towards accepting it.
Decision: Accept (poster)