Cost-Aware Routing for Efficient Text-To-Image Generation
Authors: Qinchan Li
Kenneth Chen
Changyue Su
Wittawat Jitkrittum
Qi Sun
Patsorn Sangkloy
Venue: Accepted by TMLR ()

Abstract: Diffusion models are well known for their ability to generate a high-fidelity image for an input
prompt through an iterative denoising process. Unfortunately, the high fidelity also comes at
a high computational cost due to the inherently sequential generative process. In this work,
we seek to optimally balance quality and computational cost, and propose a framework to
allow the amount of computation to vary for each prompt, depending on its complexity. Each
prompt is automatically routed to the most appropriate text-to-image generation function,
which may correspond to a distinct number of denoising steps of a diffusion model, or a
disparate, independent text-to-image model. Unlike uniform cost reduction techniques (e.g.,
distillation, model quantization), our approach achieves the optimal trade-off by learning to
reserve expensive choices (e.g., 100+ denoising steps) only for a few complex prompts, and
employ more economical choices (e.g., small distilled model) for less sophisticated prompts.
We empirically demonstrate on COCO and DiffusionDB that by learning to route to nine
already-trained text-to-image models, our approach is able to deliver an average quality
that is higher than that achievable by any of these models alone.

Published in Transactions on Machine Learning Research (03/2026)
Cost-Aware Routing for Efficient Text- Text-To-lmage Generation
Qinchan (Wing) Li
g18400myu edu kc4 906 amgu. edu es ym edu willawall @rraill...mmmmmmmmmmmm com gisun Kryu edu patsorn. mobsom..angkkooy@@mma.... com
Kenneth Chen
Changyue (Tina) Su
Wittawat Jitkrittum 1M
Qi Sun
Patsorn Sangkloy
Tandon School of Engineering, New York University Google, New York
Reviewed on OpenReview: https: // openreview. forum? id= id=Jbe9AVs
Abstract
Diffusion models are well known for their ability to generate a high-fidelity image for an input prompt through an ilerative denoising process. Unfortunately, the high fidelity also comes at high computational cost due the inherently sequential generative process. this work, we seek optimally balance quality and computational cost, and propose framework to allow the amount of computation to vary for each prompt, depending on its complexity. Each prompt is automatically routed to the most appropriate text-to-image generation function, which may correspond to a distinct number of denoising steps of a diffusion model, or disparate, independent text-to-image model. Unlike uniform cost reduction techniques (e.g., distillation, model quantization), our approach achieves the optimal trade-off by learning to reserve expensive choices (o.g., 100+ denoising steps) only for few complex prompts, and employ more economical choices (e.g., small distilled model) for less sophisticated prompts. We empirically demonstrate on COCO and DiffusionDB that by learning to route to nine already-trainee text-to-image models, approach is able to deliver average quality that is higher than that achievable by any of these models alone. Code is available at https: com cowp//ingglccppp///Toaao..o
1 Introduction
While diffiusion models have set a new standard for photorcalism in generative art (Ho et al., 2020), their operational costs remain a major challenge. The generation of a single image can involve many denoising steps, each utilizes a learned denoiser model with potentially over billion parameters (Rombach al., 2022). This makes in-the-wild adoption (i.e., on-device) challenging and raises valid concerns about their environmental sustainability (Sarah Wells, 2023; Kate Crawford, 2024: Kaack et al., 2022). To address this, significant body of research has explored optimization strategies such as network simplification (Li et al., 2024; 2023a) and model distillation (Sauer et al., 2024; Salimans & Ho, 2022; Meng et al., 2023; Liu et al., 2023).
However, these existing methods typically apply the same degree of optimization irrespective of the task's intrinsic difficulty. This results in single model with fixed computational cost, which is inherently suboptimal as the generative effort required to synthesize prompt. For example, simple prompt like white and empty woll requires fewer denoising steps to generate high-quality image than a complex one like colorful park with crowd, as shown in Figure 1.
image varies with the complexity of the input
"Now Figen 4D Inc.

---

Published in Transactions on Machine Learning Research (03/2026)
0.28
1
CLiPScore
0.25
whilte
" colorful
50
100
Denoising Steps
(a) 'a while and emply wall'
(b) 'a colorful park with a crowd'
(c) quality trends across numbers of steps
Two input prompts that require different denoising steps to ensure quality. As shown in Figure 1c, prompt Figure only requires a small number of denoising steps to reach a high CLIPScore. By contrast, the more complex prompt Figure 1b requires over 100 steps to reach a similar quality. Key to our proposed CATImage is to allocate appropriate amount of computation for each prompt, so that the overall computational cost is reduced while the quality remains the same.
Figure 1:
With the motivation to adaptively allocate computational budget, we present CATImage, framework that allows the amount of computation for text-to-ing to-image generation to vary for each prompt. Our framework operates with a pre-defined set of choices that can be chosen adaptively for each input prompt. Each choice represents text-to-image gencration function and has a distinct profile of computational cost and the expected image quality. Concretely, these choices may correspond to different numbers of denoising steps of the same diffusion model (i.e., homogeneous choices), disparate, independent text-to-ing text-to-image generative models (i.c., heterogencous choicos), or a combination of both. The proposed CATImage aims to adaptively select the right choice (i.e., "routing") for each input prompt, in such a way that expensive choices (e.g., 100+ denoising steps) are reserved only for complex prompts. Our approach enables a joint deployment of diverse text-to-image models and has a potential to deliver higher average image quality compared to using any individual model in the pool, while allowing the average computational cost to be adapted at deployment time.
In summary, our contributions are follows.
1. We precisely formulate a constrained optimization problem for the above routing problem (Section 3.1). The formulation aims to maximize average image quality subject to budget constraint the generation cost.
2. We study the theoretically optimal routing rule that optimally trades off the average quality and cost. (Section 3.2). Based on the optimal rule, we construct plug-in estimator that can be trained from data.
3. We perform a series of objective analyses on the COCO (Lin et al., 2014) and DiffusionDB datasets (Wang et al., 2022b). Our findings show that, through adaptive routing, our proposal matches the quality of the largest model in the serving pool (namely, Stable Diffusion XI. from Radford et al. (2021) with 100 denoising steps) with only fraction of its computational cost. (Table 3).-
2 Background: Text- To-l Generative Models
Let x E X denote an input text prompt, and i E [o, 1] 1| denote an image described by the prompt, where W, H E N denote the width and the height of the image (in pixels), and the last. dimension denotes the number of color channels. text-to-image generative model is a stochastic map h: + J that takes a prompt x as input and generates an image h(x) e J that fits the description the prompt x. There are
We will release the code and data upon paper publication.
2

---

Published in Transactions on Machine Learning Research (03/2026)
Our pipeline. Dur- ing training (dashed box), qual- ity estimator is trained to prodict per-prompt quality scores for all routing candidates ((M),, At inference time (bottom), given a prompt, predicted quality scores of all routing candidates are adjusted by their respective costs. The rout- ing candidate that has the highest cost- cost-adjusted score is chosen (see Eq. (3))-
Figure 2:
1.00
A C0
A
oat looking intevtly something as duck floats by."
v(x, Nimn (x))
Text Prompt
Quality Estimator
C
Generred Image
Praditedd Quality
Tnining
qualtty
A km))
km)
many model classes one may use to construct such a model h, including conditional Generative Adversarial Networks (GANs) (Zhang el al., 2017; Goodfellow al., 2014), Variational Auto-Eucoder (VAE) (Kingma & Welling, 2022), and diffusion models (Ho al., 2020), among others.
Diffusion models A specific class of text-to-image generative models that has recently been shown to produce high-fidelity images is given by dilfusion--eesee based models (Saharia et al., 2022; Ho et al., 2020; 2022). A diffusion generative model relies on a function g: X x N RD - + that takes as input a prompt x, the number of denoising steps E N, a noise vector z E R with D = 3 S. and generates an image i = 9xx, T,2). z). Image generation is done by iteratively refining the initial noise vector z for T iterations to produce the final image. The noise vector z ~ (o, typically sampled from the standard multivariate normal distribution and the T' refinement steps correspond to the reverse diffusion process, which reconstructs an image from a random initial state (Ho et al., 2020). With z ~ (O,, I) understood to be an implicit source of randomness, we define (x) = g(x, T, 2) to be an image sampled from the diffusion model using T diffusion steps. With T chosen, hr: X + J is thus an instance of text-to-image generative models as described carlier. The importance of this view will be apparent when we describe our proposed method in Section 3, which enables an automatic selection of the number of denoising steps, separately for each prompt. Typically, the number denoising steps is pre-chosen according the computational budget available inference time, with low value of T, giving a lower computational cost at the expense of image quality.
3 Cost-Aware To-gaee Generation
We now describe our main proposal termed CATimage (Cost-Aware Text-based Image Generation), which seeks to minimize inference cost by adaptively adjusting the cost per prompt, depending on its complexity. As illustrated in Figure 1, in the case of a diffusion model, our key observation that not all prompts require a large number of denoising steps to ensure quality. Thus, inference efficiency can be achieved by spending a small amount of computation for casy prompts. Our proposed framework is general and allows cost adjustment in per-prompt manner via selecting an appropriate amount of resources from homogeneous choices (i.e., adaptively varying the number of denoising steps of single diffusion model), or heterogeneous choices (i.c., adaptively route prompts to disparate, independent generative models).
We start by formalizing the cost-aware Lexi-lo-image generation lask as learning terning-to-route to-route problem in Section 3.1. The formulation can be theoretically shown (Section 3.2) to have a simple Bayes optimal routing rule, involving subtracting off the expected quality metrics with the costs of candidate numbers of denoising steps. We show that the optimal rule can be estimated from data, and propose two estimators: a brasselrner-based estimator (Vaswani et al., 2017), and a K-nearest neighbors (KNN) model.
3

---

Published in Transactions on Machine Learning Research (03/2026)
3.1 Problem Formulation
Let [n] = 21, of .M choices DC = denole the set of counting numbers up to nt. Suppose that are given a fixed set h((²} where each choice h();; X + represents a trained generative model (sce Section 2 for a procise definition). Our goal is to derive a routing rule that optimally (in the sense of quality-cost trade-offs) chooses the best model to invoke for each input prompt. These M base models may be homogeneous, being derived from a single diffusion model with varying numbers of diffusion steps, a mix of heterogeneous generative model classes, or a combination of both. For example, if we want to decide whether to use 20 or 50 number of denoising steps in the Stable Diffusion XL (SDXL) model (Podell et al., 2023), then M = 2, and H = (h(), }(2) where the two models are both SDXL, with the number of denoising steps fixed to 20 and 50, respectively. We will abstract away the details of the underlying M base models and propose general framework that supports both the homogeneous and heterogeneous cases (as shown in our experiments in Section 5).
Suppose we are given a quality metric of interest q: x -> R (sec Quality Metrics under Section 5.1), which takes as input a prompt-image tuple, and estimates a quality score. We seek router r: X - [M] that predicts the index of the M choices from a given prompt. We posit two desirable properties that the router ought to possess:
1. The router must respect a specified budget constraint on the inference cost.
2. Routing prompts to candidates in H must maximize average quality metric.
Following similar formulations considered in Jitkrillum et al. (2023; 2025); Mao et al. (2023); Tailor et al. (2024), the above desiderata may be realized as constrained optimization problem:
max Q(r) subject to C(r) < B,
where
(1)
Q E
Σ ME(M]
1 (x(x) = m] g(xx h (m)) (x)) (x)) and C(r) E
Σ M
1 [r(x) = m] (mm)
(2)
where for E [M], cfm) 2 0 denotes the cost for the model ((m)) to produce one image for given prompt, E denotes the expectation with respect to the population joint distribution all random variables (i.e. prompt x, and the sampled output of h(m)),, B 2 0 is hyperparameter specifying an upper bound on the average cost. The optimization problem (1) thus seeks a router r that maximizes the average quality Q(r) subject to the constraint that the average cost (over all prompts) is bounded above by B.
Remark. The optimization problem is general and allows the per-model costs to be in any unit suitable for the application (e.g., latency in seconds, FLOP counts). Further, no practical constraint is imposed on the quality metric function For instance, 9 could be the CLIP score (Radford et al., 2021). Intuitively, the budget is large, the cost constraint C(r) < would have little effect, and the optimal router is expected to route each prompt to the base model that can produce the highest quality metric score, disregarding the cost. of the model. In practice, such a model is often the largest one in the pool H, or the diffusion model with the largest number of denoising steps. On the contrary, if B is small, the router would prioritize cost over quality, preferring to choose a small base model (or a small number of denoising steps) over a. larger candidate. This proposal offers a framework to allow trading off average quality with cost in a unified way by varying B.
3.2 Theoretically Optimal Routing Rule
Having formulated the constrained problem in (1), we now investigate its theoretically optimal solution. We will use the optimal solution to guide us on how to design a practical router. Based on the results in Jitkrittum et al. (2023; 2025), the optimal solution to (1) is shown in Proposition
Proposition 1. For a cost budget B > O, the optimal router r²: X (l,..., M} to the constrained optimization problem (1)
r" r"(x) arg max E (1m) (x)) x > (m)
mE(M]
4

---

Published in Transactions on Machine Learning Research (03/2026)
where the conditional expectation is over the sampled output from the model h!"), and X 2 0 is Lagrange multippier inversely proportional to B.
The result follows from Proposition in Jitkrittum et al. (2025). The result states that the choice/ /mooel we choose to route a prompt x to is the one that maximizes the average quality, adjustod additively by the cost of the model. The hyperparameter controls the trade-off between quality and cost, and inversely proportional to the budget B. For instance, if À (corresponding to B oo), then the model with the highest expected quality for will be chosen, regardless of its cost. Increasing À enforces the routing rule to account more for model costs, in addition to the expected quality.
Estimating the Optimal Rule The optimal rule r" in Proposition 1 depends on the population conditional expectation (m) (x) = E [1(x, (x)) x], which is unknown. Following a similar reasoning as in Jitkrittum et al. (2025), we propose plugging in empirical estimator g(m); X + R in place of ?(m), resulting in the empirical rule îx:
(x) = arg max (mm) (x) A c(().
(3)
mE[M]
For each m E [M]; the idea is to train an estimator 5(m) to estimate the true expected quality. That is, suppose we are given a collection of N training prompts {x For each prompt x,, we may sample S (m)
times from (mm)) to produce output images 1,1 ..., ..., These output images allow one to estimate the
empirical expectation of the quality ú; = -x= ((x, 1()) (m)). With the labeled training set {(x; 9:) we may then procoed to train a predictive model fxx = (x).....,
which has M output heads for predicting the expected qualities of the M models. There are several standard machine learning models one can use the model class for 7.
We emphasize that we do not advocate specific model class part of our proposal since different model classes offer distinct properties on training and inference costs, which may be best tailored to the application. What we propose is an application of the generic routing rule in (3) to text-to-image model routing. The rule is guaranteed to give a good quality-cost. trade-off provided that the estimator 5 g/()) well estimates (((m) In experiments (Section 5), we demonstrate estimating (m)) with two model classes: 1) K-nearest neighbors, and 2) Multi-Layer Perceptron (MLP) with Transformer backbone (Vaswani et al., 2017). Likewise, we do not propose advocate specific value of X. The parameter is left to the user knob to control the desired degree of quality-cost, trade-ofT. In experiments, evaluate our proposed routing rule by considering a wide range of À and show the trade-off as a deferral curve (see Section 3.3). An illustration summarizing our pipcline is displayed in Figure 2.
3.3 Deferral Curve
In general, any methods that offer the ability to trade off quality and cost may be evaluated via a deferral curve (Bolukbasi et al., 2017; Cortes et al., 2016; Gupta et al., 2024; Narasimhan et al., 2022). A deferral curve is a curve showing the average quality against the average cost, in a quality-cost two-dimensions plane. Specifically, for our proposed routing rule fx in (3), the curve preciscly given by C = {(C(f)), ((fx)) | X E [0, oo)} where Q and C denote the average quality and cost, and are defined in Eq. (2). In practice, the population expectation in Q and C is replaced with empirical expectation over examples in test set. We generate the deferral curve of our routing decision by sweeping À from 0 to sufficiently large value. This allows the router to transition from maximizing quality at X = 0 to selecting the cheaper candidates as À increases. More generally, one evaluates the deferral curve of a method by computing its average quality and cost as we vary parameters that control the trade-off. For instance, for the SDXL diffusion model, we may produce a deferral curve by varying the number of denoising steps.
4 Related Work
Uniform Optimization Strategies for Diffusion Models Diffusion models have recently exploded in popularity due to their high performance on tasks such as image and video generation, audio generation,
5

---

Published in Transactions on Machine Learning Research (03/2026)
and 3D shape generation (Ho et al., 2020; Ramesh et al., 2021). Latent diffusion models (Rombach al., 2022) have significantly improved training and inference efficiency, but still require a large number of forward demoising neural network evaluations to produce high-quality results. To address this, an extensive body of literature has been proposed to optimize and accelcrate diffusion models, which are typically applied uniformly across all prompls. For example, optimizing the sampling stralegy may enable more efficient. denoising computation (Li et al., 2024; Chen et al., 2023b; Li et al., 2023a); such as timestep integration (Nichol & Dhariwal, 2021) or conditioning on the denoising (Preechakul et al., 2022). Optimizing solvers for the denoising step can also efficiently reduce the computation to avoid re-training or fine-tuning (Song et al., 2020; Lu et al., 2022; Liu et al., 2022; Karras et al., 2022). Alternatively, reducing the redundant. computations by caching the internal results within the denoising network is also explored in (Ma et al., 2024a;b). Another common approach includes model-based optimizations, such as distilling a fully trained model into a smaller student model that achieves comparable results with fewer denoising steps (Sauer et al., 2024; Salimans & Ho, 2022; Mong et al., 2023; Liu et al., 2023) or combining multiple denoising models with different sizes to accelerate the denoising process (Yang et al., 2023; Li el al., 2023b; Pan el al., 2024). An alternative strategy is to approximate the direct mapping from initial noise to generated images, further reducing the number of denoising steps (Luo et al., 2023; Song et al., 2023).
Adaptive Optimization Strategies for Diffusion Models Instead of fixed reduction in computational resources, AdaDiff (Tang et al., 2023) explores a more dynamic approach where the number of denoising steps is decided based on the uncertainty estimation of the intermediate results during denoising. Our work shares a similar motivation for flexible resource allocation. However, we adaptively allocate resources according to prompt complexity and thus can select the most suitable number of steps or model before any denoising process. Concurrently, AdaDiff (Zhang et al., 2023) tackles optimal number of steps selection using prompt-specific policy, with weight network trained on reward function that balances image quality and computational resources. In contrast, we decouple the quality estimation from the routing decision, which allows our framework to adapt to different resource constraints without any retraining.
and Modeling Routing The idea of adaptively invoking a different expert on each input is a widely studiod area in machine learning under the topic of learning to defer. Here, cach export may be a human expert. (Mozannar & Sontag, 2020; Mozannar et al., 2023; Sangalli et al., 2023), or a larger model (Narasimhan et al., 2022; Jitkrittum et al., 2023; Mao et al., 2023; Gupta et al., 2024). In the latter, depending on the topology or order the models are invoked, a learning-to- to-defer method may yield a cascade if models are arranged in a chain (Wang et al., 2022a; Jitkrittum et al., 2023; Kolawole et al., 2024); or yield a routed model there is a central routing logic (i.e., the router) which selectively sends input traffic to appropriate models (Jiang et al., 2023; Mao et al., 2023; Gupta et al., 2024; Jitkrittum et al., 2025). The latter setup is also known as model rouling and receives much attention of late, especially in the natural language processing literature. Model routing has been successfully applied to route between many Large Language Model (LLMs) of various sizes and specialties (see Chen al. (2023a); Hu et al. (2024); Zhuang et al. (2025); Ong et al. (2025); Jitkrittun et al. (2025) and references therein). To our knowledge, our work is one of the first that connects the model routing problem to efficient text-to-image generation.
5 Experiments
In this section, we show how our proposed routing method (Section 3) can be realized in practice by evaluating its effectiveness on real data. We experiment with both homogeneous (i.e., all routing candidates are derived from the same diffusion model with different candidate numbers of denoising steps), and heterogencous settings (i.e,, the routing candidates also include different generative models). Our goal is to optimally select the best model (or number of denoising steps) for each input prompt given a specified cost constraint.
5.1 Experimental Setup
Text-To-Image Generative Models As defined in Scction 3.1, our method selocts from a set of generative models H. for each input prompt. We consider a diverse range of models with varying configuratioss, each offering different trade-off between image quality and computational cost:
6

---

Published in Transactions on Machine Learning Research (03/2026)
1. SDXL: widely-used SD architecture (Rombach et al., 2022). To see the full extent of the achievable trade-off, we consider representative numbers of denoising steps in a wide range between and 100.
2. TURBO (Sauer et al., 2024) and LIGHTNING (Lin et al., 2024): distilled versions of SDXL for faster generation. We use the SDXL variant with step for Turbo, and steps for Lighting.
3. DDIM (Song et al., 2020): a non-Markovian diffusion process allowing faster sampling. We use this sampling strategy on the SDXL variant at 50 steps.
4. DEEPCACHE (Ma et al., 2024b): caching method that reduces redundant computation in SDXL. We use the official implementation released from Ma et al. (2024b), and set the cache interval parameter to 3.
5. INFINITY (Han et al., 2024): non-diffusion, auto-regrossive text-to-image model based on the Transformer encoder-decoder. We use the pre-trained Infinity-2B variant with visual vocabulary size of 232.
Quality Metrics The effectiveness of generative models largely depends on the criteria used to evaluate their output. Our proposed method can adaptively identify the optimal allocation of generative model for any instanco-level image quality metric. As there no consensus on the optimal metric for evaluating image quality, we explore several widely-used metrics: CLIPScore (Radford et al., 2021) for text-image semantic alignment, ImageReward (Xu et al. 2023) with reward model tuned to human preferences, and Aesthetic Score (Beaumont & Schuhmann, 2022) trained on human aesthetic ratings from LAION (Schuhmann et al., 2022). Additionally, we also introduce Sharpness metric adapted from Paris et al. (2011), defined as, [Sharp (x, Σ.g (1.3 [
where @ denotes the convolution operator, is is the pixel intensity at location (i, and G is a Gaussian kernel with standard deviation of 1. Intuitively, this metric measures the relative distance between the given image i and itself after a Gaussian blur filter is applied.
E.,
Quality Estimator One of the key components of our routing method is the quality estimator which estimates the expected quality of the m-th model given an imput prompt (see 8TH in Eq. (3)). We explore two model classes: K-NEAREST NEIGHBORS (K-NN) model and TRANSFORMER--basd model. Both of these models incur a negligible inference cost: less than 0.001 TFLOPs compared to 1.5 TFLOPs of the smallest base model in the pool (Infinity).
The K-NN approach provides non-parametric way to estimate quality by averaging the quality scores of K nearest training prompts in the space of CLIP embeddings (Radford al., 2021). This method is simple, and can generalize well with sufficient data. The Transformer model takes input the per-token embeddings produced by the frozen CLIP text encoder. A two-layer MLP with M output heads is added to each output token embedding. Pooling across all tokens gives M output scores (x,)))))),,,,,)))) estimating the expected quality of the m-th model on prompt x (see Appendix Section D for details).
(sce Eq. (3)), cach
All base models except Infinity already use CLIP embeddings, making router overhead negligible. Infinity uses Flan-T T5 embeddings (= 13 GFLOPs overhead), but this cost minimal compared to one SDXL call (+= 200 TFLOPs for 17 steps).
We train a separate model for each of the quality metrics considered. In each case, the quality scores are linearly scaled across all training examples to be in [0, 1]. These scaled metrics are treated as ground-truth probabilities, and the model is trained by minimizing the sum of the sigmoid cross-entropy losses across all heads.
5.2 Dataset Details
We utilize two datasets: 1) the COCO captioning datasct (Lin et al., 2014), which contains high-quality and detailed image captioning, and 2) the DiffusionDB dataset (Wang et al., 2022b), which contains a larger collection of realistic, user-generated text prompts for text-to-image generation. From both datasets, we sub-sample prompts by retaining only those with pairwise CLIP similarity below 0.75, resulting in diverse set of 18,384 prompts in COCO dataset, and 97,841 prompts on the DiffusionDB dataset. We split each dataset independently into 80% for training, 10% for validation, and 10% for testing. We then generate
7

---

Published in Transactions on Machine Learning Research (03/2026)
0.2835
0.115
SDXL
SDXL REF (100 steps)
0.110
Turbo step)
CLIPScore
0.2625
Sharpness
Lighting (4 steps)
0.2620
0.105
DDIM (50 steps)
DeepCache (interval 3)
0.2815
0.100
Infinity (VAR)
0.2810
Lighting
Proposed Transformer (SDXL+)
Proposed Transformer (SDXL)
0.2805
Infinity
0.095
Proposed K-N (SDXL+)
0,2800
200
400
600 TPlops
800
1000
1200
0.090
0
200
400
600
800
1000
1200
Proposed K-NN (SDXL)
TFlops
(a) CLiPScore
(b) Sharpness
Figure 3: Deferral curves of our proposed methods and baselines on the COCO dataset. Each data point represents the average quality and cost across the entire test set (1839 prompts). Quality is measured by CLIPScore in (a) and pixel sharpness in (b), as defined in Section 5.1. Our Proposed Transformer (SDXL+), which selects from all available SDXL denoising steps and baseline models, offers the best quality-cost trade-off (TFLOPs). Baselines that are not visible are shown the bottom-right corner the format of (cost, CLIPScore).
images from those prompts using all the base text-to-image models as described earlier. For SDXL, we generate images with various numbers of denoising steps ranging from 100. The costs terms of FLOPPs from these candidates cover the full range of costs of all other baselines.
For each model, we generate four images per prompt (i..., S = 4 in Section 3.2) using different random seeds, with fixed seed across different numbers of steps for SDXT. The generated images for each prompt allow us to compute the average quality metric, which is then used as the training label ii (as described in Section 3.2). Unless otherwise specified, we use the widely used Euler Scheduler (Karras et al., 2022) for diffusion- based image generation.
5.3 Experiments on COCO dataset
We present experimental results on subset of COCO's test set (Lin et al., 2014) consisting of 1.8k image- caption pairs Figure 3. We evaluate the deferral curves (sce Section 3.3) of our proposed method and all the baselines. The results are shown Figures 3a and 3b for the two different. quality metrics: CLIPScore and image sharpness (Section 5.1), respectively. The deferral curves plot average quality against average cost measured in TFLOPs (Tera Floating Point Opcrations). Baselines that do not support dynamic quality-cost trade-off are shown isolated dots in the same quality-cost plane; these baselines for image generation for each input prompt. For instance, each point * of SDXL, represents the performance of the SDXL model with the number of denoising steps fixed. For our proposed methods, Proposed (SDXL) refers to the homogeneous configuration in which the model candidate set H( consists solely of the SDXL model at multiple numbers of denoising steps settings. Proposed (SDXL++) extends this configuration by incorporating other text-to-image models considered, namcly, TURBO, DDIM, DEEPCACHE, and INFINITY. Each of these has two variants based on Transformer or K-NN the model class for estimating the expected quality metric.
the same compute cost
Homogencous vs. Heterogeneous setting In both settings, our methods outperform basclines with static inference costs per prompl. The heterogeneous setting further benefits from models with strong quality-to- cost trade-offs (e.g., INFINITY, TURBO), improving our dynamic routing's effectiveness and cost-efficiency. Morcover, our strategy remains adaptive, scamlessly allocating prompts higher-performance models when additional computational resources are available, improving performance beyond what is attainable using each model alone (see Appendix Section G for details on model selection rates).
8

---

Published in Transactions on Machine Learning Research (03/2026)
Table 1: We train the k-NN quality estimator over T = 100 trials on COCO. In each trial, we generate S images for each training prompt with 1) S and 2) S 3. Each of these two cases results in T k-NN models, and hence T (random) deferral curves evaluated on the same test set as used in Fig 3. We report the mean performance (CLIPScore) across the T trials for approach.
CLIPScore
Infinity
Turbo
Lighting
SDXL-9
DeepCache
SDXL-22
DDIM
SDXL-65
Ours (S=1)
0.2816
0.2814
0.2828
0.2828
0.2828
0.2828
0.2824
0.2819
Ours (S=3)
0.2816
0.2816
0.2831
0.2832
0.2832
0.2832
0.2828
0,2819
Fixed
0.2816
0.2794
0.2798
0.2773
0.2749
0.2805
0.2820
0.2819
Win Rate (S=1)
0.98
Win Rate (S=3)
1
1
TFLOPs
1.5
1.54
23.92
107.73
210
263.34
598.5
778.05
Transformer vs. KNN Between the two proposed variants, the Transformer-based variant generally outperforms the K-NN variant, suggesting that directly learning to predict the quality metric can be more effective than estimating it from neighboring prompts.
5.4 Effect of Sample Size S
As the estimation error depends on the randomness in image generation, we aggregate signals from multiple generated images by averaging their quality metric values across multiple random seeds. To further quantify variability across trials, we vary the number of generated images for each prompt and train separate model for each case, In Table 1, we report the mean performance of CLIPScores on the COCO datasel, as well as the Win Rate, defined as the fraction of trials that our router has higher average quality across the test set than the baseline. The maximum standard deviation across trials both above variants is less than 2 x 10-4 throughout the cost range.
The results show that using S=3 improves the test performance compared to S=1. Additionally, the win rate of "Ours (S=1)" compared to the baseline is 100% in almost all cost ranges. This win rate implies that our approaches are statistically significantly better than the baseline according to the sign test at significance level a < 10-6 This means using one image per prompt is already sufficient to improve the baseline using fixed compute costs. Deviation across trials is minimal relative to the mean metric, suggesting that S-3 is sulficient. In all other experiments in the paper, we used S-4.
5.5 Experiments on DiffusionDB dataset
In this section, we present results on a subset of prompts from the DiffusionDB dataset (Wang et al., 2022b), which aligns more closely with real-world prompts used in text-to-image generation. We evaluate the performance across four metrics: CLIPScore, ImageReward, Aesthetic Score, and Sharpness.
Quantitative results comparing our dynamic routing method to the fixed-model baselines are summarized Table 2. This table effectively captures the trade-offs shown in the deferral curves at specific cost equal to each baseline. We use KNN as quality estimator to efficiently evaluate multiple metrics at scale. For reference, also provide the Oracle performance, which selects the optimal candidate for each prompt based ground-truth quality scores rather than predicted estimates. This is an absolute upper bound on the best quality-cost trade-off altainable by any routing methods. In practice, it is extremely challenging to realize a quality-cost operating point that is close to that of the Oracle (see Hu et al. (2024)). The results show that our method consistently matches or exceeds fixed-model bascline performance across all four quality metrics. Additionally, the highest value of each score (highlighted in Table 2 in bold) is attainable only with routing strategy. In other words, even under unconstraineddan computational budget, none of
9

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Quality-cost trade-off of our proposed approach on DiffusionDDB (Section 5.5). We report the average quality achieved by our routing approach when operating at the cost (TFLOPs) of each model in the pool. For each metric, the highest score achieved between our approach and the fixed baseline is highlighted in bold. The Oracle performance is provided in gray for context.
CLIPScore (Radford al.,
2021)
Oracle
0.259 = Go-4
0.309
do-4
0.323 + 4e-4
0.334 +: 40-4
0.337 +: 4o-4
0.337 + 4o-4
0.336 = 6o-4
0.336
= 4o-4
0.318 4o-4
Ours
0.250 = 6e-4
0.304
de-4
0.308 4--4
0.314 + 4
0.316 + 4e-4
0.317 + 4e-4
0.318 4i-4
0.318
4m-1
0.318 4n-4
Fixed
0.250 = 6i-4
0.304
de-4
0.300 4--d
0.308 4a-4
0 + te-4
0.315 + te-d
0.317
6--1
0.315 So-1
0.318 In-4
Sharpness (Section 5.1)
Oracle
0.131 4e-4
0.140 3
0.144 1
0.145
0.146 3:344
0.146 3:34-4
0.145
0.145 4o
0.126
Ouns
0.131
4o-4
0.135 4e-4
0.135 30-4
0.136 36-4
0.187 3:34
0.137 3:30-4
0.137 3o-4
0.137
2o-4
0.126
Fixed
0.131 4o-4
0.122
0.110 3o-4
0.103 20-4
0.101 20-4
0.114
0.123 2o-4
0.107 3o-4
0.126 3e-4
Aesthetic Score (Beaumont & Schuhmenn 2022)
Oracle
6.824 = 8mx3
7.132 at 8m-3
7.385 8m-3
7.426 +: 7w33
7,421 4: 8i-3
7,416 at 6i-8
7,284 = Be-3
7.284 = 8m-3
6.707 =: 6m-3
Ours
6.824 = Bo-3
6.913 = 9e-3
7.042 9c-3
7.032 + 9--B
7.012 a 9e-3
7.012 + 0e-3
6.935 = 9e-3
6.935 = 9e-3
6.707 8e-3
Fixed
6.8244 = Be-3
6.780
a 9w-3
7.010 + 9u-3
6.285 + 8o-3
6.625 + 9w-3
6.588 + 8o-3
6.690 8e-3
6.600 = 9w-3
6.707 6o-3
Teage-Reward (Xu et al., 2023)
f
Oracle
1.029 Beis
1.303
1,416 55-3
1.446 seia
1.446 a
1.444 1
1.376 5ei3
1.370 bei3
0.891
Ours
1.029 9e-3
1.083
1.086
1.086 se-a
1.079 8
1.076 1
1.037 Bei3
1.037 Be-3
0.891
Fixed
1.029
90.3
0.960
0.932 1
0.497 se-a
0.809 ne-3
0.769 8o-3
0.866 8o-3
0.861 9e.3
0.891
INFI 1.50
TURB
LIGH 23.92
SDXL 119.70
DEEP 210.00
SDXL 239.40
SDXL 598.50
DDIM 598.50
SDXL 1197.00
TFLOPs
1.54
the individual baselines can attain the quality that our adaptive routing achieves through prompt-based allocation across the model pool.
Table 3 quantifies the computational cost reduction achieved by our routing method compared to the baseline at equivalent quality levels (on Sharpness metric). For inherently efficient models (c.g. Infinity(Han et al., 2024), Turbo (Sauer et al., 2024)), the savings appear marginal. However, com- pared to Lighting (Lin et al., 2024), a distilled SDXL variant, our method achicves the same performance at only 6% of its computational cost. For figher-performante models, such as SDXL at 100 denoising steps, the savings are even more significant.
Table 3: Cost ratio (%) of our method compared to baselines to match the quality score (Sharp- ness)
Model
Our cost
Infinity
100%
Turbo
97.40%
5.6 Incorporating FLUX.1-dev model
Lighting
6.27%
DeepCache
0.71%
The quality-cost trade-off of our router improves as add better models to the pool. With the same experimental setup on DiflusionDB as in Section 5.5, we now add a high-performing model FLUX.1-dev' at three different denoising steps (1, 15, and 30), resulting in a total of 12 models in the routing pool. The addition of a new model results in an improved performance across all cost ranges compared to using any individual model alone, as shown Table 4. Specifically, at an average cost of less than 600 TFLOPs, our router achieves an acsthetic score of 7.109, surpassing the best single model (FLUX-30) which scores 6.98 al a fixed cost of 1785.6 TFLOPs. The router learns that the most powerful model (e.g., FLUX-30) is not necessarily the best for every input; cheaper model can sometimes yield superior results on prompts for which it is better suited.
DDIM
0.25%
SDXL100
0.13%
5.7 Qualitative Comparison
To better illustrate the effects of routing, we further present qualitative comparisons between our routing results and the FLUX model (with 30 denoising steps) in Figure 4. We perform routing using Aesthetic
2httpe://hnugiingface. co/black- -abesttlabb/FLL...... 1-dev
10

---

Published in Transactions on Machine Learning Research (03/2026)
Table 4: Average quality achieved by our routing approach when operating at the cost (TFLOPs) of each model in the pool. In this experiment, we include FLUX.1-dev at 1, 15, and 30 denoising steps, for the total of 12 models in the candidate pool.
Aesthetic Score 1
Infinity
Turbo Lighting
FLUX-1
SDXL-10
DeepCache SDXL-20
DDIM
SDXL-50
FLUX-15
SDXL100 FLUX-30
Ours Fixed
6.824
6.913
7.045
7.055
7.068
7.082
7.0S7
7.109
7.109
7.108
7.080
6.983
6.821
6.780
7,010
3,160
6,285
6.625
6.588
6,000
6,600
6.952
6.707
6.983
TFLOPs
1.50
1.54
23.92
59.52
119.70
210
239.4
598.5
598.5
892.8
1197.00 00
1785.60
Fraction of TFLOPs
0.1%
0.1%
1.3%
3.3%
6.7%
11.8%
13.4%
33.5%
33.5%
50.0%
67.0%
100.0%
Score, which has been shown to correlate well with human preference (Section A). Note that in this case, our routing framework can be used to select the number of denoising steps (i.e. the same FLUX model with 15 denoising steps) or to select an entirely different model (Infinity, Turbo, elc.). Interestingly, we observe that the router tends to select Infinity for scenic prompts and Turbo for stylized (cartoon (cartoon/painting) /painting) images, whereas for gencral/human hhmann related prompts it typically reduces the number of denoising steps from 30 to 15.
6 Conclusion and Future Work
In this paper, present CATImage, cost-aware routing approach that dynamically selects optimal models and numbers of denoising steps based on prompt complexity. We show that incorporating multiple base models, such as distilled versions of diffusion models and alternative architectures, improves the quality cost trade-off. Extensive experiments on COCO and DiffusionDB datasets across multiple quality metrics validate our method's effectiveness and generalization capability. Nevertheless, several limitations are worth highlighting. To determine optimal routing decisions, CATImage relics on estimating the expected quality per prompt, which excludes metrics such Fréchet Inception Distance (FID) (Heusel et al., 2017) that measure statistical similarity across the entire image distribution. Additionally, the routing problem consider assumes static pool setup where the set of text-to-image models is considered fixed. In practice, the model pool can change which can add additional maintenance cost of retraining the router. Addressing these limitations remains a direction for future research.
References
Romain Beaumont and Christoph Schuhmann. LATON-Aestheticc Predictor V1. https://github. com/ 2022. GitHub repository, commit 6d122ad (15 Aug 2022); accessed 15
May 2025.
Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik. Make it count: Text-to-image generation with accurate number of objects. ar preprint arXiv:2466 10210, 2024.
Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive neural networks for efficient inference. Tn Proceedings of the 34th International Conference on Machine Learning Volume 70, ICMT/17, pp. 527-536. JMLR.org, 2017.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance, 2023a. URL
wrg/ 05176
Yu-Hui Chen, Raman Sarokin, Juhyun Lee, Jiuqiang Tang, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann. Speed is all you need: On-device acceleration of large diffusion models via gpu-aware optimizations, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4650-4654, 2023b.
Corinna Cortes, Giulia DeSalvo, and Mchryar Mohri. Learning with rejection. In Algorithmic Learning Theory: 27th International Conference, ALT 2016, Bari, Italy, October 19-21, 2016, Proceedings 27, pp. 67-82. Springer, 2016.
11

---

Published in Transactions on Machine Learning Research (03/2026)
epic painting a futuristic soliary astronaut walking along the icy surface of Titan, Saturn visible through the haze, unreal 5, DAZ, detailed, soft locus, brilliant, 4k, 8k, HD, trending on artstation, art by Rick Guidice painting by Robert McCall by John Harris, abstract
Prompt:
Prompt: Looking towards The Ettenmoors from The Mountains Weathertop seen distance the horizon, Eriador, Middle Earth, Tolkien, Albert Bierstadt and Henri Mauperch
Angmar.
Prompt: awe inspiring bruce pennington landscape, orange sky, cyan forest d gital art painting of s, japan at night, hyperdetailed, minimalist
Router's (Infinity) Save 99%
Fixed Cost (FLUX-30))
Router's Choice (Infinity) Save 99%
Fixed Cost (FLUX-30))
Router's Choice
Cost (FLUX-30)
(Infinity) Save 99%
Prompt: tabby cat wearing clothes designed by Issey Miyake painted Alex Katz
Prompt:
excited brown and tan domino finnish lapphund, studio ghibli art style
Prompt: portrait of elton john lennon thoughtful in 1970 by ilya repin
Router's Choice (Turbo) Save 99%
Fixed Cost (FLUX-30))
Router's Choice (Turbo) Save 99%
Fixed Cost (FLUX-30))
Router's Choice
Fixed Cost (FLUX-30))
(Turbo)
Save 99%
Prompt: cinematic scene with mckenna grace as eowyn from forc of the rings, live action film, batle armor chain mail, dramatic, small details, volumetric lighting, frame
Prompt: 1930's photo of cat rootop
photograph of Kim Kardashian cutting a banana, with pen, in tent
Prompt:
in japan
Router's Choice (FLUX-15) Save 50%
Fixed Cost (FLUX-30))
Router's Choice (FLUX-15)
Fixed Cost (FLUX-30))
Router's Choice (FLUX-15) Save 50%
Fixed Cost (FLUX-30))
50%
Figure 4: Example results from routing framework compared to fixed FLUX model (30 denoising steps). Examples are grouped by the three most frequently selected models: Infinity, Turbo, and FLUX-15. We observe a lendency for the router to select Infinity for scenic prompts, while Turbo often preferred for cartoon and painting styles. FLUX-15 (FLUX at 15 steps) appears to be favored for general prompts, especially those involving humans.
Ian Goodfellow, Jean Abadie, Mehdi Mirza, Bing Xu, David Warde- Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advonces in neural information processing systems, 27, 2014.
Neha. Gupta, Harikrishna Narasimhan, Wittawat Jitkrittum, Ankit Singh Rawat, Aditya Krishna Menon, and Sanjiv Kumar. Language model cascades: Token-level uncertainty and beyond. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrevieet net wwtt/forrm
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan, Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis, 2024. URL https:
12

---

Published in Transactions on Machine Learning Research (03/2026)
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Newral Information Processing Systems (NeurIPS), volume 30, pp. 6626-6637, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 116840-68511, 2020.
Jonathan Ho, wai Saharia, William Chan, David Fleet, Mohammad Norousi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23 (47):1-33, 2022.
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay, Routerbench: A benchmark for multi-LLM, routing system. In Agentic Markets Workshop at ICML 2024, 2024 URL attps://openreviii wttp:/oorom/mdddd///aaaat
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023.
Wittawat Jitkrittum, Neha Gupta, Aditya Krishna Menon, Harikrishna Narasimhan, Ankit Singh Rawat, and Sanjiv Kumar. When does confidence-basee cascade deferral suffice? In Thirty-seventh Conference on Newral Information Processing Systems, 2023. URL httpss//openrewwee
Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawai, Jeevesh Juneja, Zifeng Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal model routing for efficient IIm inference, 2025. URL https://arxiv... http//abb222200/7///
Lynn H Kaack, Priya L Donti, Emma Strubell, George Kamiya, Felix Creutzig, and David Rolnick. Aligning artificial intelligence with climate change mitigation. Nature Climate Change, 12(6):518 527, 2022.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Newral Information Processing Systems, 35:26565-26577, 2022.
Kate Crawford. Generative ai's environmental costs are soaring and mostly secret. Nature World View, 2024.
Diederik P Kingmna and Max Welling. Auto-encoding variational bayes, 2022. URL https://arxiv.org/ abs/1312.6114
Steven Kolawole, Don Dennis, Ameet Talwalkar, and Virginia Smith. Agreement-based cascading for efficient. inference, 2024. URL htap:s://arxv....
Lijiang Li, Huixia Li, Xiawu Zheng, Jie Wu, Xuefong Xiao, Rui Wang, Min Zheng, Xin Pan, Fei Chao, and Rongrong Ji. Autodiffusion: Training-Tree optimization of time steps and architectures for automated diffusion model acceleration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7105-7114, 2023a.
Wenhao Li, Xiu Su, Shan You, Tao Huang, Fei Wang, Chen Qian, and Chang Xu. Not all steps are equal: elficient generation with progressive diffusion models. arXiv preprint ar arXiv:2312. 13307, 2023b.
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devicos within two seconds. Advances in Neural Information Processing Systems, 36, 2024.
Shanchuan Lin, Anran Wang, and Xiao Yang. ligltnigh Progressive adversarial diffusion distillation. Xio, abs/2402.1322 13929, 2024. URL https://api. semanticscholar org/Corpus :
13

---

Published in Transactions on Machine Learning Research (03/2026)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision -CC-- 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740 755. Springer, 2014.
Luping Liu, Yi Ren, Zhijic Lin, and Zhou Zhao. Pscudo numerical methods for diffusion models on manifolds. arXiv preprinl or arXiv:2202.097 2022.
Xingchao Liu, Xiwen Zhang, Jianzhu Ma, Jian Peng, et al. Instaflow: One step is enough for high- quality diffusion-based lext-to-image generation. In The Twelfth International Conference on Learning Representations, 2023.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Newral Information Processing Systems, 35:5775 5787, 2022.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXi::2310.04978, 2023.
Xinyin Ma, Gongfan Fang, Michael Bi Mi, and Xinchao Wang. Learning-to-cache: Accelerating diffusion transformer via layer caching. Advances in Neural Information Processing Systems, 37:133282 133304, 2024a.
Xinyin Ma, Gongfan Fang, and Xinchao Wang. Deepcache: Accelerating diffusion models for free. In The IEEE/CVF Conference on Computer Vision and Pattern Recognation, 2024b.
Anqi Mao, Christopher Mohri, Mchryar Mohri, and Yutao Zhong. Two-stage learning to defer with multiple experts. In Thirly-seventh Conference on Neural Information Processing Systems, 2023. URL https: //openreview.net wtt.:/eeeettttoomtddddddattteeee
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diederik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14297-14306, 2023.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International conference on machine learning, pp. 7076 7087. PMLR, 2020.
Hussein Mozannar, Hunter Lang, Dennis Wei, Prasanna Sattigeri, Subhro Das, and David Sontag. Who should predict? exact algorithms for learning defer to humans. In International conference on artificial intelligence and statistics, pp. 10520-10545 PMLR, 2023.
Harikrishna Narasimhan, Wittawat. Jitkrittum, Aditya Krishna Menon, Ankit Singh Rawai, and Sanjiv Kumar. Post-hoc estimators for learning to defer to an expert. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
https://openreeveer sattforummid-- _d___66ff66.77777.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162 8171. PMLR, 2021.
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. RouteLLM: Learning to route LLMs from preference data. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openrevierr wttp:/ewwweeeto//iiteeeeee 8eSqNintMMr.
Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, and Tali Dekel. Teaching clip count to ten. arXiv preprint arxvo:2302. 2tl:2300.12066, 2023.
Zizheng Pan, Bohan Zhuang, De-An Huang, Weili Nie, Zhiding Yu, Chaowei Xiao, Jianfei Cai, and Anima Anandkumar. T-stitch: Accelerating sampling in pre-trained diffusion models with trajectory stitching. Xix preprint Xrx:::2402.1 14167, 2024.
14

---

Published in Transactions on Machine Learning Research (03/2026)
Sylvain Paris, Samuel W Hasinoff, and Jan Kautz. Local laplacian filters: Edge-aware image processing with a laplacian pyramid. ACM Trans. Graph., 30(4):68, 2011.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. URL
httpp:://rxxvv.rgg wrwpora///aa///////a////// 2307.0192..
Konpal Preechakul, Nattanat Chalthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion autoencoders: Toward meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMTR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Πya Sutskever. Zero-shot text-to-image generation. In International conference machine learning, pp- 8821 8831. Pmlr, 2021.
Robin Rombach, Andreas Blattmamn, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffussin models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10684 10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Bureu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. 2022. URL https: /ttp///axvvvv..e///e//eee/eeaav/ http//awwvi...r.aaa...//../o.llt hts//bbs/220555 htts//225.114... 11487.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00519 2022.
Sara Sangalli, Ertune Erdil, and Ender Konukoglu. Expert load matters: operating networks at high accuracy and low manual effort. Advances in Neural Information Processing Systems, 00:16283-16300 2023.
Sarah Wells. Generative ai's energy problem today is foundational. IEEE Spectrum, 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rombach. Fast high- resolution image synthesis with latent adversarial diffusson distillation. arXiv preprint arXiv:2408. 12015, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. Laion-5b: open large-scale dataset for training next generation image-text models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22 Red Hook, NY, USA, 2022. Curran Associates Tne. ISBN 9781713871088.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02500 2020.
Yang Song, Prafulla Dhariwal, Mark Chen, and llya Sutskever. Consistency models. arXiv preprint arXin:2308.01469, 2023.
Dharmesh Tailor, Aditya Patra, Rajeev Verma, Putra Manggala, and Eric Nalisnick. Learning to defer to a population: meta-learning approach, 2024. URL https://arxiv wwtp:///////a////a/eeeee httpb/////23..........
Shengkun Tang, Yaqing Wang, Caiwen Ding, Yi Liang, Yao Li, and Dongkuan Xu. Adadiff: Accelerating diffusion models through step-wise adaptive computation. arXiv preprint or aix::3309 17074, 2023.
15

---

Published in Transactions on Machine Learning Research (03/2026)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Movshovitz-Attias, and Flad Eban. Wisdom of committecs: An overlooked approach to faster and more accurate models. In International Conference on Learving Represenlations, 2022a. URL
Zijie J. Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. DiffusionDB: A large-scale prompt gallery dataset for lext-lo-image generative models. Xiv:22210..1 14896 [es], 2022b. URL https://arxis: org/ ab///2bss222.... 14896.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprinl ar Xiv:2230066000 09341,,, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pp. 15903-15935, 2023.
Shuai Yang, Yukang Chen, Luozhou Wang, Shu Liu, and Yingcong Chen. Denoising diffusion step-awe aware models. arXiv preprint arXiv:2310.03337, 2023.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 5907-5915, 2017.
Hui Zhang, Zuxuan Wu, Zhen Xing, Jie Shao, and Yu-Gang Jiang. Adadiff: Adaptive step selection for fast diffusion. ArXiu, abs/2311.147700 2023. URL https: http:///pppp.eeeeasscschoooooooe www.CorpussDriCCC 2665455553.
Richard Zhuang, Tianhao Wu, Zhaojin Wen, Andrew Li, Jiantao Jiao, and Kannan Ramchandran. EmbedLLM: Learning compact representations of large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL
16

---

Published in Transactions on Machine Learning Research (03/2026)
Cost-Aware Routing for Efficient Text-To-Image Generation
Appendix
A Human Preference Score
To quantitatively evaluate how our routing decisions from each metric align with human perception, we adopt the Human Preference Score v2 (HPSv2) benchmark (Wu et al., 2023). We trained four separate routers, one for each metric (CLIPScore, Sharpness, Aesthetic Score, and ImageReward), and then report the average HPSv2 quality score achieved by our routing approach when operating at the cost (TFLOPs) of each model in the pool. For this analysis, also expand candidate pool to include the recent, stato-of-the-art FLUX model IFLUX.1-dev with 30 denoising steps)². The results are shown below.
HPSv2 score
Infinity
Turbo 0.288
Lighting 0.304
SDXL-10
DeepCache
SDXL-20
DDIM
SDXL-50
SDXL100
FLUX
Fixed Baseline
0.293
0.251
0.277
0.275
0.284
0.284
0.286
0.298
Ours (core)
0.293
0.288
0.287
0.282
0.277
0.278
0.284
0.284
0.286
0.298
Ours (Sharpness)
0.293
0.293
0.293
0.293
0.293
0.293
0.295
0.295
0.296
0.298
Ours (ImageReward)
0.293
0.293
0.293
0.293
0.293
0.293
0.295
0.295
0.296
0.298
Ouns (Aesthetic)
0.293
0.293
0.304
0.304
0.303
0.303
0.302
0.302
0.300
0.298
TFLOPs
1.50
1.54
23.92
119.70
210.00
239.40
598.50
598.50
1197.00
1785.00
Notably, the results indicate that routing with the Aesthetic Score produces outcomes that correlate most closely with human preferences as evaluated by HPSv2. This is likely because the Acsthetic Score model itself is trained human ratings of aesthetic value. Note that when the cost constraint is set to match the cheapest (Infinity) or most expensive (FLUX) model, the router must select that model for all prompts, resulting in identical performance.
B Additional Qualitative Analysis
In Figure 5, we analyze scenarios showing both successes and failures of our adaptive routing method (Proposed Transformer (SDXL+) on CLIPScore metric). Specifically, we focus cases where our method uses the same overall computational cost as the baseline (SDXL with a fixed 22 denoising steps). Within these scenarios, we consider cases where our method allocates more than 22 denoising steps, indicating that the prompts are particularly complex and require additional refinement.
For the prompt A young kid stands before birthday cake decorated with Captain America, method correctly recommends more denoising steps, as fewer would not generate accurate images. In contrast, the prompt There are two traffic signals on a metal pole, each with three light signals on them includes an exact number of objects, concept which both diffusion models and CLIP often struggle with (Binyamin et al., 2024; Paiss et al., 2023). Our approach accounts for this difficulty by recommending more steps than average. However, in this case, more denoising steps act uallally degrade image quality which uncommon and ends up hurting the router performance.
We also perform Appendix Section T). All parlicipants rate Figure 5b (ours) as the better image, while 14 of 19 participants select Figure 5c (baseline) as the better image.
study to compare the subset of these routing decisions with the fixed cost baseline (see
C Cross Dataset Generalization
To evaluate cross-dataset generalization, we conducted an experiment where our quality estimator (KNN) was trained on DiffusionDB prompts and ovaluated on the prompts from COCO captioning dataset. This a
hnttps://huggiiggaac.. co/black- Forest-labs/FLug 1-dev
17

---

Published in Transactions on Machine Learning Research (03/2026)
Success case. A young kid stands before birthday cake decorated with Captain America.
Failure case. There two traffic signals metal pole, each with three light signals on them.
(a) 22 steps (fixed)
(b) 27 steps (routed)
(c) 22 steps (fixed)
(d) 27 steps (routed)
Figure 5: Success and failure cases of the baseline SDXL with static 22 denoising steps, and approach Proposed Transformer (SDXL+) in Pigure 3a operating at the same average cost as the baseline. (a), (b): Our approach is able to recognize the need for larger number of denoising steps to generate an image that matches the prompt. (c), (d): Prompts that specify an exact number of objects are difficult for diffusion models in general. The number of objects may fluctuate during the denoising process, making it difficult to predict the right number of steps.
challenging test due to the significant stylistic differences between the datasets. In Section C, we report the average quality (Sharpness score) achieved by our routing approach when operating at the cost (TFLOPs) of each model in the pool. Our framework still achieves a good quality-cost trade-off even without being trained on this new prompt distribution. This suggests the estimator learns a robust understanding of prompt complexity beyond simple keyword correlation.
Sharpness
Turbo
Lighting
DeepCache
DDIM
SDXI.-100
Ours
0.1152
0.1150
0.1134
0.1100
0.1048
Fixed
0.1152
0.1069
0.0979
0.1049
0.1048
D Model Architecture
In this section, we provide more details on the architecture of the models we used for the quality estimator 8m in Eq. (3).
K-NN The K-NN approach provides non-parametric way to estimate quality by retrieving and averaging the quality scores of K nearest. training prompts, in appropriate embedding space. This method is simple, and generalize well with sufficient data. As no iterative training is required for K-NN (besides producing search index), it can be suitable model when the underlying pool H of base models changes frequently. We use the CLIP text encoder Radford et al. (2021) to produce prompt embeddings The text encoder is used directly without fine-tuning.
Transformer Our Transformer- estimator built on top of the text emboddings produced by the CLIP text encoder. Specifically, for each prompl, CLIP considers the first. 77 tokens and produces 77 per-token embeddings, each of 768 dimensions. To construct quality estimation model, first add two self-attention layers with position embeddings, resulting in output sequence of 77 per-token embeddings, each of 512 dimensions. We then add a small output head with 2-layer linear MLP with Sigmoid activation function on top of each of these token embeddings. Averaging across the Lokens produces M scores (x),.... (see Eq. (3)), each estimating the expected quality the m-th model on prompt x.
We train separate model for each of the quality metrics considered. In each case, the quality scores linearly normalized across all training examples to be in [0, 1). These normalized metrics treated
18

---

Published in Transactions on Machine Learning Research (03/2026)
ground-truth probabilities, and the model is trained by minimizing the cross-entropy losses. Only the attention layers and the MLP are trained with the frozen CLIP text encoder.
Note that all the base models except Infinity consume the CLIP text embedding as input. Thus, the cost of invoking our router is only from the extra layers added the case Transformer, or neighbor lookup in the case of K-NN. The overhead in terms of FL.OPs negligible compared invoking SDXL. Since the Infinity baseline uses Flan-T5 instead of CLIP, this incurs an additional 13.087 GFLOPs if Infinity is selected. To put it in perspective, calling SDXL for prompt with 17 denoising steps would incur roughly 200 TFLOPs.
E
Computational Resources
To generate our training set (quality score per prompt), we used 50 A100/H100 S0G GPU with approximately o days per model to generate 391,364 images for the filtered DiffusionDB prompt set (97,841 prompts) and less than 1 day per model for the filtered COCO prompt set (18,384 prompts). For each quality metric, we trained our Transformer with one A100 40G GPU in ~2 hours for the COCO prompt set and ~7 hours for the DiffusionDB prompt set. The trained transformer (33.88M parameters, 15.61 GFTOPs)) represents just 1.7% of the size and 1.04% of the computational cost relative to the smallest candidate, Infinity (2B parameters, 1.5 TFLOPs). For our kNN-based router, the overhead is even more marginal, as only requires single-pass search over small set of 103-10# reference samples. The trained Transformer model only takes 0.00 seconds to predict the scores for single prompt in one A100 40G GPU. We trained KNN on the CPU in less than 1 minute with negligible inference Lime(<0.00 second per prompt).
F Quality-Neutral Costs
We provide quantitative metrics to complement the deferral curves shown in Figure 3a (COCO dataset, evaluated with CLIPScore). We consider the quality-neutral cost (QNC) Ong et al. (2025); Jitkrittum et al. (2025) defined as the fraction of cost required to reach the performance of a reference model. The lower the QNC, the better because this means that our mothod can achieve the same performance as a reference model using a lower cost. The QNCs of the two proposed methods are shown in the following table, where the reference set to cach of the individual model in the pool (described in Scction 5.1).
Method
QNC (%)
INFINITY
DDIM
DEEPCACHE
LIGHTING
TURBO
SDXL100
Transformer (SDXL+)
100
2.5
11.1
6.4
99.7
2.
K-NN (SDXL+)
100
2.5
12.3
6.4
100.2
2.4
For example, QNC of 100% to INFINITY indicates that our approaches would ncod the full cost of INFINITY to reach its average performance. In other words, visually, the deferral curves of the two proposed methods would pass through the quality-cost operating point of INFINITY. As another example, the QNC of the Proposed Transformer (SDXL+) relative to the bascline DEEPCACHE is 11.1%, indicating that Proposed Transformer (SDXL+) only needs 11.1% of the cost of DEEPCACHE to have the same performance.
Overall, our proposed approaches are able to match the quality of all the baselines with either a significantly lower cost, or almost the same cost.
19

---

Published in Transactions on Machine Learning Research (03/2026)
G
Model Selection Rates
100
DDIM
Model selection rate
80
DeepCache
Infinity
Lighting
60
SDXL
Turbo
40
20
100
200
300
400
500
700
TFlops
Figure 6: (SDXL+) in Figure 3a. Our approach is able to adaptively mix and match different model choices throughout the cost range.
The rate at which each choice in the candidate routing pool is selected by Proposed Transformer
Figure shows the rate at which each choice in H. selected by Proposed Transformer (SDXL+) in Figure 3a. All the 12 candidate diffusion steps offered by the base SDXL model are collapsed into one curve for clarity. We observe that, when the cost budget is large, the router increasingly allocates resources to the full SDXL model. On the other hand, in the lower cost range, TURBO is the prominent choice, it provides a good balance between cost and quality. This analysis shows that router is able to adaptively mix and match different choices throughout the cost range to achieve a good quality-cost trade-off.
K-NN Parameter Selection
Figure shows the deferral curve when using K-NN as a quality estimator at various values of K. As shown in the figure, the routing performance (on validation set of 828 prompts drawn from the COCO datasct) is similar across wide range of K values. We set K = 100 for our final model.
0.2865
0.2860
0.2855
Quality lity
0.2850
0.2845
20-NN
50-NN
0.2840
70-NN
100-NN
0.2835
150-NN
SDXL
0.2830
40
100
Number of denoising steps
Figure 7: Performance of the proposed K-NN-based routing model validation subset drawn from COCO.
20

---

Published in Transactions on Machine Learning Research (03/2026)
User Study
Here provide details the user study to evaluate our routing decision. For this qualitative analysis, we consider the same trained router shown in Figure 3a as Proposed Transformer (SDXL+). Our reference baseline is the SDXL. model with the number of demoising steps set to 22; this setting results in a per-prompt. cost of 263.3 TFLOPs. For a fair comparison, we accordingly consider the operating point of our method that has the same average cost as this baseline by adjusting À in Eq. (3). Of all the prompts in the lest set. used in Figure 3a, we consider random subset of 100 prompts where our method does not select SDXL with 22 denoising steps as the routed decision; this filtering is done to facilitate contrast between the two approaches. We proceeded to recruit 19 participants through the Prolific platform for human preference analysis (https://www httt:://www..roliffic. prolific. com). We run a two-alternative forced-choice (2AFC) study to measure participants' preference for images produced by both approaches. During each trial, each participant is presented with a text prompt, and two test images produced by the two approaches. The participant is instructed select the image that better matches the prompt.
Protocol For each trial, a prompt is shown first to each participant. Two randomized-order images are then presented participants' screens: 1) image from Proposed Transformer (SDXL+) (in Figure 3a), and 2) image produced by the baseline SDXL, at fixed 22 denoising steps. Note that the produced image may be from non-SDXL. model (c.g., Turbo) since our approach may route to other bascline models described in Section 5.1.
Participants were instructed to select the image that matches the input text prompt better. Each participant will be assigned 100 trials (100 prompts) in total, with prompts randomly sampled from the COCO datasct described in Section 5.2. Our crowdsourcing user study protocol is visualized in Figure 8 a sequence of web pages that will be shown to participants with example stimuli.
Figure 8: User study protocol Ours SDXL with static 22 denoising steps. In each user study trial, the participant will see two images: Image A and Image B. The task is to select the image that match better with the text prompt provided. The participant needs to click on the button below or press the keyboard to choose A/B.
Additional Results
Conect Parar Phrreppeii Stady
mustatore aladd
sakatari maay.
useco slad.
eutited mostc
ortwset abok.
Figure 9: The rate at which each participant prefers our suggested image the image produced by the baseline SDXL with 22 denoising steps.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Figure shows the rate, in percentage, at which each participant selects Proposed Transformer (SDXL+) Here, we define percentage selection the proportion of trials in which ours is selected. We note that for Stable Diffusion XL, many images generated from 22 denoising steps already show reasonable results with hard-to-notice artifacts, which can limit the perceivable differences. In the main paper, we highlight that the majority of parlicipants only agreed on those most notable cases. On average, the participants prefer ours 52.37% of the time.
J Comparison with DeepCache on Quality-Cost Trade-off
Figure 10 compares our adaptive routing method (Proposed Transformer (SDXL+)) on CLIPScore with the DeepCache approach Ma et al. (2024b) (on SDXL model 50 denoising steps). DeepCache caches intermediate activations at predefined intervals (cache intervals) to balance image quality and computational cost. Varying the cache interval enables different quality cost trade-offs, which can then be compared with our method on a deferral curve. As shown in Figure 10, our method, which adaptively utilizes multiple models, consistently suraassss fixed-interval caching methods across all computational costs. Note that our adaptive routing strategy can also incorporate any DeepCache configurations to even further enhance performance.
Ours DeepCache
0.284
0.282
CLiPScore
0.280
0.278
Proposed Transformer (SDXL+)
0.276
DeepCache (Cache Interval 2-9)
100
200
300
TFlops
Figure 10: CLIPScore-TFIE TFllps trade off comparison with Deepcache at various cache interval on the test set of COCO dataset
K Statistical Significance of COCO Results
Recall that Figure 3 shows the deferral curves of our methods on COCO dataset. To give a more precise view the performance improvement, in Table 5, we report average quality scores attained by our Proposed K-NN (SDXL+) at the same costs the individual models in the pool. We observe that at the operating cost of each individual model in the pool, by CLIPScore and Sharpness). In most cases, the gains are statistically significantly better as warranted by the Welch's I-test. Note that at the two extreme ends of operating costs (i.e., calling the cheapest and most expensive models, respectively), any routing approach necessarily reduces to trivial routing: sending all prompts to one model. follows that, an extreme operating point (cither at the lowest possible or highest possible cost), the average quality achieved must be exactly the same that of the individual model that end point.
approach is able to deliver higher average quality score (as measured
22

---

Published in Transactions on Machine Learning Research (03/2026)
Table Quality cost breakdown for Proposed K-NN (SDXL+) presented in Table (on COCO dataset). An entry bold text indicates that, with the same cost, our approach is statistically significantly better than the corresponding individual model (Welch's t-test significance level 0.05).
CLIPScore Radford et al. (2021) 1
Ours 0.2672
0.2801
0.2817
0.2830 0.2773
0.2832
0.2832 0.2805
0.2832
0.2830 0.2816
0.2822
Fixed
0.2672
0.2798
0.2749
0.2820
0.2810
0.2822
Sharpness f
Ours 0.1061
0.1161
0.1166 0.1070
0.1
0.1160
0.1158
0.1 131
0.1121
0. 1048
Fixed
0.1061
0.1152
0.0997
0.0979
0.0858
0.0995
0.1049
0.1048
Model Cost
Infinity 1.50
Turbo 1.54
Lighting 23.92
SDXL-9 107.73
DeepCache 210.00
SDXL-22 263.34
SDXL-42
DDIM 598.50
SDXL100
502.74
1197.00
L
Additional Deferral Curve
Here we show the complete deferral curves for the four quality metrics CLIPScore, Sharpness, ImageReward, and Aesthetic Score, on the test set DiffusionDB dataset in Figure 11. These curves complement the fixed-cost comparison Table by showing the changes quality score across the entire cost spectrum. Here can clearly see how our adaptive routing consistently achieves a higher quality score than the fixed-model baselines at every computational budget.
Image Reward
Aesthetic
7.0
6.9
6.8
6.7
0.7
6.5
0.6
6.4
6.3
200
400
600
800
1000
1200
400
600
800
1000
1200
Sharpness
CLIPScore
0.32
0.31
0.130
DDIM
0.125
0.30
DeepCache
Infinity
0.120
0.29
Lighting
SDXL 10
0.28
0.110
0.27
SDXL 100
0.105
Turbo
0.26
Ours
0.100
200
400
600
800
1000
1200
0
200
400
600
800
1000
1200
Figure 11: Deferral Curves of our Proposed Transformer (SDXL+) on DiffusionDB dataset. Our approach exceeds the quality of fix-step texl-to-image models in all quality metrics (ImageReward, Aesthetic, Sharpness, and CLIPScore).
23

---

Published in Transactions on Machine Learning Research (03/2026)
M Estimation Errors and Quality-Cost Trade-offs
0.287
0.286
Turbo (1 step)
0.285
Lighting (4 steps)
DDIM (50 steps)
CLiPScore
0.284
DeepCache (Interval 3)
0.283
Infinity (VAR)
0.282
Oracle (Noise = 0)
Oracle (Noise STD)
0.281
Oracle (Noise STD)
0.280
Oracle (Noise = STD)
Noisc
0.279
Proposed Transformer (SDXL+)
0.278
200
400
600
800
1000
1200
TFLOPs
The deferral curves as presented in Figure 3a, supplemented with baselines derived from the oracle routing rule with noise added to simulate routing errors. See Section M for details.
Figure 12:
Table 6: Average quality scores the routing approaches presented in Figure 12.
CLIPScore 1
Infinity
Turbo
Lighting
SDXL-9
DeepCache
SDXL-22
DDIM
SDXL-65
Ours
2672
.2800
.2220
2836
.2837
.2840
.2888
.2877
Fixed
2672
.798
.2799
.773
.2820
.2805
.8166
.8199
Oracle
.2622
.2855
.2928
.973
2984
2986
2969
2944
Oracle (Noise STD)
26622
.8866
.2829
2854
2862
.2864
.8800
.2851
Oracle (Noise = 2 STD)
.672
.2800
.7822
.8066
.819
.2823
.2832
.8822
Oracle (Noise STD)
.6622
.2688
.2763
.7844
.2000
.2006
.2822
.2825
Random (Noise oo)
.2622
.2727
.2711
.2733
.2756
.2763
.2799
.2811
TFLOPs
1.5
1.54
23.92
107.73
210
263.34
598.5
778.05
In this section, we discuss the relationship between estimation errors of a quality metric, and the resulted quality-cost trade-off. Specifically, the goal is to quantify the degradation of the routing performance from the oracle routing rulc, as cstimation crrors increase. To demonstrate this, we consider the same experimental setup as used in Figure 3a i.e, with COCO as the dataset, and with CLIPScore as the image quality metric.
Plug-in estimate of the oracle rule We start with a plug-in empirical estimator f* of the optimal (oracle) routing rule in Theorem Recall that the optimal routing rule in Theorem 1 is given by
r (x) arg max E (mm) (x)) mE[Md]
x
X (m))
Recall from Section 3.2 that f.m denotes the empirical estimate E [q(x,, (x)) xx] ). Define y, (0......, For a labeled example (x,, Di), the optimal routing rule can thus be estimated as
f (x:) arg max @r.m A (m).
(4)
mE[M]
This data- based oracle rule directly applicable to test examples in the test set without require any estimation. The resulting deferral curve denoted by "Oracle (Noisc 0)" Figure Evidently, this routing rule exhibits an excellent quality-cost trade-off compared other routing approaches. Indced, it makes use of the ground-truth quality label y to make routing decision. By construction, no other routing methods can give
24

---

Published in Transactions on Machine Learning Research (03/2026)
better trade-off curve than this deferral curve (on this specific dataset). We emphasize that in practice it is extremely challenging to realize quality-cost operating point that is close to this oracle routing rule. See, for instance, Figure and Figure 4 in Hu et al. (2024) for the performance gap to the oracle in the context of LLM routing (i.c., not routing to text-to-image models, as considered our work). Nevertheless, the oracle curve serves as an upper bound on the trade-off achievable by any routing methods.
Noisy oracle We now consider adding noise to the routing rule in (4) to examine how noise affects the quality-cost trade-off:
(x,) arg max Di,m MEIM
BB-STD, Di. c(m)
(5)
where
controls the strength of Gaussian noise to add;
STD,.. m Noe E (Yn,m vm)?, and ym = Noc En Un,mi
No.
Nic denotes the number of test examples; and
Ii. l.d. ((0,1) 1) an independent rcalization from the standard normal distribution.
Clearly, if B = O, we recover the empirical oracle rule in (4). The parameter 3 represents the amount of Gaussian noise added to the quality scores, in the unit of the standard deviation of the quality scores of the respective model.
In Figure 12 and Table 6, present the performance of the noisy routing rule with B E (1, When 3 co, simply perform random routing, which gives poor trade-off. It can be seen that proposed the deferral curve of our proposed router lies between the curves of the oracle with 3 and the oracle with 8 = 2. Roughly, this means that our router has similar performance to an oracle router where the per-model ground-truth scores are corrupted with independent Gaussian noise at a factor 3 E (1,2) of their standard deviations.
25


Reviews:
[{"review_id": "VfV500BHQn", "reviewer": "Reviewer_zXQn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed a method to improve quality/cost tradeoff of text-to-image generation by employing a router to select the optimal choice from a pool of generation functions, which could include different models and/or same model with different number of denoising steps. The optimization problem is basically framed as $argmax_m( quality(m) - \\lambda* cost(m))$ where m is one of the options in the pool and lambda is a hyper-parameter depending on quality and cost metrics. Author further demonstrated two possible quality estimators, i.e. K-nearest neighbors-based and transformer-based, to plug into the optimization formula. Compared to using model inference to compute quality metrics, estimators only cost a fraction of FLOPs. Experimental results showed that the proposed method can achieve comparable or better quality under the same budget compared to a single fixed model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "a decent amount of experimental data is provided. but some may need further clarifications to make it more convincing and clear. please see  Requested Changes below.", "audience_yn": "Yes", "audience_explanation": "text-image generation is a very important application and widely used nowadays.", "requested_changes": "**1. Accuracy of quality estimator**\n\nIn Sec 5.3 second paragraph, the paper does not clearly distinguish the quality score predicted by the estimator from the one computed from real generated images using the option router selected. Take Fig. 3a, SDXL-only, transformer-based estimator (orange solid line) as an example. The only variable that router can control in this case is the number of denoising steps. For a given inference cost (in TFLOPs), the number of steps is determined accordingly, therefore, the router has no degree of freedom anymore. In other words, the orange solid line (the predicted score) should (ideally) match the actual quality scores calculated from SDXL generated images, i.e., red and brown stars. Compared to its counterpart in Fig 3b, apparently the SDXL estimator for CLIP score is not as accurate as the estimator for Sharpness score and tends to overestimate. While it's understandable that estimators do not always work perfectly, it is quite confusing to claim that \"the proposed method\" can outperform the baseline at a given inference cost in this case, because the proposed method can only choose baseline. Author may want to add a few sentences regarding estimator accuracy in this paragraph and consider clarifying whether a reported quality score is predicted by estimator or calculated from real (prompt, image) pairs for all the tables and figures. \n\n\n**2. 100% Win rate in Table 1**\n\nIn Sec 5.4, \"...Win Rate, defined as the fraction of trials that our router has higher test quality than the baseline...\" Because router is choosing from all the possible options including baseline, intuitively there would be a certain probability that the router will choose the baseline model. For example, in Fig. 6 at ~260TFLOPs the router has ~45% of chance to pick SDXL, which means when compared to SDXL(-22), router will tie with baseline about half of the time. Therefore, 100% Win Rate in most of the cases in Table 1 doesn't seem very reasonable. Maybe author can elaborate a bit in Sec 5.4.\n\n\n**3. Effect of hyper-parameter $\\lambda$**\n\nAt the end of Sec 3.2, author intentionally stated that $\\lambda$ is a hyper-parameter for users to tune. But as mentioned in Sec 3.3, this parameter ranges from 0 to inf, which makes it very difficult for readers to work with. It would be very helpful if the author can further discuss the effect and the behavior of this parameter, possibly provide a simple ablation study just like Sec 5.4 on Sample Size S.", "broader_impact_concerns": "No ethical implication concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "tuFHaSIAM5", "reviewer": "Reviewer_R7ob", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a framework to adaptively selects the most efficient text to image generation models for a given prompt based on its complexity through cost-aware routing. The major contribution is applying the optimization formulation from Universal Model Routing to text-to-image generation with its specific image quality estimator. It uses the optimal routing rule inferenced from the proposition 1 to route the high-complexity prompt to a more expensive model and simple tasks to more efficient models, which differs from most of the existing inference optimization methods for image generation. The empirical results demonstrated the efficacy of the proposed frameowk, which can achieve an average quality than a single model can achieve.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The proposed framework is adapted from the formulation and proposition in Universal Model Routing for Efficient LLM Inference paper. It results in a plug-in estimator that can be applied to different model architectures and quality metrics.\n\nThe empirical evaluation in this paper is strong and demonstrates the efficacy of the proposed framework on two benchmark datasets (COCO, DiffusionDB) across multiple image quality metrics (CLIPScore, Sharpness, etc.). The results show that the proposed method can outperform static model baselines when using the same amount of TFLOPs, and on DiffusionDB it only needs 0.13% cost of a static model to achieve the same Sharpness.", "audience_yn": "Yes", "audience_explanation": "Compared to prior works in the image generation area, which mostly focus on parameter or step optimization, this method proposes a different solution that can employ various model architectures to achieve more diverse coverage and cost reduction. It successfully adapts the formulation and method from UMR to the text-to-image generation task and demonstrates its effectiveness with image specific quality metrics and estimators.\n\nTherefore, I think it presents a solid innovation that can interest researchers in the related domains.", "requested_changes": "I would recommend the authors bring the evaluation results with the FLUX.1-dev to the main paper as it demonstrates the method's efficacy on more recent architectures.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ENf3MBOSHT", "reviewer": "Reviewer_3zX2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "To address the limitations of existing methods in effectively balancing quality and efficiency under fixed inference budgets, this paper proposes CATImage, a cost-aware routing framework for text-to-image generation tasks. First, the authors identify that input prompts exhibit significant variance in their intrinsic generation difficulty, rendering uniform computational allocation suboptimal for balancing overall quality and cost. Consequently, the paper formalizes the problem as an optimization task subject to a computational budget constraint and derives a theoretically optimal Bayesian routing strategy based on this formulation. The framework employs a trained lightweight quality estimator to predict the expected quality of various candidate generators—ranging from homogeneous adjustments of denoising steps to heterogeneous switches between distinct model architectures. During inference, it dynamically selects the most appropriate generation path for each input prompt according to the derived Bayesian optimal rule. Systematic experiments on the COCO and DiffusionDB datasets demonstrate that CATImage establishes a performance curve in the quality-cost plane that outperforms all single static models, significantly reducing average computational costs while maintaining or even enhancing generation quality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The work is well-motivated and features a methodological design with strong generality, effectively extending the concepts of \"Learning-to-Defer\" and \"Model Routing\" to the text-to-image generation domain for the first time. The proposed scheme is model-agnostic, requiring no modifications to the base generative models. By simply training a quality estimator to predict performance and applying the derived cost-aware Bayesian optimal criterion, it dynamically routes input prompts—based on their intrinsic difficulty—to the most cost-effective generation model or number of denoising steps. Consequently, the framework demonstrates strong engineering practicality and scalability, allowing for the seamless integration of more efficient models developed in the future. In terms of experimental design, the use of deferral curves intuitively demonstrates the superiority of dynamic routing over static strategies, rendering the conclusions credible.", "audience_yn": "Yes", "audience_explanation": "Insufficient Analysis of Router Overhead: While the overall methodology is clear, the paper lacks a detailed quantitative analysis of the router's own computational overhead. Key metrics such as the actual inference latency and parameter size of the quality estimator are missing, which are crucial for evaluating the net efficiency gains.\n\nHigh Maintenance Cost for Updates: Although the paper emphasizes the framework's flexibility, the discussion regarding maintenance costs is limited. Introducing a new candidate model requires re-generating images and computing quality scores across the entire training dataset to retrain the router. \n\nLack of Error Analysis and Confidence Estimation: Each prompt can only be routed to a single model, and an incorrect selection will directly degrade the generation quality. The paper neither analyzes the router's error rate nor its impact on final image quality, nor does it provide uncertainty or confidence estimates.", "requested_changes": "The submission supports its claims through both theoretical grounding and empirical validation. Theoretically, the authors derive an optimal routing rule based on Bayesian decision theory, providing a sound mathematical foundation for the proposed decision-making strategy. Empirically, the authors conduct a systematic evaluation on two representative datasets, COCO and DiffusionDB, benchmarking against multiple strong baselines, including SDXL-Turbo, Lightning, and DeepCache. By presenting detailed \"deferral curves,\" the results demonstrate that CATImage achieves a superior quality-cost trade-off compared to individual generative models across varying computational budgets. Furthermore, the inclusion of ablation studies on homogeneous versus heterogeneous routing schemes, along with an analysis of specific failure cases (e.g., counting-based prompts), further bolsters the empirical findings. Overall, the experimental setup is comprehensive and the comparisons are sufficient, providing robust support for the paper's main methodological claims.", "broader_impact_concerns": "This work is highly relevant to researchers focusing on Efficient ML, generative model acceleration, and deployment optimization. As diffusion models become increasingly ubiquitous in real-world applications, reducing inference costs while maintaining generation quality has emerged as a pervasive technical challenge. The proposed routing framework, grounded in the \"Learning-to-Defer\" paradigm, offers a novel perspective on enhancing inference efficiency without modifying the underlying model architectures. Consequently, it holds potential value for both researchers and practitioners exploring heterogeneous generative model collaboration and strategies to reduce latency and energy consumption. Furthermore, the discussion regarding the relationship between prompt difficulty and computational requirements may provide further insights into understanding the performance boundaries of diffusion models.", "additional_comments": "1. Quantify Router Overhead\n\nPlease report the parameter size of the quality estimator and its actual inference latency on the hardware used in the experiments. Additionally, provide the ratio of this latency relative to the inference time of the fastest model in the candidate pool (e.g., SDXL-Turbo or Lightning). If this proportion is significant (e.g., >10%), please revise the relevant claims regarding \"negligible overhead\" or discuss the implications for real-time applications.\n\n2. Discuss Maintenance Cost for Extending the Candidate Pool\n\nI suggest explicitly clarifying in the Limitations or Discussion section that introducing a new model to the candidate pool requires re-generating quality labels across the entire training set and retraining the router. Please acknowledge that this offline overhead may limit the framework's scalability in scenarios where models are rapidly updated or iterated.\n\n3. Provide Error Rate and Impact Analysis of Misrouting\n\nTo provide a more comprehensive understanding of the router's robustness, I suggest reporting the misrouting rate on the test set. Furthermore, please quantify the average degradation in quality caused by these misclassifications compared to the optimal routing choice.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)