D2D: Detector-to-Differentiable Critic for Improved Numeracy in Text-to-Image Generation

## Abstract

Abstract Text-to-image (T2I) diffusion models have achieved strong performance in semantic alignment, yet they still struggle with generating the correct number of objects specified in prompts. Existing approaches typically incorporate auxiliary counting networks as external critics to enhance numeracy. However, since these critics must provide gradient guidance during generation, they are restricted to regression-based models that are inherently differentiable , thus excluding detector-based models with superior counting ability, whose count-via-enumeration nature is non-differentiable . To overcome this limitation, we propose Detector-to-Differentiable ( D2D ), a novel framework that transforms non-differentiable detection models into differentiable critics, thereby leveraging their superior counting ability to guide numeracy generation. Specifically, we design custom activation functions to convert detector logits into soft binary indicators, which are then used to optimize the noise prior at inference time with pre-trained T2I models.
Our extensive experiments on SDXL-Turbo, SD-Turbo, and Pixart-DMD across four benchmarks of varying complexity (low-density, high-density, and multi-object scenarios) demonstrate consistent and substantial improvements in object counting accuracy (e.g., boosting up to 13.7% on D2D-Small, a 400-prompt, low-density benchmark), with minimal degradation in overall image quality and computational overhead.

## 1 Introduction

Figure: Figure 1: Qualitative examples illustrating the count-correction ability of our detector-based critic on a variety of objects, counts 1-10. SDXL-Turbo (Sauer et al., 2025) is a base model with no post-enhancement. ReNO (Eyring et al., 2024) is a generic semantic alignment method that exhibits limited performance in this setting. More recent methods, like Make It Count (Binyamin et al., 2025) and Counting Guidance (Kang et al., 2025), explicitly address count-correction. Our method proposes a new and effective way to leverage detectors for this challenging task. Prompt template: “A realistic photo of a scene with [count] [object class].”
Refer to caption: x1.png

Diffusion-based text-to-image generative models (Podell et al., 2024; Rombach et al., 2022; Sauer et al., 2025; Chen et al., 2024; 2025b) have achieved promising performance in semantic alignment between the synthesized images and text prompts, particularly with recent post-enhancement techniques such as fine-tuning (Clark et al., 2024; Chen et al., 2025a; Yang et al., 2024; Wallace et al., 2024; Black et al., 2024; Xu et al., 2023; Fan et al., 2023) or sampling-based, training-free strategies (Wallace et al., 2023; Eyring et al., 2024; Chung et al., 2024; Chefer et al., 2023).
However, even with those advanced alignment techniques, T2I diffusion models continue to struggle at generating exact numbers of objects.
As illustrated in Fig. [1](https://arxiv.org/html/2510.19278v1#S1.F1), recent semantic alignment methods, such as ReNO (Eyring et al., 2024), which enhances generic image alignment with user intent via human preference rewards, shows limited ability to synthesize images with the exact number of objects specified in the text input. Motivated by this observation, we tackle the challenge of accurate numeracy generation in this work.

Since vanilla T2I models are not explicitly trained to count, existing methods (Kang et al., 2025; Zafar et al., 2024) introduce auxiliary counting critics to provide additional supervision during generation. These correction signals are propagated to the generative backbone through gradients from the external critics, which restricts current approaches to differentiable, regression-based models such as RCC (Hobley & Prisacariu, 2022) and CLIP-Count (Jiang et al., 2023). However, this design inherently excludes detector-based models, which perform counting via bounding box enumeration. Despite being non-differentiable, such detectors (e.g., OWLv2 (Minderer et al., 2023), YOLOv9 Wang et al. (2024)) often outperform regression-based counterparts (e.g., RCC (Hobley & Prisacariu, 2022), CLIP-Count (Jiang et al., 2023), CounTR (Chang et al., 2022)) in low-density object scenarios due to their more advanced object localization ability, as illustrated in Fig. [2(b)](https://arxiv.org/html/2510.19278v1#S1.F2.sf2). To this end, we propose resolving this bottleneck by converting existing object detectors into differentiable critics, thereby allowing T2I diffusion models to benefit from stronger counting models for improved numeracy.

Our Detector-to-Differentiable (*D2D*) framework builds on two key insights that set it apart from existing numeracy-enhancement methods (Kang et al., 2025; Zafar et al., 2024; Binyamin et al., 2025). First, rather than relying on the conventional non-differentiable *“count-via-enumeration”* mechanism, we design a high-curvature activation function that converts bounding box logits outputted from detectors into soft binary indicators, thereby making them gradient-friendly for count optimization. Second, to leverage our *“count-via-summation”* gradient, unlike prior approaches that intervene at intermediate states or denoised predictions along the sampling trajectory, we instead optimize the initial noise using a test-time tunable module, the Latent Modifier Network. This backbone-agnostic design enables broader generalization of our method across diverse diffusion-based T2I architectures, including U-Net (Ronneberger et al., 2015) and DiT (Peebles & Xie, 2023).

We demonstrate the effectiveness of *D2D* via comprehensive experiments using various generative backbones (i.e., SDXL-Turbo (Sauer et al., 2025), SD-Turbo (Sauer et al., 2025), Pixart-DMD Chen et al. (2025b)) and multiple benchmarks (i.e., CoCoCount (Binyamin et al., 2025), D2D-Small, D2D-Multi, D2D-Large), covering diverse numeracy generation scenarios, including single and multiple objects. *D2D* yields the highest numeracy across all multi-step and one-step baselines and benchmarks. In particular, on base model SDXL-Turbo, *D2D* effectively corrects 42% of under-generations (i.e., where the initial generation contains fewer than requested objects) and 40% of over-generations, nearly or more than 2x ReNO (Eyring et al., 2024) and Token Optimization (TokenOpt)’s (Zafar et al., 2024) correction rate. In summary, our contributions are as follows:

- •
We highlight the importance of accurate numeracy in T2I generation and propose a framework to convert robust object detectors into differentiable critics for count-correction with a newly designed activation function, addressing the bottleneck of having to rely on existing regression-based methods.
- •
We reposition the count-correction problem within the initial noise optimization framework, motivated by the presence of structural priors that exhibit cross-model consistency.
- •
Our method *D2D* outperforms previous one-step and multi-step count-correction methods by up to 13.7% points (from 30% with Make It Count to 43.7% with *D2D* on D2D-Small), with minimal degradation in image quality (Fig. [1](https://arxiv.org/html/2510.19278v1#S1.F1)). On single-object prompts with counts $\leq$ 10, our method introduces less or comparable computational overhead than baselines.

Figure: (a) Low and high-density examples with incorrect numeracy, generated by SDXL-Turbo.
Refer to caption: figures/count_motiv1.jpg

## 2 Related work

Generic alignment-enhancement methods. As noted in the literature (Black et al., 2024; Chen et al., 2025a), the base log-likelihood objective of diffusion models is insufficient on its own to achieve state-of-the-art alignment. To address this, prior works optimize human preference scores via post-enhancement strategies ranging from fine-tuning the U-Net or text encoder (Clark et al., 2024; Xu et al., 2023; Yang et al., 2024; Wallace et al., 2024; Black et al., 2024; Fan et al., 2023; Chen et al., 2025a) to inference-time, training-free strategies that update the intermediate latents (Chung et al., 2024; Chefer et al., 2023). A promising recent line of work (Wallace et al., 2023; Eyring et al., 2024) proposes inference-time alignment via initial noise selection, motivated by the presence of semantic/structural priors in the initial noise (Wang et al., 2025) that influence the semantics/structure of the generated output consistently across diffusion models even with different backbones. But regardless of whether the specific approach is to fine-tune model components or update latents, the problem remains that generic alignment objectives like human-preference scores are insufficient to solve numeracy, as we find there remains a significant gap relative to state-of-the-art count-correction methods like Binyamin et al. (2025). In our work, we specifically address the challenge of improving numeracy with a new formulation for the objective, as well as adopt initial noise optimization as the method of learning, for the ease with which it can be applied across different backbones and the ability to leverage optimized seeds to complement existing methods, as we demonstrate in experiments.

Numeracy correction methods.
Existing count-correction methods leverage two main mechanisms at inference-time to correct count: (1) apply the gradient of external counting models to correct a tunable portion of the generation process, like Counting Guidance (Kang et al., 2025) and TokenOpt (Zafar et al., 2024), or (2) use attention to control the layout of generated instances, like Make It Count (Binyamin et al., 2025). Counting Guidance uses the RCC counting model (Hobley & Prisacariu, 2022) to optimize the predicted noises, and TokenOpt uses CLIP-Count (Jiang et al., 2023) to optimize the embedding of a count token injected into the prompt as well as a detector to scale down CLIP-Count’s overestimates, which increases the computational overhead (about 2-6 times longer than *D2D* on average). Make It Count (Binyamin et al., 2025) is an SDXL-specific (Podell et al., 2024) method that uses self-attention features of the U-Net to extract masks of generated instances and cross-attention to enforce a corrected set of masks. These works are either limited by the need to rely on regression-based counters or manner in which they enforce structure at the cost of image quality, a phenomenon documented in Dinh et al. (2023); Zafar et al. (2024); Patel & Serkh (2025) and noted in our experiments. Instead, *D2D* leverages a more robust *detector-based* critic that enables more effective correction in the low-density setting.

Regression vs. detector-based counting models.
Regression-based counting methods take an input image and estimate count on a continuous scale. Different variations allow for (1) exemplar-based (i.e., count the instances that look similar to the user-provided example), (2) zero-shot (i.e., count the most salient object), and (3) text-prompted counting (i.e., count the text-specified object). Designed to help count high-density images, where continuous-scale predictions are appropriate, they exhibit limited performance in low-density images (Zhang et al., 2025), as illustrated in Fig. [2(b)](https://arxiv.org/html/2510.19278v1#S1.F2.sf2). On the other hand, our *D2D* critic is derived from detectors which show robust performance given low-density images, which is critical to the generative setting (Fig. [2](https://arxiv.org/html/2510.19278v1#S1.F2)). Furthermore, our critic can be used to generate objects in the open set by leveraging *open-vocabulary* detectors, like OWLv2 (Minderer et al., 2023), with minimal modification to detector architecture. In our work, we compare our critic against three regression-based counting methods: RCC (Hobley & Prisacariu, 2022) (zero-shot), CLIP-Count (Jiang et al., 2023) (text-specified), and CounTR (zero-shot) (Chang et al., 2022).

## 3 The D2D framework

Problem statement.
Given a pre-trained, one-step T2I model $G_{\theta}$ and prompt $p$ requesting $N$ counts of an object of class $C$, our goal is to generate an image with exactly $N$ counts of $C$.

Summary of approach. We propose a detector-based count critic that provides a more effective gradient signal. We then design a method to use that signal to influence the generation process, by leveraging the structural priors in the initial latent which we modify to align with the gradient.

### 3.1 Detector-to-differentiable critic

Detector $\mathcal{D}$ takes as inputs an object class $C$ and image $I$ and outputs a set of $n$ bboxes $\{B_{i}|1\leq i\leq n\}$ and logits $\mathbf{z}=\{z_{i}|1\leq i\leq n\}$. A standard sigmoid $\sigma(z_{i})=\frac{1}{1+e^{-z_{i}}}$ converts the logits into confidence scores between 0 and 1, with the most salient bboxes filtered using threshold $\tau$, as follows: $\mathbf{B}=\{B_{i}|\sigma(z_{i})\geq\tau\}=\{B_{i}|z_{i}\geq\tau_{z}\}$, where $\tau_{z}=\sigma^{-1}(\tau)$. The final count is $|\mathbf{B}|$. Our goal is to derive a gradient from $\mathcal{D}$ that can effectively increase or decrease $|\mathbf{B}|$ as needed. Our approach is to first, define a differentiable function $f:\mathbf{z}\in\mathbb{R}^{n}\mapsto\mathbb{N}$ that can extract the count from the logits $\mathbf{z}$, and second, transform $f$ so its gradient is more amenable to convergence, arriving at critic $\mathcal{L}_{\text{D2D}}$.

Extract the count via $\bm{f}$. Counting is discrete in nature, but we convert it into a continuous, differentiable one by borrowing techniques from logistic regression for binary classification, which optimizes the steepness and transition threshold of a sigmoid-curve for discrete 0/1 prediction. We convert each logit into an approximate binary indicator of whether to “count” the corresponding bbox, by applying to each logit a steep sigmoid curve with transition threshold $\tau_{z}$ and steepness coefficient $\beta$, with the final differentiable count formulated as a sum of sigmoids (Eq. [1](https://arxiv.org/html/2510.19278v1#S3.E1)). The next challenge is to make this gradient-friendly.

Transform $\bm{f}$ to effectively handle over/under-generation.
An effective critic provides a strong gradient signal above/below $\tau_{z}$ (our domain of interest) to push logits below or beyond the threshold as needed to erase/add objects in response to over/under-generation. However, by nature of its sigmoidal shape, $f$ has significant plateauing (i.e., weak gradient signals) above and below $\tau$. To improve the gradient steepness in our domain of interest, we scale each sigmoid output by the corresponding logit (Eq. [2](https://arxiv.org/html/2510.19278v1#S3.E2)), arriving at $\mathcal{L}_{\text{D2D}}$. At inference-time, we use $\nabla\mathcal{L}_{\text{D2D}}$ to optimize the generated image.(^1^11Unless otherwise noted, we use $f$ to perform early-stopping once the requested count is met.)

$$ $f_{\beta,\tau_{z}}(\mathbf{z})=\sum_{i=1}^{n}\sigma(\beta\cdot(z_{i}-\tau_{z})).$ (1) $$

$$ $\mathcal{L}_{\text{D2D}}=\begin{cases}\sum_{i=1}^{n}\sigma(\beta\cdot(z_{i}-\tau_{z}))\cdot(z_{i}-\tau_{z}),&\text{if }f_{\beta,\tau_{z}}>N\text{ (i.e., over-generation)}\\ \sum_{i=1}^{n}\sigma(\beta\cdot(\tau_{z}-z_{i}))\cdot(\tau_{z}-z_{i}),&\text{if }f_{\beta,\tau_{z}}<N\text{ (i.e., under-generation)}\end{cases}$ (2) $$

Extension to multiple classes.
The main consideration in extending *D2D* to prompts with $m>1$ object classes $\{C_{j}|1\leq j\leq m\}$, is that every bbox comes with $m$ logits, the maximum of which determines its class label. To extend *D2D*, we update Eq. [2](https://arxiv.org/html/2510.19278v1#S3.E2) to correct each bbox’s largest logit, while minimizing all others. Details in Appendix [E](https://arxiv.org/html/2510.19278v1#A5).

Figure: Figure 3: The *D2D* pipeline for improving T2I numeracy. *D2D* consists of two main components that work together to improve numeracy: our detector-based count critic guides the Latent Modifier Network (LMN) on how to transform the original initial noise $\mathbf{x}_{T}$ into a more optimal $\mathbf{x}_{T}^{\prime}$. Our count critic uses sigmoid-based activation functions to convert logits into gradient signals, which are then backpropagated through the frozen generator to update the weights of the LMN.
Refer to caption: figures/pipeline.png

### 3.2 The Latent Modifier Network (LMN)

Given our proposed count critic, we now turn to the learning method used to optimize this objective. Motivated by the presence of meaningful priors in the initial noise, previous works (Eyring et al., 2024; Wang et al., 2025) have used various generic alignment metrics to tune the initial noise directly. Building on this motivation, we propose the Latent Modifier Network (LMN), a test-time tunable module whose output is mixed with the original noise to determine the optimal initial noise and whose weights are updated using our critic $\mathcal{L}_{\text{D2D}}$.

Given initial noise $\mathbf{x}_{T}\sim\mathcal{N}(0,{\bm{I}})$, $\mathbf{x}_{T}\in\mathbb{R}^{d}$ and prompt $p$ that requests $N$ counts of an object of class $C$, one-step T2I model $G_{\theta}$ generates image $I$. Our goal is to find an optimal $\mathbf{x}_{T}^{*}$ that produces an image $I^{*}$ with exactly $N$ of the specified object. To achieve this, we introduce a tunable Latent Modifier Network (LMN) $M_{\phi}$: a small, 3-layer perceptron, between the initial random latent and T2I model (Fig. [3](https://arxiv.org/html/2510.19278v1#S3.F3)), with input/output dimensions equal to that of the initial latent and whose output dictates how to update $\mathbf{x}_{T}$. As shown in Eq. [3](https://arxiv.org/html/2510.19278v1#S3.E3), the new latent is a weighted sum of $\mathbf{x}_{T}$ and $M_{\phi}(\mathbf{x}_{T})$, with mixing weight $w=0.2$. Compared to tuning the initial latent directly, the LMN composes a relatively larger parameter space and enforces more incremental updates that preserve a portion of the original latent even through all iterations. At inference-time, we tune $\phi$ using $\nabla\mathcal{L}_{\text{D2D}}$ with the goal of correcting the initial noise, and thereby the numeracy, as described in the following section.

$$ $\mathbf{x}_{T}^{\prime}=w\cdot\mathbf{x}_{T}+(1-w)\cdot M_{\phi}(\mathbf{x}_{T}).$ (3) $$

### 3.3 Optimization

The goal is to find the optimal set of parameters $\phi$ that minimizes the error between the generated and requested count, as seen in Eq. [4](https://arxiv.org/html/2510.19278v1#S3.E4). Since detector $\mathcal{D}$ is non-differentiable, we leverage $\mathcal{L}_{\text{D2D}}$ to optimize $\phi$ iteratively, rendering our final update rule (Eq. [5](https://arxiv.org/html/2510.19278v1#S3.E5)), with regularization term $\mathcal{L}_{\text{reg}}$, learning rate $\eta$, and weights $\alpha$ and $\lambda$. We adaptively rescale the loss to address exploding gradients that we may encounter due to the large number of tunable parameters. During numeracy optimization, we apply a variant of the regularization term used in ReNO (Eyring et al., 2024), using the negative log-likelihood of the norm of $\mathbf{x}_{T}$ as follows: $\mathcal{L}_{\text{reg}}^{\prime}=||\mathbf{x}_{T}^{\prime}||^{2}/2-(d-1)\cdot\log(||\mathbf{x}_{T}^{\prime}||)$. We use $\mathcal{L}_{\text{reg}}=(a\mathcal{L}_{\text{reg}}^{\prime}+c)^{10}$, with scaling coefficient $a$ and shift constant $c$.

$$ $\phi^{*}=\arg\min_{\phi}|\mathcal{D}(G_{\theta}(\mathbf{x}_{T}^{\prime}))-N|.$ (4) $$

$$ $\phi\Leftarrow\phi-\eta\nabla(\alpha\mathcal{L}_{\text{D2D}}+\lambda\mathcal{L}_{\text{reg}}).$ (5) $$

$\bm{\phi}$ initialization. To give $M_{\phi}$ a good starting point (i.e., initialize the network’s initial output distribution to Gaussian), we propose a short, pre-inference alignment stage to be performed one time per base model using only the regularization term. Specifically, we train $M_{\phi}$ on 100 different randomly sampled latents ($\mathbf{x}_{T}$) for 200 epochs each (Algorithm [1](https://arxiv.org/html/2510.19278v1#alg1) in the appendix).

At inference-time, given a new, randomly sampled $\mathbf{x}_{T}$ the network has never seen before, we introduce a ~0.2-second calibration phase to allow the network to adapt to the new input, using only the regularization term. Afterward, we leverage both *D2D* and regularization terms, according to Eq. [5](https://arxiv.org/html/2510.19278v1#S3.E5). The full algorithm is detailed in Algorithm [2](https://arxiv.org/html/2510.19278v1#alg2) in the appendix.

## 4 Experiments and analysis

### 4.1 Experimental setup

Benchmarks. Our main experimental setting of single-object, low-density prompts leverages two benchmarks, CoCoCount (Binyamin et al., 2025) and D2D-Small. D2D-Small is a set of 400 prompts created using 40 countable objects from COCO (Lin et al., 2014) with counts ranging from 1-10 and a prompt template adapted from Lian et al. (2024): “A realistic photo of a scene with [count] [object].” CoCoCount consists of 200 prompts from 20 COCO classes and requested counts roughly equally split among 2, 3, 4, 5, 7, and 10. Experiments on multi-object or high-density prompts were performed on D2D-Multi (400 prompts with two objects sampled from 40 countable COCO classes, with $N_{1},N_{2}<10$, and following the template: “A realistic photo of a scene with [count] [object] and [count] [object]”) and D2D-Large (similarly constructed with counts 11-20).

Base models. We apply *D2D* to three one-step models: SDXL-Turbo (Sauer et al., 2025), SD-Turbo (Sauer et al., 2025), and Pixart-DMD (Chen et al., 2025b). SDXL-Turbo and SD-Turbo, respectively distilled from SDXL (Podell et al., 2024) and SD2.1 (Rombach et al., 2022), have U-Net backbones. Pixart-DMD, distilled from Pixart-$\alpha$ (Chen et al., 2024), has a Transformer backbone.

Comparison of numeracy enhancement methods.
We compare *D2D* against count-correction baselines (1) Make It Count (Binyamin et al., 2025) (multi-step), which uses attention-based mechanisms to identify and correct object layout via updates to the intermediate latents, (2) Counting Guidance (Kang et al., 2025) (multi-step), which uses the auxiliary counting network RCC (Hobley & Prisacariu, 2022) to correct the predicted noises, and (3) TokenOpt (Zafar et al., 2024), a one-step method which injects a count token into the prompt and tunes it using CLIP-Count (Jiang et al., 2023). We run each baseline following its original experiment setup. Importantly, Make It Count is an SDXL-based method; Counting Guidance (original experiments and codebase) is primarily centered on SD1.4; and TokenOpt is built on SDXL-Turbo, so we evaluate Make It Count on SDXL, Counting Guidance on SD1.4, and TokenOpt on SDXL-Turbo. Furthermore, Make It Count addresses the low-density, single-object setting and TokenOpt addresses the single-object setting, so we only evaluate Make It Count on CoCoCount and D2D-Small and TokenOpt on CoCoCount and D2D-Small/Large.

Comparison with generic prompt-alignment method. The most relevant prior initial noise optimization work is ReNO (Eyring et al., 2024), a framework for one-step T2I models that uses the combined gradient of multiple image quality and prompt-image alignment metrics (ImageReward (Xu et al., 2023), PickScore (Kirstain et al., 2023), HPSv2 (Wu et al., 2023), and CLIPScore (Hessel et al., 2021)) to optimize semantic alignment and image quality. Instead of tuning an LMN, ReNO directly tunes the initial latent over 20-50 iterations, with regularization to keep the noise within the initial distribution and gradient clipping to prevent gradient explosion. Though its use of human-preference reward models does improve numeracy relative to the base model, there remains a gap between using such generic objectives and our count-tailored critic (Tab. [1](https://arxiv.org/html/2510.19278v1#S4.T1)). A key difference between our method and ReNO’s is our introduction of the LMN, which expands the tunable parameter space while preserving a portion of the original initial noise throughout the optimization process. To assess the impact of introducing the LMN, we compare our initial noise optimization method with ReNO’s, controlling for the loss by swapping out ReNO’s human-preference models for our *D2D* critic.

Count critic. We demonstrate *D2D* on detectors OWLv2 (Minderer et al., 2023) (open-vocabulary, robust) and YOLOv9 (Wang et al., 2024) (high-throughput and trained on COCO (Lin et al., 2014) objects). We expect a small accuracy-cost tradeoff, where OWLv2 enables superior numeracy with greater computational overhead, while YOLOv9 yields slightly lower numeracy but faster inference.

Evaluation. Following similar evaluation protocols (Binyamin et al., 2025; Kang et al., 2025; Zafar et al., 2024), we use CountGD (Amini-Naieni et al., 2024), a state-of-the-art counting model built on detector GroundingDINO (Liu et al., 2025), to extract the count of generated objects and compute the proportion of correctly-generated images (see Appendix [H](https://arxiv.org/html/2510.19278v1#A8) for CountGD’s counting accuracy compared to other regression/detector-based methods). Like Eyring et al. (2024), we analyze image-quality/prompt alignment with human-preference-trained models (ImageReward (Xu et al., 2023), PickScore (Kirstain et al., 2023), HPSv2 (Wu et al., 2023)), and CLIPScore (Hessel et al., 2021).

Implementation details. Our main experiments were completed on an L40. For Make It Count (Binyamin et al., 2025) which requires $>$ 50 GB, we used an A100. Additional details in Appendix [D](https://arxiv.org/html/2510.19278v1#A4).

**Table 1: Quantitative results. *D2D* outperforms all baselines across all four benchmarks (CoCoCount (Binyamin et al., 2025) and D2D-Small/Multi/Large), even generalizing across detector variants OWLv2 (Minderer et al., 2023) and YOLOv9 (Wang et al., 2024). *D2D* with YOLOv9 on base model SDXL-Turbo is in bold italics to show that while it outperforms all baselines, it is second to using OWLv2. The higher-performing OWLv2 detector is used in all subsequent experiments on SD-Turbo and Pixart-DMD. Standard deviations indicate the significance of our findings. Base models with no post-enhancement highlighted in gray. Avg. over four seeds.**
| Method | CoCoCount | D2D-Small | D2D-Multi | D2D-Large |
| --- | --- | --- | --- | --- |
| \rowcolorgray!22 SDXL (Podell et al., 2024) | 24.88 $\pm$1.70 | 16.06 $\pm$1.86 | 2.44 $\pm$0.59 | 1.44 $\pm$0.38 |
| + Make It Count (Binyamin et al., 2025) | 46.75 $\pm$2.10 | 30.00 $\pm$1.93 | —– | —– |
| \rowcolorgray!22 SDXL-Turbo (Sauer et al., 2025) | 27.38 $\pm$2.69 | 20.31 $\pm$1.95 | 2.12 $\pm$0.83 | 2.56 $\pm$0.55 |
| + ReNO (Eyring et al., 2024) | 41.88 $\pm$1.03 | 27.50 $\pm$0.68 | 5.31 $\pm$0.38 | 4.69 $\pm$1.25 |
| + TokenOpt (Zafar et al., 2024) | 35.12 $\pm$0.75 | 23.31 $\pm$1.66 | —– | 3.94 $\pm$0.72 |
| + *D2D* w/ YOLOv9 (Ours) | 52.75 $\pm$1.55 | 36.69 $\pm$2.40 | 6.25 $\pm$1.77 | 7.50 $\pm$1.06 |
| + *D2D* w/ OWLv2 (Ours) | 55.62 $\pm$2.72 | 43.69 $\pm$2.36 | 9.81 $\pm$0.97 | 9.94 $\pm$1.57 |
| \rowcolorgray!22 SD2.1 (Rombach et al., 2022) | 32.75 $\pm$1.32 | 24.75 $\pm$2.85 | 4.81 $\pm$1.23 | 2.94 $\pm$0.75 |
| \rowcolorgray!22 SD1.4 (Rombach et al., 2022) | 27.62 $\pm$4.11 | 16.69 $\pm$2.59 | 2.81 $\pm$0.31 | 2.12 $\pm$0.32 |
| + Counting Guidance (Kang et al., 2025) | 28.38 $\pm$1.11 | 17.12 $\pm$1.69 | 3.38 $\pm$1.16 | 1.88 $\pm$0.60 |
| \rowcolorgray!22 SD-Turbo (Rombach et al., 2022) | 20.88 $\pm$3.07 | 15.31 $\pm$0.87 | 2.56 $\pm$0.83 | 3.19 $\pm$1.18 |
| + ReNO (Eyring et al., 2024) | 43.38 $\pm$3.47 | 32.06 $\pm$0.99 | 8.94 $\pm$1.76 | 4.25 $\pm$1.14 |
| + *D2D* w/ OWLv2 (Ours) | 48.38 $\pm$3.09 | 39.44 $\pm$2.37 | 10.75 $\pm$1.06 | 11.44 $\pm$1.98 |
| \rowcolorgray!22 Pixart-$\alpha$ (Rombach et al., 2022) | 19.62 $\pm$1.03 | 14.00 $\pm$1.08 | 1.31 $\pm$0.75 | 1.81 $\pm$0.66 |
| \rowcolorgray!22 Pixart-DMD (Chen et al., 2025b) | 38.12 $\pm$2.32 | 27.88 $\pm$1.51 | 6.25 $\pm$0.46 | 3.19 $\pm$0.62 |
| + ReNO (Eyring et al., 2024) | 44.75 $\pm$1.44 | 37.25 $\pm$1.70 | 9.44 $\pm$0.75 | 4.75 $\pm$0.74 |
| + *D2D* w/ OWLv2 (Ours) | 53.25 $\pm$2.40 | 41.25 $\pm$2.81 | 13.31 $\pm$1.36 | 7.62 $\pm$1.18 |

### 4.2 Numeracy improvements

Tab. [1](https://arxiv.org/html/2510.19278v1#S4.T1) shows our main *D2D*-to-baseline comparisons. Baseline evaluations illustrate that though the prompt setting is relatively simple, generating accurate counts remains challenging. On numeracy, *D2D* consistently outperforms baselines across low-density, single-object, multi-object, and high-density prompts, across base models with U-Net and DiT backbones. On SDXL-Turbo, we demonstrate that performance boosts from *D2D* generalize across OWLv2 and YOLOv9 detector backbones (i.e., the detector used to compute $\mathcal{L}_{\text{D2D}}$), with a small accuracy-cost tradeoff as expected (Fig. [6(b)](https://arxiv.org/html/2510.19278v1#S4.F6.sf2)). The robust OWLv2 detector yields higher numeracy with slightly more overhead, while the real-time YOLOv9 detector yields slightly lower (but still high) numeracy with faster inference (in all other experiments, we use the higher-performing OWLv2 backbone unless otherwise noted). Additionally, *D2D* effectively complements baselines, boosting numeracy across all four benchmarks when used in combination with TokenOpt or ReNO (Tab. [6](https://arxiv.org/html/2510.19278v1#A6.T6) in appendix). For example, applying *D2D*-optimized seeds to TokenOpt improves numeracy by 13.63% points, relative to TokenOpt’s baseline performance (from 35.12% to 48.75%) on CoCoCount.

Improved numeracy on multi-object/high-density prompts.
*D2D* maintains relative improvement over baselines even in the more challenging multi-object/high-density settings. Nevertheless, the accuracy drop from low-density benchmarks to D2D-Large illustrates the remaining challenge of correctly generating large counts (e.g., base SDXL-Turbo: from 43.69% on D2D-Small to 9.94% on D2D-Large). Unsurprisingly, upon parsing D2D-Multi results, we see this holds within multi-object prompts as well (Tab. [7](https://arxiv.org/html/2510.19278v1#A7.T7) in appendix). For example, the accuracy of SDXL-Turbo + *D2D* w/ OWLv2 on D2D-Multi prompts with low total-density ($N_{\text{tot}}=N_{1}+N_{2}\leq 10$) is 12.08%, which drops to 3.00% for prompts with higher $N_{\text{tot}}$ (though both are still higher than all baseline scores).

$\bm{\mathcal{L}_{\textbf{D2D}}}$ effectively boosts numeracy across all classes. Fig. [4](https://arxiv.org/html/2510.19278v1#S4.F4) shows $\mathcal{L}_{\text{D2D}}$ improves numeracy across all 41 object categories in CoCoCount and D2D-Small, spanning a large variety (e.g., apples, elephants, cars, etc.) Upon applying D2D to SDXL-Turbo, umbrella and vase are the two classes that see the most improvement, each jumping from 2.50% (SDXL-Turbo base) to 52.50% (*D2D*) accuracy. Wine glass and bottle, both of which are (semi)transparent objects, are among the classes that see the least improvement (47.50% to 52.50% and 25.00% to 30.00% accuracy, respectively), which may suggest a future direction where detectors are fine-tuned on more difficult classes, or similar, with the purpose of generating highly-tailored scenes of objects.

Figure: Figure 4: *D2D* improves numeracy on the majority of the 41 objects in CoCoCount and D2D-Small. Evaluated against ReNO (Eyring et al., 2024) and TokenOpt (Zafar et al., 2024) on base SDXL-Turbo. Avg. over four seeds.
Refer to caption: x3.png

Figure: Table 2: Given the same initial conditions, *D2D* is effective at correcting both over and under-generation. We report the correction rate of initial over/under-generations, as well as the proportion of correct generations that were maintained. On SDXL-Turbo, across CoCoCount and D2D-Small benchmarks. Avg. over four seeds.

*D2D* best handles over and under-generation. Tab. [2](https://arxiv.org/html/2510.19278v1#S4.T2) breaks down results by the numeracy of the initial generation $I$, illustrating how well different methods are able to correct over/under-generation while maintaining the numeracy of already-correct images. Specifically, we compare TokenOpt, ReNO, and *D2D* on base model SDXL-Turbo, across benchmarks CoCoCount and D2D-Small. *D2D* has the highest correction rate, correcting 40.13% of over-generations and 41.83% of under-generations, which is at least 16% points over the baselines, while maintaining 72.57% of already-correct generations, which is also more than both TokenOpt and ReNO. Fig. [5](https://arxiv.org/html/2510.19278v1#S4.F5) illustrates *D2D*’s iterative correction process on two sample prompts, going from 16 dogs to the requested 10 dogs and from three cups to the requested four. Additional qualitative examples in Appendix [J](https://arxiv.org/html/2510.19278v1#A10).

### 4.3 Additional analysis and ablations

Impact of hyperparameters.
We report our key hyperparameter studies on values for $\tau$ (detector threshold) and $\beta$ (steepness coefficient). Studies of hyperparameters were conducted using base model SDXL-Turbo on benchmark CoCoCount (seed=0) on an A6000/L40. Results (Tab. [3](https://arxiv.org/html/2510.19278v1#S4.T3)) show that $\tau=0.2$, $\beta=300$ are optimal; we use these values in all other experiments.

**Table 4: Among count critics, $\bm{\mathcal{L}_{\textbf{D2D}}}$ is the most effective. On SDXL-Turbo. Avg. over four seeds.**
| Count Critic | CoCoCount | D2D-Small | D2D-Multi | D2D-Large |
| --- | --- | --- | --- | --- |
| RCC (Hobley & Prisacariu, 2022) | 37.75 | 26.38 | —– | 04.25 |
| CounTR (Chang et al., 2022) | 38.38 | 25.62 | —– | 05.31 |
| CLIP-Count (Jiang et al., 2023) | 40.00 | 25.88 | 05.19 | 06.38 |
| $f$ (OWLv2) | 32.00 | 20.75 | 03.06 | 03.38 |
| $\mathcal{L}_{\text{D2D}}$ (OWLv2) | 55.62 | 43.69 | 09.81 | 09.94 |

**Table 5: The LMN boosts numeracy. We compare *D2D* against ReNO (Eyring et al., 2024) using $\mathcal{L}_{\text{D2D}}$ and $\mathcal{L}_{\text{reg}}^{\prime}$ for both, controlling for the number of iterations tuned. We note boosts in numeracy, with comparable image quality. On SDXL-Turbo. Avg. over four seeds.**
| Method | CountGD $\uparrow$ |  | ImageReward $\uparrow$ |  | PickScore $\uparrow$ |  | HPSv2 $\uparrow$ |  | CLIPScore $\uparrow$ |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  |
| ReNO w/ $\mathcal{L}_{\text{D2D}}$, $\mathcal{L}_{\text{reg}}^{\prime}$ | 43.25 | 32.00 |  | 1.04 | 0.45 |  | 23.25 | 21.98 |  | 0.296 | 0.281 |  | 32.81 | 31.79 |
| *D2D* w/ $\mathcal{L}_{\text{D2D}}$, $\mathcal{L}_{\text{reg}}^{\prime}$ | 53.88 | 42.44 |  | 1.08 | 0.52 |  | 23.28 | 21.99 |  | 0.299 | 0.282 |  | 32.77 | 31.71 |

Figure: (a) Image quality/alignment scores (ImageReward (Xu et al., 2023), PickScore (Kirstain et al., 2023), HPSv2 (Wu et al., 2023), CLIPScore (Hessel et al., 2021)) by method. Aside from ReNO (Eyring et al., 2024), which often scores highest (it specifically optimizes those metrics), *D2D* is comparable to counting baselines. Min-max normalized.
Refer to caption: figures/quality.png

D2D vs. regression-based counters.
Tab. [4](https://arxiv.org/html/2510.19278v1#S4.T4) compares the effectiveness of our critic against existing regression-based ones and additionally shows that the formulation $\mathcal{L}_{\text{D2D}}$ is indeed more convergence-friendly than $f_{\beta,\tau_{z}}$. Across all four benchmarks, our detector-based critic outperforms regression-based methods RCC, CLIP-Count, and CounTR on numeracy (e.g., ours reaches 55.62% when the max score reached by any regression-based model is 40% on CoCoCount). Notably, $\mathcal{L}_{\text{D2D}}$ outperforms the others even on the high-density benchmark D2D-Large, though regression-based methods outperform detectors in the non-generative, counting setting (Fig. [2(b)](https://arxiv.org/html/2510.19278v1#S1.F2.sf2)). Furthermore, not only does $\mathcal{L}_{\text{D2D}}$, which produces a stronger gradient signal, outperform $f_{\beta,\tau_{z}}$ on numeracy; $f_{\beta,\tau_{z}}$ yields the lowest numeracy, which indicates that though it composes the mathematical backbone of $\mathcal{L}_{\text{D2D}}$, $f_{\beta,\tau_{z}}$ itself is not a suitable critic, as expected (Tab. [4](https://arxiv.org/html/2510.19278v1#S4.T4)).

The Latent Modifier Network $\bm{M_{\phi}}$.
Next, we assess the impact of introducing the LMN, a module whose output is mixed with the original noise to arrive at the optimal noise, by comparing our method with ReNO’s, controlling for the optimization objectives used ($\mathcal{L}_{\text{D2D}}$, $\mathcal{L}_{\text{reg}}^{\prime}$) and number of iterations tuned. Tab. [5](https://arxiv.org/html/2510.19278v1#S4.T5) shows the LMN generally improves numeracy while maintaining image quality; numeracy jumps 10% points on CoCoCount and D2D-Small from 43.25% to 53.88% and from 32.00% to 42.44%, respectively. Fig. [13](https://arxiv.org/html/2510.19278v1#A11.F13) in the appendix, which breaks down the numeracy by the absolute error between the requested and generated count in the initial generation, shows the LMN boosts the correction rate across initial generations.

Impact on image quality and computational overhead.
ImageReward, PickScore, HPSv2, and CLIPScore metrics in Fig. [6(a)](https://arxiv.org/html/2510.19278v1#S4.F6.sf1) show *D2D*’s image quality and overall prompt alignment is comparable to counting baselines and even surpasses multi-step baselines in many cases, including the layout control-based method, Make It Count (MIC). For example, SDXL-Turbo + *D2D* (OWLv2) yields ImageReward 0.51 (MIC: 0.30), PickScore 21.98 (MIC: 21.48), and HPSv2 0.28 (MIC: 0.26) on D2D-Small. *D2D* does not add significantly to inference cost, averaging between 11 and 21 seconds, compared to counting baselines, which average upwards of 28 to 100 seconds (Fig. [6(b)](https://arxiv.org/html/2510.19278v1#S4.F6.sf2)).

## 5 Conclusion and discussion

In this work, we address the challenge of correcting numeracy in generation. We identify a central limitation of previous methods, specifically their reliance on differentiable, regression-based counting models as critics. We propose *D2D*, novel way to convert more robust detectors into differentiable count critics, which we then use to optimize the initial noise at inference-time to improve numeracy. Our method yields the highest numeracy across prompt scenarios, including low-density, single-object, multi-object, high-density settings, effectively correcting both over and under-generation, with minimal additions to temporal overhead and minimal degradation in image quality.

Limitation and future directions. While our method exhibits significant improvements in numeracy, high-density scenarios remain challenging. Given regression-based methods are more appropriate in this setting, a future direction may explore how to adapt them into the generative setting. *D2D* is not a layout-control approach and so is limited in more fine-grained control (e.g., object placement). But future directions may explore using *D2D* to perform more complex tasks, such as object positioning and attribute binding, by leveraging detectors that can robustly work with prompts specifying objects and associated attributes.

#### Reproducibility Statement

The paper and appendix, along with code which we will release, contain the details for reproducibility.

#### Acknowledgments

This research was partially supported by an Amazon Research Award.
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of Amazon. We are also grateful to Princeton Research Computing for compute resources.

## References

- Acharya et al. (2019)
Manoj Acharya, Kushal Kafle, and Christopher Kanan.
Tallyqa: Answering complex counting questions.
*Proceedings of the AAAI Conference on Artificial Intelligence*, 33(01):8076–8084, Jul. 2019.
doi: 10.1609/aaai.v33i01.33018076.
- Amini-Naieni et al. (2024)
Niki Amini-Naieni, Tengda Han, and Andrew Zisserman.
Countgd: Multi-modal open-world counting.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 48810–48837. Curran Associates, Inc., 2024.
- Binyamin et al. (2025)
Lital Binyamin, Yoad Tewel, Hilit Segev, Eran Hirsch, Royi Rassin, and Gal Chechik.
Make it count: Text-to-image generation with an accurate number of objects.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 13242–13251, June 2025.
- Black et al. (2024)
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine.
Training diffusion models with reinforcement learning.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Chang et al. (2022)
Liu Chang, Zhong Yujie, Zisserman Andrew, and Xie Weidi.
Countr: Transformer-based generalised visual counting.
In *British Machine Vision Conference (BMVC)*, 2022.
- Chefer et al. (2023)
Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or.
Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models.
*ACM Trans. Graph.*, 42(4), July 2023.
ISSN 0730-0301.
doi: 10.1145/3592116.
- Chen et al. (2025a)
Chaofeng Chen, Annan Wang, Haoning Wu, Liang Liao, Wenxiu Sun, Qiong Yan, and Weisi Lin.
Enhancing diffusion models with text-encoder reinforcement learning.
In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), *Computer Vision – ECCV 2024*, pp. 182–198, Cham, 2025a. Springer Nature Switzerland.
ISBN 978-3-031-72698-9.
- Chen et al. (2024)
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\alpha$: Fast training of diffusion transformer for photorealistic text-to-image synthesis.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Chen et al. (2025b)
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li.
Pixart-$\Sigma$: Weak-to-strong training of diffusion transformer for 4k text-to-image generation.
In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), *Computer Vision – ECCV 2024*, pp. 74–91, Cham, 2025b. Springer Nature Switzerland.
ISBN 978-3-031-73411-3.
- Chung et al. (2024)
Hyungjin Chung, Jeongsol Kim, Geon Yeong Park, Hyelin Nam, and Jong Chul Ye.
Cfg++: Manifold-constrained classifier free guidance for diffusion models.
*CoRR*, abs/2406.08070, 2024.
- Clark et al. (2024)
Kevin Clark, Paul Vicol, Kevin Swersky, and David J. Fleet.
Directly fine-tuning diffusion models on differentiable rewards.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Dinh et al. (2023)
Anh-Dung Dinh, Daochang Liu, and Chang Xu.
Rethinking conditional diffusion sampling with progressive guidance.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 42285–42297. Curran Associates, Inc., 2023.
- Eyring et al. (2024)
Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexey Dosovitskiy, and Zeynep Akata.
Reno: Enhancing one-step text-to-image models through reward-based noise optimization.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), *Advances in Neural Information Processing Systems*, volume 37, pp. 125487–125519. Curran Associates, Inc., 2024.
- Fan et al. (2023)
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee.
Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 79858–79885. Curran Associates, Inc., 2023.
- Hessel et al. (2021)
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi.
CLIPScore: A reference-free evaluation metric for image captioning.
In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), *Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, pp. 7514–7528, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
doi: 10.18653/v1/2021.emnlp-main.595.
- Hobley & Prisacariu (2022)
Michael Hobley and Victor Prisacariu.
Learning to count anything: Reference-less class-agnostic counting with weak supervision.
*arXiv preprint arXiv:2205.10203*, 2022.
- Jiang et al. (2023)
Ruixiang Jiang, Lingbo Liu, and Changwen Chen.
Clip-count: Towards text-guided zero-shot object counting.
In *Proceedings of the 31st ACM International Conference on Multimedia*, MM ’23, pp. 4535–4545, New York, NY, USA, 2023. Association for Computing Machinery.
ISBN 9798400701085.
doi: 10.1145/3581783.3611789.
- Kang et al. (2025)
Wonjun Kang, Kevin Galim, Hyung Il Koo, and Nam Ik Cho.
Counting guidance for high fidelity text-to-image synthesis.
In *2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pp. 899–908, 2025.
doi: 10.1109/WACV61041.2025.00097.
- Kirstain et al. (2023)
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy.
Pick-a-pic: An open dataset of user preferences for text-to-image generation.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 36652–36663. Curran Associates, Inc., 2023.
- Lian et al. (2024)
Long Lian, Boyi Li, Adam Yala, and Trevor Darrell.
LLM-grounded diffusion: Enhancing prompt understanding of text-to-image diffusion models with large language models.
*Transactions on Machine Learning Research*, 2024.
ISSN 2835-8856.
Featured Certification.
- Lin et al. (2014)
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.
Microsoft coco: Common objects in context.
In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), *Computer Vision – ECCV 2014*, pp. 740–755, Cham, 2014. Springer International Publishing.
ISBN 978-3-319-10602-1.
- Liu et al. (2025)
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang.
Grounding dino: Marrying dino with grounded pre-training for open-set object detection.
In *Computer Vision – ECCV 2024*, pp. 38–55, 2025.
- Minderer et al. (2023)
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby.
Scaling open-vocabulary object detection.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 72983–73007. Curran Associates, Inc., 2023.
- Patel & Serkh (2025)
Zakaria Patel and Kirill Serkh.
Enhancing image layout control with loss-guided diffusion models.
In *2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pp. 3916–3924, 2025.
doi: 10.1109/WACV61041.2025.00385.
- Peebles & Xie (2023)
William Peebles and Saining Xie.
Scalable diffusion models with transformers.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 4195–4205, October 2023.
- Podell et al. (2024)
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach.
SDXL: Improving latent diffusion models for high-resolution image synthesis.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Ranjan et al. (2021)
Viresh Ranjan, Udbhav Sharma, Thu Nguyen, and Minh Hoai.
Learning to count everything.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 3394–3403, June 2021.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 10684–10695, June 2022.
- Ronneberger et al. (2015)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox.
U-net: Convolutional networks for biomedical image segmentation.
In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), *Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015*, pp. 234–241, Cham, 2015. Springer International Publishing.
ISBN 978-3-319-24574-4.
- Sauer et al. (2025)
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach.
Adversarial diffusion distillation.
In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (eds.), *Computer Vision – ECCV 2024*, pp. 87–103, Cham, 2025. Springer Nature Switzerland.
ISBN 978-3-031-73016-0.
- Wallace et al. (2023)
Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Naik.
End-to-end diffusion latent optimization improves classifier guidance.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 7280–7290, October 2023.
- Wallace et al. (2024)
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik.
Diffusion model alignment using direct preference optimization.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 8228–8238, June 2024.
- Wang et al. (2024)
Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao.
Yolov9: Learning what you want to learn using programmable gradient information.
In *Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XXXI*, pp. 1–21, Berlin, Heidelberg, 2024. Springer-Verlag.
ISBN 978-3-031-72750-4.
doi: 10.1007/978-3-031-72751-1_1.
- Wang et al. (2025)
Ruoyu Wang, Huayang Huang, Ye Zhu, Olga Russakovsky, and Yu Wu.
The silent assistant: Noisequery as implicit guidance for goal-driven image generation.
In *ICCV*, 2025.
- Wu et al. (2023)
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li.
Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis.
*arXiv preprint arXiv:2306.09341*, 2023.
- Xu et al. (2023)
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong.
Imagereward: Learning and evaluating human preferences for text-to-image generation.
In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), *Advances in Neural Information Processing Systems*, volume 36, pp. 15903–15935. Curran Associates, Inc., 2023.
- Yang et al. (2024)
Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li.
Using human feedback to fine-tune diffusion models without any reward model.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pp. 8941–8951, June 2024.
- Zafar et al. (2024)
Oz Zafar, Lior Wolf, and Idan Schwartz.
Iterative object count optimization for text-to-image diffusion models.
*arXiv preprint arXiv:2408.11721*, 2024.
- Zhang et al. (2025)
Ruisu Zhang, Yicong Chen, and Kangwook Lee.
Improving CLIP counting accuracy via parameter-efficient fine-tuning.
*Transactions on Machine Learning Research*, 2025.
ISSN 2835-8856.

## Appendix

In Appendix [A](https://arxiv.org/html/2510.19278v1#A1), we discuss the broader impacts of our work in downstream uses. In Appendix [B](https://arxiv.org/html/2510.19278v1#A2), we note our use of LLMs as assistant. Appendix [C](https://arxiv.org/html/2510.19278v1#A3) details the algorithms for pre-inference alignment of the Latent Modifier Network (LMN) and inference-time, numeracy optimization algorithms. Appendix [D](https://arxiv.org/html/2510.19278v1#A4) details hyperparameters used, including for the LMN architecture (Appendix [D.1](https://arxiv.org/html/2510.19278v1#A4.SS1)), core *D2D* formulation (Appendix [D.2](https://arxiv.org/html/2510.19278v1#A4.SS2)), and optimization (Appendix [D.3](https://arxiv.org/html/2510.19278v1#A4.SS3)). Appendix [E](https://arxiv.org/html/2510.19278v1#A5) discusses the implementation details of extending *D2D* to multi-object scenarios. Appendix [F](https://arxiv.org/html/2510.19278v1#A6) shows the performance improvements that result from using *D2D* to complement existing baselines. Appendix [G](https://arxiv.org/html/2510.19278v1#A7) shows a breakdown of numeracy by total density on D2D-Multi. Appendix [H](https://arxiv.org/html/2510.19278v1#A8) shows CountGD’s baseline counting performance, relative to other regression/detector-based counters. Appendix [I](https://arxiv.org/html/2510.19278v1#A9) compares *D2D*’s image quality to base model SDXL-Turbo’s. Additional qualitative results are illustrated in Appendix [J](https://arxiv.org/html/2510.19278v1#A10). Appendix [K](https://arxiv.org/html/2510.19278v1#A11) includes results from additional ablation experiments, including studies on the value of mixing weight $w$ used to mix the LMN output with the original noise (Appendix [K.1](https://arxiv.org/html/2510.19278v1#A11.SS1)); numeracy breakdown by requested count using $\mathcal{L}_{\text{D2D}}$ vs. regression-based count critics (Appendix [K.2](https://arxiv.org/html/2510.19278v1#A11.SS2)); correction rate by numeracy of initial generation with the introduction of the LMN (Appendix [K.3](https://arxiv.org/html/2510.19278v1#A11.SS3)); the necessity of inference-time calibration of the LMN (Appendix [K.4](https://arxiv.org/html/2510.19278v1#A11.SS4)), and the regularization formulation used (Appendix [K.5](https://arxiv.org/html/2510.19278v1#A11.SS5)).

## Appendix A Broader impacts

As a text-to-image pipeline, our method has many practical downstream uses, so it is necessary to exercise caution in deployment. Our method offers the advantage of stronger numeracy, which may be desirable in applications where users need to generate specific counts of objects. Our model may inherit biases of pre-trained diffusion base models and detectors. We suggest using strict NSFW filters and building more robust detectors.

## Appendix B Use of Large Language Models (LLMs)

We used LLMs to help with word choice for clarity and debugging.

## Appendix C Algorithms

Figure: Algorithm 1 Pre-inference alignment stage (done once per model)

Figure: Algorithm 2 Inference

## Appendix D Hyperparameters

### D.1 Architecture of the Latent Modifier Network

The LMN is a three-layer perceptron, with input/output layers of size $d$ (dimension of initial latent, $\mathbf{x}_{T}\in\mathbb{R}^{d}$), and two hidden layers of size 100 each. On base models SDXL-Turbo, SD-Turbo, and Pixart-DMD, the LMN has 3,303,384 tunable parameters.

### D.2 Core D2D hyperparameters

The core *D2D* hyperparameters are the detector threshold $\tau$ and steepness coefficient $\beta$, which determine the transition threshold and curvature of the sigmoid underlying $\mathcal{L}_{\text{D2D}}$, and the mixing weight $w$, which determines the ratio with which to combine the LMN output and original noise to obtain the optimal noise. We use $\tau=0.2$, $\beta=300$, and $w=0.2$. Studies on hyperparameters $\tau$ and $\beta$ are reported in Tab. [3](https://arxiv.org/html/2510.19278v1#S4.T3) in Sec. [4.3](https://arxiv.org/html/2510.19278v1#S4.SS3) of the main text. Experiments on $w$ are reported in Tab. [9](https://arxiv.org/html/2510.19278v1#A11.T9) in Appendix [K.1](https://arxiv.org/html/2510.19278v1#A11.SS1).

### D.3 Optimization

During pre-inference alignment and inference-time calibration, when we optimize only the regularization term, we use $\mathcal{L}_{\text{reg}}^{\prime}$. At inference-time, during numeracy optimization, we use $\mathcal{L}_{\text{reg}}$ in conjunction with $\mathcal{L}_{\text{D2D}}$. Appendix [K.5](https://arxiv.org/html/2510.19278v1#A11.SS5) reports ablation studies on using $\mathcal{L}_{\text{reg}}^{\prime}$ vs. $\mathcal{L}_{\text{reg}}$ during numeracy optimization.

In the pre-inference alignment stage, we use learning rate $\eta=10^{-4}$ and loss weight $\lambda=0.01$. At inference-time (stage 1: calibration), we use minimum number of iterations $t_{\text{min}}=70$; “good enough” threshold for determining when calibration is done $\tau_{\text{reg}}=-712.8$, learning rate $\eta_{\text{calib}}=10^{-3}$, and loss weight $\lambda_{\text{calib}}=0.01$. During stage 2 (numeracy optimization), we use learning rate $\eta=5\times 10^{-4}$ and loss weights $\alpha=5$ and $\lambda=10^{-4}$. During this stage, we use loss-based, adaptive learning rate scheduling of $\eta$ to ease steady convergence when the generated count approximates the requested count; we also adaptively rescale (i.e., increase) $\lambda$ to counteract larger deviations from Gaussian (i.e., divergence).

For *D2D* experimental results with Pixart-DMD on benchmark D2D-Large, we use the base detector OWLv2, instead of $f_{\beta,\tau_{x}}$, to perform early-stopping. For single-object scenarios, we set $K=200$, except for *D2D* with Pixart-DMD on D2D-Large, for which we use $K=400$. For the multi-object scenario, we use $K=400$.

## Appendix E Multi-object ℒ D2D \mathcal{L}_{\text{D2D}}

In extending *D2D* to prompts with $m>1$ object classes $\{C_{j},1\leq j\leq m\}$, we have to note that every predicted bbox $B_{i}$ now comes with $m$ corresponding scores $z_{i}^{(1)},z_{i}^{(2)},...,z_{i}^{(m)}$, the max of which, $z_{i}^{\text{max}}$, indicates its corresponding label. Only the bboxes where $z_{i}^{\text{max}}\geq\tau_{z}$ are counted. To correct over/under-generation, our approach then is to focus on appropriately adjusting the largest score $z_{i}^{\text{max}}$ of each bbox, while minimizing all other (i.e. non-max) scores. This means minimizing the $z_{i}^{\text{max}}$ of bboxes that correspond to over-generated classes and maximizing the $z_{i}^{\text{max}}$ of those that correspond to under/correctly generated classes (to increase/maintain the count). We use roughly the same formulation as Eq. [2](https://arxiv.org/html/2510.19278v1#S3.E2) in the main text, with $z_{i}$ replaced by the specific per-class, per-bbox logits $z_{i}^{j}$ that need to be minimized or maximized (Eq. [6](https://arxiv.org/html/2510.19278v1#A5.E6)).

If ${\mathbb{C}}_{\geq}$ is the set of classes that are under-generated or correctly generated, then

$$ ${\mathbb{S}}_{\text{bbox}}=\{(i,j)|1\leq i\leq n\land j=(\arg\max_{j^{\prime}}z_{i}^{j^{\prime}})\in C_{\geq}\}$ $$

refers to the set of the bboxes corresponding to the under/correctly generated classes, along with their max logits. We compute the loss $\mathcal{L}_{\text{D2D}}$ like so:

$$ $\mathcal{L}_{\text{D2D}}^{\text{multi}}=\sum_{(i,j)\in{\mathbb{S}}_{\text{bbox}}}\sigma(\beta\cdot(\tau_{z}-z_{i}^{j}))\cdot(\tau_{z}-z_{i}^{j})+\sum_{(i,j)\notin{\mathbb{S}}_{\text{bbox}}}\sigma(\beta\cdot(z_{i}^{j}-\tau_{z}))\cdot(z_{i}^{j}-\tau_{z})$ (6) $$

## Appendix F D2D yields boosts in numeracy, in complement with baselines

**Table 6: D2D additionally yields improvements, in combination with T2I counting/enhancement baselines. On base models SDXL-Turbo, SD-Turbo, and Pixart-DMD; on benchmarks CoCoCount (Binyamin et al., 2025) and D2D-Small/Multi/Large. Avg. over four seeds.**
| Base Model | Method | CoCoCount | D2D-Small | D2D-Multi | D2D-Large |
| --- | --- | --- | --- | --- | --- |
| SDXL-Turbo | TokenOpt (Zafar et al., 2024) | 35.12 | 23.31 | —– | 3.94 |
| TokenOpt (Zafar et al., 2024) + *D2D* | 48.75 | 34.00 | —– | 8.56 |  |
| ReNO (Eyring et al., 2024) | 41.88 | 27.50 | 5.31 | 4.69 |  |
| ReNO (Eyring et al., 2024) + *D2D* | 54.38 | 41.25 | 9.69 | 10.38 |  |
| SD-Turbo | ReNO (Eyring et al., 2024) | 43.38 | 32.06 | 8.94 | 4.25 |
| ReNO (Eyring et al., 2024) + *D2D* | 53.62 | 42.44 | 11.62 | 10.19 |  |
| Pixart-DMD | ReNO (Eyring et al., 2024) | 44.75 | 37.25 | 9.44 | 4.75 |
| ReNO (Eyring et al., 2024) + *D2D* | 52.50 | 40.62 | 12.00 | 8.12 |  |

## Appendix G Density-based differential performance persists in D2D-Multi

**Table 7: *D2D* yields higher performance on low total-density prompts than high total-density ones. Avg. over four seeds.**
| Method | D2D-Multi |  |
| --- | --- | --- |
| Low Total Density | High Total Density |  |
| \rowcolorgray!7 SDXL (Podell et al., 2024) | 3.08 | 0.50 |
| Make It Count (Binyamin et al., 2025) | —– | —– |
| \rowcolorgray!7 SDXL-Turbo (Sauer et al., 2025) | 2.75 | 0.25 |
| ReNO (Eyring et al., 2024) | 6.67 | 1.25 |
| TokenOpt (Zafar et al., 2024) | —– | —– |
| *D2D* w/ OWLv2 (Ours) | 12.08 | 3.00 |
| *D2D* w/ YOLOv9 (Ours) | 7.83 | 1.50 |
| \rowcolorgray!7 SD2.1 (Rombach et al., 2022) | 6.17 | 0.75 |
| \rowcolorgray!7 SD1.4 (Rombach et al., 2022) | 3.75 | 0.00 |
| Counting Guidance (Kang et al., 2025) | 4.42 | 0.25 |
| \rowcolorgray!7 SD-Turbo (Rombach et al., 2022) | 3.00 | 1.25 |
| ReNO (Eyring et al., 2024) | 11.58 | 1.00 |
| *D2D* w/ OWLv2 (Ours) | 13.42 | 2.75 |
| \rowcolorgray!7 Pixart-$\alpha$ (Rombach et al., 2022) | 1.67 | 0.25 |
| \rowcolorgray!7 Pixart-DMD (Chen et al., 2025b) | 8.00 | 1.00 |
| ReNO (Eyring et al., 2024) | 12.25 | 1.00 |
| *D2D* w/ OWLv2 (Ours) | 17.17 | 1.75 |

## Appendix H CountGD in comparison with other counters

GroundingDINO (Liu et al., 2025)-based counter CountGD (Amini-Naieni et al., 2024) performs well in low and high-density settings.

Figure: Figure 7: CountGD is a state-of-the-art counter, based on GroundingDINO (Liu et al., 2025).
Refer to caption: x4.png

## Appendix I D2D exhibits minimal degradation in image quality, compared to layout-control method Make It Count

Tab. [8](https://arxiv.org/html/2510.19278v1#A9.T8) reports the image quality of layout control-based, count-correction method Make It Count (Binyamin et al., 2025), its base model SDXL, and *D2D*, along with base model SDXL-Turbo. Make It Count exhibits some tradeoffs between numeracy and image quality, yielding slightly lower PickScore and HPSv2, compared to its base model SDXL (e.g., on D2D-Small: PickScore: 21.70 (SDXL), 21.48 (MIC); HPSv2: 0.272 (SDXL), 0.264 (MIC)).

*D2D* offers minimal image quality degradation (and sometimes better image quality) than base model SDXL-Turbo (e.g., on D2D-Small: ImageReward: 0.40 (SDXL-Turbo), 0.51 (*D2D* w/ OWLv2); HPSv2: 0.279 (SDXL-Turbo), 0.282 (*D2D* w/ OWLv2)).

**Table 8: D2D yields minimal degradation in image quality and alignment, with minimal computational overhead, compared to layout control-based, multi-step method Make It Count. Base models with no post-enhancement highlighted in gray. On CoCoCount and D2D-Small. Avg. over four seeds.**
| Method |  | ImageReward $\uparrow$ |  | PickScore $\uparrow$ |  | HPSv2 $\uparrow$ |  | CLIPScore $\uparrow$ |  | Inference Time (s) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  | CoCoCount | D2D-Small |  |  |  |
| \rowcolorgray!22 SDXL (Podell et al., 2024) |  | 0.87 | 0.31 |  | 22.99 | 21.70 |  | 0.290 | 0.272 |  | 32.91 | 32.58 |  | 7.69 |
| + Make It Count (Binyamin et al., 2025) |  | 0.91 | 0.30 |  | 22.83 | 21.48 |  | 0.286 | 0.264 |  | 32.92 | 32.96 |  | 37.16 |
| \rowcolorgray!22 SDXL-Turbo (Sauer et al., 2025) |  | 0.96 | 0.40 |  | 23.12 | 21.98 |  | 0.293 | 0.279 |  | 32.80 | 31.89 |  | 0.16 |
| + *D2D* (w/ OWLv2) |  | 1.06 | 0.51 |  | 23.30 | 21.98 |  | 0.300 | 0.282 |  | 32.84 | 31.69 |  | 19.42 |
| + *D2D* (w/ YOLOv9) |  | 1.06 | 0.50 |  | 23.33 | 22.04 |  | 0.300 | 0.282 |  | 32.88 | 31.90 |  | 11.39 |

## Appendix J Additional qualitative results

Additional qualitative results on prompts from benchmarks CoCoCount (Binyamin et al., 2025) and D2D-Small shown.

Figure: Figure 8: Our method *D2D* effectively corrects numeracy mistakes. Qualitative examples from count correction/alignment methods *D2D*, ReNO (Eyring et al., 2024), and TokenOpt (Zafar et al., 2024) on base model SDXL-Turbo (Sauer et al., 2025), and Make It Count (Binyamin et al., 2025) on base SDXL (Podell et al., 2024).
Refer to caption: figures/app_sdxl.png

Figure: Figure 9: Our method *D2D* effectively corrects numeracy mistakes. Qualitative examples from count correction/alignment methods *D2D* and ReNO (Eyring et al., 2024) on base model SD-Turbo, and Counting Guidance (Kang et al., 2025) on base SD1.4 (Rombach et al., 2022), along with SD2.1 (Rombach et al., 2022) shown.
Refer to caption: figures/app_sd.png

Figure: Figure 10: Our method *D2D* effectively corrects numeracy mistakes. Qualitative examples from count correction/alignment methods *D2D* and ReNO (Eyring et al., 2024) on base model Pixart-DMD, along with Pixart-$\alpha$ (Chen et al., 2024).
Refer to caption: figures/app_pix.png

## Appendix K Additional ablations

### K.1 w = 0.2 w=0.2 is optimal for numeracy and quality

$w$ is the weight used to determine how to mix the LMN output with the original noise. $w=0$ is numeracy-wise the best, followed by $w=0.2$, but results in patchy visual artifacts, relative to $w=0.2$ (Fig. [11](https://arxiv.org/html/2510.19278v1#A11.F11)). Hence, the hyperparameter value we use in our main experiments $w=0.2$.

**Table 9: In terms of just numeracy, $\bm{w=0}$ is optimal, followed by $\bm{w=0.2}$. Base model SDXL-Turbo, on CoCoCount. Seed = 0.**
|  |  | $\bm{w}$ |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | 0.0 | 0.2 | 0.5 | 0.8 |
| CountGD |  | 62.50 | 55.50 | 37.50 | 48.00 |

Figure: Figure 11: $\bm{w=0}$ results in patchy visual artifacts, relative to $\bm{w=0.2}$. So, we use $w=0.2$.
Refer to caption: figures/w_ablation.png

### K.2 Across requested counts, our count critic ℒ D2D \mathcal{L}_{\text{D2D}} is more effective than regression-based methods

Fig. [12](https://arxiv.org/html/2510.19278v1#A11.F12) reports the numeracy breakdown per requested count, comparing the performance of our count critic $\mathcal{L}_{\text{D2D}}$ against regression-based critics, RCC (Hobley & Prisacariu, 2022), CounTR (Chang et al., 2022), and CLIP-Count (Jiang et al., 2023), as well as $f$, which is the mathematical backbone of $\mathcal{L}_{\text{D2D}}$. Across all requested counts, our critic achieves the highest numeracy, peaking at 77% for $N=2$, whereas the highest score achieved by any regression-based method is 68%. $f$ scores lower, often even lower than regression-based methods, as expected, as it is not amenable to convergence.

Figure: Figure 12: Our detector-based critic $\bm{\mathcal{L}_{\textbf{D2D}}}$ is more effective than regression-based count critics. Given the same initial seed, compared to regression-based methods RCC (Hobley & Prisacariu, 2022), CounTR (Chang et al., 2022), and CLIP-Count (Jiang et al., 2023), our method yields the highest numeracy across requested counts. On SDXL-Turbo on benchmarks CoCoCount and D2D-Small. Avg. over four seeds.
Refer to caption: figures/acc_by_reqcount.png

### K.3 The Latent Modifier Network effectively corrects numeracy

Fig. [13](https://arxiv.org/html/2510.19278v1#A11.F13) reports the numeracy breakdown by absolute error between the requested and generated count in the initial generation, comparing the performance of *D2D* to ReNO (Eyring et al., 2024), using $\mathcal{L}_{\text{D2D}}$ and $\mathcal{L}^{\prime}_{\text{reg}}$ for both. The introduction of the LMN boosts numeracy across all initial absolute errors depicted.

Figure: Figure 13: The LMN effectively corrects numeracy. Breakdown of numeracy by absolute error between requested and generated count in the initial generation. Depicted on the x-axis are the subset of absolute errors in the range 1-10. Given the same initial seed, the addition of the LMN yields additional boosts in numeracy, compared to ReNO (Eyring et al., 2024). On SDXL-Turbo on benchmarks CoCoCount and D2D-Small. Avg. over four seeds.
Refer to caption: x5.png

### K.4 Inference-time alignment of M ϕ M_{\phi}

At inference-time, given a new $\mathbf{x}_{T}$, we allow a few iterations to calibrate $M_{\phi}$’s output to be close to Gaussian. Fig. [14](https://arxiv.org/html/2510.19278v1#A11.F14) shows this calibration stage is crucial to maintaining high image quality.

Figure: (a) Without alignment.
Refer to caption: figures/noalign.png

### K.5 Regularization

Our regularization term (Eq. [8](https://arxiv.org/html/2510.19278v1#A11.E8)) is a variant of the one ReNO uses (Eq. [7](https://arxiv.org/html/2510.19278v1#A11.E7)) that penalizes larger deviations from Gaussian more and smaller deviations less, to allow enough flexibility in the vicinity of the initial distribution to accommodate updates specified by the count critic. We scale $\mathcal{L}_{\text{reg}}^{\prime}$ by coefficient $a$ and shift by constant $c$, which we take to the power of 10.

$$ $\mathcal{L}_{\text{reg}}^{\prime}=\frac{||\mathbf{x}_{T}^{\prime}||^{2}}{2}-(d-1)\cdot\text{log}(||\mathbf{x}_{T}^{\prime}||),$ (7) $$

$$ $\mathcal{L}_{\text{reg}}=(a\mathcal{L}_{\text{reg}}^{\prime}+c)^{10}.$ (8) $$

We set coefficient $a$ to 0.03. Shift constant $c$ is set to 2139 to offset the optimal value of $a\mathcal{L}_{\text{reg}}^{\prime}$, which we find to be -2139 in practice.

Our ablation analysis confirms that this variant of the regularization leads to overall numeracy improvements across both CoCoCount and D2D-Small (Tab. [10](https://arxiv.org/html/2510.19278v1#A11.T10)), indicating our regularization objective allows for effective numeracy correction, even while enforcing a heavier penalty on non-Gaussian initial noises.

**Table 10: Our variant of regularization is beneficial to numeracy optimization. $\mathcal{L}_{\text{reg}}$, which penalizes larger deviations from Gaussian more and and smaller deviations less, yields higher numeracy on CoCoCount and D2D-Small benchmarks. On SDXL-Turbo. Avg. over four seeds.**
| Method | CoCoCount | D2D-Small |
| --- | --- | --- |
| *D2D* w/ $\mathcal{L}_{\text{reg}}^{\prime}$ | 53.88 | 42.44 |
| *D2D* w/ $\mathcal{L}_{\text{reg}}$ | 55.62 | 43.69 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - How does the LMN perform for human-preference rewards, e.g. ReNO?
- What's the effect of D2D on the diversity of generated images?
- What is the motivation behind the sharper noise regularization? How does it imact diversity?
- What's the effect of D2D on general image quality? This could be made more clear through adding baseline performances to Table 6
- Could D2D be also used to fine-tune a T2I model, e.g. with Adjoint Matching?
- Are there scenarios where the surrogate diverges from the true count?
- How many optimization steps are used? How important is the pre-inference alignment stage?
soundness: 2
strengths: - **Novel reward function design:** Converting non-differentiable detectors into differentiable counting rewards through steep sigmoids and logit scaling (Eq. 1-2) addresses a limitation where detectors outperform regression models in low-density counting but couldn't previously be used for gradient-based optimization.
- **Two complementary contributions:** The differentiable detector-based reward ($\mathcal{L}_{D2D}$) and the LMN architecture for reward optimization. Table 5 shows the LMN improves numeracy by 10% points over direct noise optimization when using the same reward, suggesting the LMN is a valuable contribution independent of the specific reward function.
- **Consistent empirical results:** Consistent improvements across multiple base models and settings in reasonable time-frames.
summary: This paper addresses numeracy in text-to-image models through reward-based inference-time optimization, making two main contributions. First, they design a differentiable counting reward $\mathcal{L}_{D2D}$ derived from object detectors, which are more accurate than regression-based counters in low-density scenarios but previously unusable as rewards due to non-differentiable enumeration. They achieve this through custom activation functions that convert detector logits into gradient-friendly signals. Second, they propose the Latent Modifier Network (LMN), a lightweight 3-layer MLP that transforms the initial noise for reward optimization. Unlike prior work that directly tunes the noise, the LMN provides a larger parameter space while preserving portions of the original noise through a weighted mixing scheme. Experiments show consistent improvements in counting accuracy.
weaknesses: - **Limited Discussion of General-Purpose Application**: The methodology is tailored for prompts containing numerical targets, but the paper does not discuss how the framework should behave with general, non-numeric prompts. It is unclear whether the D2D optimization is intended to be selectively activated, or what the default behavior might be in the absence of a numerical target (e.g., "few", "some", or "many"). How about very long complex prompts with lots of content. Additionally, it's not really clear what the effect of D2D is on aesthetics and diversity of generated images.
- **Justification for Reward Function Design Choices**: While the activation function design is intuitive, several design choices lack theoretical justification or empirical comparison to alternatives:
	- **Scaling formulation (Eq. 2):** The justification for multiplying by $(z_i - \tau_z)$ to combat sigmoid plateauing is limited. Did the authors explore other options, such as exponential scaling, learned scaling functions, or other monotonic transformations that could provide stronger gradients? The paper does not compare the chosen formulation against alternatives.
	- **Mixing weight:** Table 9 shows $w=0$ achieves the best numeracy but produces "patchy visual artifacts," suggesting a tradeoff between reward optimization and image quality. The basis for choosing $w=0.2$ could be better explained. More detailed analysis of this numeracy vs. image quality tradeoff in the main paper would be valuable.
	- **Noise regularization sharpening**: The paper proposes a much more sharped version of previous regularization in noise space by exponentiating it by a power of 10. To me it's unclear why this change was made, and not really motivated in the paper.
- **LMN Contribution Not Fully Explored**: The LMN appears to be a contribution independent of the counting reward (Table 5 demonstrates improvements over direct noise tuning), yet it receives limited investigation.
	- It is unclear whether the LMN improves optimization of other rewards beyond counting (e.g., ImageReward or similar like ReNO). Currently, Table 5 only compares LMN vs. direct noise tuning using $\mathcal{L}_{D2D}$. Evaluation with other rewards would help establish whether the LMN is a general contribution to inference-time reward optimization or specific to counting tasks.
	- The choice of a 3-layer MLP is not justified through ablations. It would be informative to explore variations such as 2 or 4 layers, different widths, or alternative architectures.

Minor Issues:
- Including the base model performance in Table 6 would facilitate easier comparison of how the reward-based optimization affects image quality metrics.
- D2D is proposed as a differentiable surrogate for the non-differentiable detector count, but no analysis validates this approximation. It would be informative to analyze how well $\mathcal{L}_{D2D}$ correlates with the actual detector count $D(I)$ throughout the optimization trajectory, and under what conditions the surrogate diverges from the true count (e.g., when many bboxes cluster near threshold $\tau_z$).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. (related to Table 1) Is the proposed method only applicable to single-step diffusion models? If so, is there a specific reason why it cannot be applied to multi-step diffusion models such as SDXL or Pixart-α? Applying the proposed method to SDXL would be particularly valuable, as it would enable direct comparisons with Make It Count.
soundness: 3
strengths: 1. The proposed method is logical and well-motivated, with clear explanations that make the underlying rationale and implementation easy to understand.

2. The analysis of the proposed framework is in-depth: for instance, the comparison between detector-based and regression-based counting models (Figure 2), the exploration of class-wise performance (Figure 4), and the ablation between direct noise optimization and LMN-based methods (Table 5).

3. The demonstrated generality of the proposed method across diverse diffusion backbones and evaluation scenarios is commendable.
summary: This paper tackles the problem of improving object numeracy in text-to-image diffusion models, which often fail to generate the correct number of objects specified in text prompts. 
The paper proposes Detector-to-Differentiable (D2D), a novel framework that converts non-differentiable object detectors into differentiable critics.  
By optimizing the initial noise during inference, D2D enhances count accuracy while preserving image quality. 
Experiments across multiple T2I backbones and benchmarks show meaningful improvement in object counting accuracy with minimal computational overhead.
weaknesses: 1. The evaluation relies solely on a detector-based protocol to assess the proposed detector-based critic. As noted around line 315, the paper employs the SOTA counting model CountGD (built upon GroundingDINO). Although the proposed critic uses different detectors such as OWL-ViT or YOLO for initial noise optimization, this setup risks giving an unfair advantage aligned with the evaluation criterion. To more robustly validate the superiority of the method, additional evaluation metrics, such as human evaluation (as adopted in Make It Count, CVPR 2025), should be considered.

2. It would be interesting to investigate whether the proposed noise optimization process restricts the diversity of generated images. Since the method optimizes the initial noise, the input noise distribution may become narrower than that of purely random noise, potentially reducing image diversity. Thus, analyzing generation diversity would provide important insights into possible trade-offs.

3. Table 6 omits the performance of the original backbone diffusion models (e.g., SDXL-Turbo, Pixart-DMD). While image quality comparisons with other counting baselines are provided, including the results of the original backbones would strengthen the claim that “D2D yields minimal degradation in image quality” (line 456).

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: In general, I'm mostly curious about how relevant this problem is to be tackled on a standalone basis with a somewhat expensive inference-time optimization formulation. In that regard, a) showing how much of an issue this is on recent, larger models, b) ability of the noise optimization formulation to work with at least the newer distilled models, c) whether these techniques could be used to enhance the base capabilities of this model (i.e fine-tuning a base model on a high-quality dataset of optimized samples etc.) would go a long way in strengthening the contributions of the paper. While I don't really have major doubts about the "soundness" or "presentation" of the paper, I would really like to see more evidence about the "contributions" of the paper before recommending acceptance for the paper.
soundness: 4
strengths: The biggest strength of the paper is in constructing an effective objective for differentiable optimization. This ensures that there's effective inference-time optimization, which is visible from the strong empirical results on several counting benchmarks. 

The paper is also well-presented and easy to follow.
summary: The paper presents an inference-time tuning strategy for improving the numeracy/counting of text-to-image models. The main idea is to solve an inference-time optimization process for the initial noise to optimize the count obtained from an object detector. From a technical standpoint, the main contributions are in a) creating a differentiable critic by applying a sigmoid activation over the logits b) modulating the initial noise with the "latent modifier network". Results on several counting benchmarks clearly indicate that this inference-time optimization is able to drastically improve the numeracy of these models.
weaknesses: Avoids more recent larger models: The results in the paper are on the SD-Turbo, SDXL-Turbo and Pixart-alpha models which are all not only distilled (therefore a bit worse in performance), but also fairly out of date as of late 2025 (all being released between late 2023-early 2024). While some of the newer models may not be applicable, one could still see results on Flux-Schnell, SANA-Sprint, SD3.5-Turbo to see how effective this optimization framework is on newer problems. While I'd expect the counting problem to still be there for newer/larger models, I'd imagine that it is a lot less of a problem with these models (e.g. Qwen-Image etc.)


Limited Contribution: The paper is essentially applying the existing initial noise optimization formulation for improved numeracy in T2I models (which had in the past either been used for toy objectives or general human preference/prompt alignment). While effective, from a conceptual standpoint it does feel somewhat limiting, especially since it inherits the issues of inference-time optimization (i.e slower runtimes, memory overhead etc.). To that extent, while the paper's contribution itself might be an integral part of a modern diffusion models' post-training pipeline (since it's easy to verify similar to the RL formulation in LLMs for math/code), it feels rather minimal to merit acceptance in my view.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 2
questions: - Inconsistency regarding High Object Counts: Figure 2b indicates that regression models begin to outperform detection models at higher object counts. However, Table 1 shows that your approach still outperforms the baselines in these same high-count scenarios. Could the authors please explain this apparent discrepancy?
- Computational Cost of $L_{\text{D2D}}$: Examining Algorithm 2, it appears that computing the loss term $L_{\text{D2D}}$ requires backpropagating through all diffusion steps. Could the authors elaborate on why this only has a minor effect on latency? Furthermore, what are the implications of this on memory usage during training and inference?
- Handling Duplicate Bounding Boxes: Detection models like YOLO typically utilize post-processing steps, such as Non-Maximum Suppression (NMS), to eliminate duplicate bounding boxes. How does your method handle the issue of potentially overlapping or duplicate detections?
soundness: 2
strengths: - The work demonstrates strong quantitative and qualitative performance against baselines, validated across multiple benchmarks and models, while also achieving low inference latency
summary: This paper addresses the challenge of diffusion models failing to generate the precise number of objects specified in a prompt. While many existing methods employ gradients of regression-based counting networks, the authors argue that detection models, such as YOLO, offer superior count accuracy. To overcome the non-differentiable nature of these detectors, they introduce a novel surrogate loss function. This loss enables optimization by updating the initial noise, a technique similar to that in Reno, rather than relying on traditional gradient guidance during the diffusion process.
weaknesses: - The novelty of this work appears limited, primarily consisting of replacing regression-based counting networks with detection models wrapped by a differentiable loss. Other parts of the methodology seem largely similar to previous work.
- Including Algorithm 2 from the appendix in the main paper would significantly aid in understanding the proposed algorithm.
- An explanation and analysis on why detection models are superior to regression models for this application would be very helpful and strengthen the paper's claims.
- Minor: The paper contains some grammar mistakes and typos that should be corrected throughout.


Decision: Reject