Sparse Image Synthesis via Joint Latent and RoI Flow
Authors: Ziteng Gao
Jay Zhangjie Wu
Mike Zheng Shou
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
flow matching
sparse diffusion
sparse generation
Primary area: deep_learning

Abstract: Natural images often exhibit underlying sparse structures, with information density varying significantly across different spatial locations. However, most generative models rely on dense grid-based pixels or latents, neglecting this inherent sparsity. In this paper, we explore modeling visual generation paradigm via sparse non-grid latent representations. Specifically, we design a sparse autoencoder that represents an image as a small number of latents with their positional properties (i.e., regions of interest, RoIs) with high reconstruction quality. We then explore training flow-matching transformers jointly on non-grid latents and RoI values. To the best knowledge, we are the first to address spatial sparsity using RoIs in generative process. Experimental results show that our sparse flow-based transformers have competitive performance compared with dense grid-based counterparts with significantly reduced lower compute, and reaches a competitive 2.76 FID with just 64 latents on class-conditional ImageNet $256\times 256$ generation.

Sparse Image Synthesis via Joint Latent and RoI Flow
Ziteng Gao
Jay Zhangjie Wu
Mike Zheng Shou"
Show Lab, National University of Singapore
Abstract
Natural images often exhibit underlying sparse structures, with information density varying significantly across different spatial locations. However, most generative models rely on dense grid-based pixels or latents, neglecting this inherent spar- sity. In this paper, we explore modeling visual generation paradigm via sparse non-grid latent representations. Specifically, we design a sparse autoencoder that represents an image as a small number of latents with their positional properties (i.e., regions of interest, Rols) with high reconstruction quality. We then explore training flow-matching transformers jointly on non- non-grid latents and Rol values. To the best knowledge, tive process. Experimental results show that our sparse flow-based transformers have competitive performance compared with dense grid-based counterparts with significantly reduced lower compute, and reaches a competitive 2.76 FID with just 64 latents class-conditional ImageNet 256 x 256 generation.
the first address spatial sparsity using Rols in genera-
Introduction
Deep visual generative models have advanced significantly recent years, achieving impressive visual quality on image 22. video [3, and 3D domains
[4]. Current visual generation pipelines typically start by encoding raw data (e.g., images) into compact latent representations with autoencoders, and then use diffusion, masking modeling, or autoregressive methods to generate such latents, as this pipeline exemplified by latent diffusion []. Flagship text-to-image models, e.g., Stable Diffusion and FLUX (6. follow this line and compress spatial dimensions at typically 8x factor, significantly lowering computational costs and modeling complexity in generative training.
Though being a core component of visual generation, autoencoders conventionally assume a grid- based space of latent structures with the uniform information density. However, natural images often exhibit highly non-uniform information density and require adaptive computation across spatial locations 7 g). For example, in a landscape image, the sky background occupies numerous pixels while being worth fewer latent units to reconstruct and generate. In contrast, intricate foreground objects may require more latents capture their details. Existing visual generation pipelines fail to address this point, as they rely on dense uniform grid-based latent structures and cannot adaptively allocate more computation to intricate foregrounds.
This paper aims to study this point. First, we propose a sparse visual autoencoder that learns to compress an image into a of sparse non-grid latents along with their positional property, i.c.. region of interests (RoIs), and then recover image pixels from them. The Rols explicitly characterize the spatial locations of the latents in bounding box formulation, and can be learned jointly with latents in an end-to-end manner by the plain reconstruction loss. The resulting sparse visual autoencoder reaches high compression rates by prioritizing latents detailed regions while maintaining high reconstruction fidelity. Then, we design sparse flow-based transformers to generate latents and Rols by modeling the joint flow of them with the velocity prediction in the denoising process 10 At
TTe corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

gri latent representation
noe-grid
representation
input image
regolar
generative modeling
sparse
latent Rol
generative modeling
Figure 1: Left: conventional autoencoders encode pixels into latent grid representations, Right: our method encodes them into fewer non-grid latents with region of interests (Rols).
every timestep, our model learns to estimate both latent and Rol velocity from initial noise to target samples. Divergent from prior grid-based latent approaches, our method dynamically adjusts latent spatial positions during sampling via ordinary differential equations (ODEs) at inference, allowing adaptive refinement of both content and spatial focus.
We show the feasibility of representing and generating images with sparse latents and Rols on the challenge ImageNet benchmark. Our proposed sparse flow autoencoder, SF-VAE, can represent 256 256 images with just 64 latents with 0.70 reconstruction FID, or even down 32 latents with 1.70 rFID. Then, the presented sparse flow-based transformers, SF-SiT's, have competitive performance on par with diffusion/flow-baeedd grid-based transformers [[] The largest SF-SiT, XL variant, can reach 2.76 FID with classifier-free guidance [12] on the class-conditional ImageNet generation benchmark with just 64 latents.
2 Related Work
Diffusion models. In recent years, generative models has been marked as a breakthrough in the field of visual synthesis [13 [4]. Commercial systems like DALL-E (2) or FLUX (6] are typically rooted in denoising diffusion architectures. The seminal work, denoising diffusion probabilistic models ([5), take the image generative process gradual denoising trajectory, iteratively refining pure noise into target images. Building on this foundation, subsequent advancements further accelerated and refined diffusion-based based generation. Improved variants including [16] 17 investigate the training and sampling trajectories, enabling high-quality results with fewer sampling steps. Latent diffusion models []] democratize the high resolution image synthesis by operating in compressed latent space with reduced computational costs. The following up work, including diffussionfflo-mmaccing transformers (18] TI TO), also follows this convention to speed up training. Although training diffusion models directly on raw pixels is technically feasible [19 20). the preference for latent space modeling stems from practical challenges: raw pixel data often contains high-frequency details and perceptually complex patterns that are computationally intensive and difficult for diffusion processes model effectively.
Latent space for diffusion models. The compact latent space crucial for diffusion models to achieve high-quality image synthesis. Latent diffusion models [1] propose to train an autoencoder to map raw pixels to a latent space first, where the latent space is typically 8x spatially downsampled and comes with 4 channels, reaching a compression rate of 48. The follow up work on autoencoders, including [5, 21 (22). mainly investigate the channel number and shows that increasing channel number can improve the quality of diffusion samples via larger transformer models. Recently proposed deep compression autoencoders (DC-AB) [3] compress the latent space at more aggressive spatial downsampling rates, e.g., 32 or 64, further reducing the training cost of diffusion models.
However, there a lack of exploration and discussion on the structure latents for diffusion models. Most autoencoders for diffusion models encode pixels into dense 2D grid-based based latents and ignore the underlying non-uniform and sparse structures natural images, where a background region an image might be worth less latents than foregrounds. Here in this paper, we study this sparsity as well as visual non-uniformity explicitly with region of interests (Rols) along with latents for diffusion models, following the sparse visual generation research line (8).

---

3 Methods
Our image synthesis pipeline follows the common practice of latent diffusion models: an autoencoder first that compresses images into set of latents and Rols, followed by a generative model that takes noised latents and Rols as input and predicts diffusion targets. We first describe the design of our sparse flow autoencoders, and then introduce sparse flow-based generative transformers for modeling joint flow.
3.1 Sparse Flow Autoencoders
Given an image I E R xx3x3, conventional autoencoders encode it into grid-based latent representa- tions and decode latents back into pixels by the encoder & and decoder D:
z = E(I E IR H///W///
(1)
i= D()) #xxxx
(2)
where f is the downsampling factor, typically 8 in practice, and is the latent dimension. The training for & and DD is done by minimizing the reconstruction loss Lee (I. Variational autoencoders also impose a Kulback-Leible divergence loss fKL on z to regularize the latent distribution [24].
Latent and Rol representation. Different from grid-based latent representations, we propose sparse flow variational autoencoders, SF- SF-VAE, to use a sparse set of latents z E R Nxx with their corresponding regions of interests (RoIs) r E RN to represent an image. The latent space of SF-VAE is simply structured as one flattened dimension space, and the spatial property of latents characterized in Rols. The Rols are represented as bounding boxes in the format of (æ, y, h, w), where (zr, y) and (h, w) are center points and height and width. The encoder & now outputs both latents and Rols, and the decoder D takes both latents and Rols as input to reconstruct raw pixels:
(z, r) = E()),
i ==((zrr) r).
By eliminating the grid- based spatial prior, the number of latents N can be decoupled from image pixels И W and can be arbitrarily chosen and further greatly reduced according to our experiments.
26 can encode image Encoding pixels into latents and RoIs. Typical object detectors 25
pixels into latent representations and Rols. However, they often lack emphasis on backgrounds and need bounding box annotations as individual supervision. Here, we resort to the SparseFormer architecture (27 build our encoder & from scratch, which encode image pixels into latents and RoIs in an end-to-end manner without bounding box supervision. SparseFormer takes early image features I as input and gradually refines latents z and Rols r via local image features within Rols r by several SparseFormer transformer layers, where refinement on and r are both differentiable:
(z' r²) = SPARSEFORMERLAYER/
(4)
where x. and r are parameters of the model. The Rols r are updated using formulaoruntnnon on (I. y, h w)
25
Decoding latents and Rols back into pixels. To learn the distribution of latents and Rols given an image by just pixel reconstruction, we need to decode them back into raw pixels in a fully differentiable way, that is, the reconstruction loss needs to be both differentiable to latents and Rols. Unfortunately, to our best knowledge, mapping latents and RoIs back into raw pixels in a differentiable way has not been deeply investigated in previous works. We have first tried simple idea, Rol-aware cross attention, but they struggle to recover high frequency details even in a long training schedule, and cannot learn compact latent Rols with pixel reconstruction.
Inspired by advance in neural rendering [28] 29 we design our decoder D with the neural field approach and divide-and-com -nd-cuuue strategy. Specifically, we consider a latent and its Rol indexed by i as a neural field function F(a.,r:) R²2 - Rd whose input is the pixel coordinate and output is the RGB tuple. In other words, latent and its Rol can be decoded into a pixel image individually. Considering rich high frequency details in natural images (30), we design a neural field based on
cosine transform bases similar to 2D discrete cosine transform (DCT) 31 but in a continuous
3

---

SF encoder
(z.r) conditioning
F(ar)
pica coordinate
1
pixel decoder
(z, v)
2d cosine bases
Figure 2: Structure of our SF-VAE and neural field-based decoder. For a latent tuple rz), the neural field function F(...;;) first transforms pixel coordinates into 2D cosine bases conditioned r; and then feed these bases into an MLP whose parameters are conditioned on 2; to get downsampled intermediate features. The pixel decoder then decode these features into raw image pixels. For clarity, we show only single latent case.
coordinate space. Given a raw pixel's coordinate (z, y) in the image, we first compute relative coordinate (x', v/) with regard to the Rol r, for single pixel output:
= (x x2)/wi,
y = (y wc)/h..
(5)
then compute our customized cosine transform bases X E :
X. cos [y (c) 0.5) cos (c) + 0.5)m]
where the number of bases C needs to be a squared number, and (c) and (c) are the frequency of the c-th channel in y and x directions, linearly increasing from to VC-1
We use two-layered MLP transform these cosine bases into final output values, where MLP's weights are conditioned by the latent z, through a shared feed forward layer. Note that all these computation can be done with matrix multiplication efficiently. However, we find that directly recovering raw RGB pixels from F(n,.;) is not memory friendly despite being efficient, since we need "trace" 65536 pixels with their bases and MLP activations for single latent 256 x 256 image, which is unrealistic. Therefore, we retarget the neural field output downsampled feature map I' E #/8x//8xXC use the softmax function to blend different feature maps produced by different latents, and use upsampling decoder, Dpix. to upsample I' to the final image I of the input size.
Overall autoencoder architecture. In the resulting SF- a latent does not need to correspond to a fixed spatial region across different images, and we can decouple the number of latents N from the image size and further reduce N. Our default number of latents and Rols is N = 64 for 256 x 256 image, where the latent dimension is 32, reaching high 96x compression rate We train SF-VAE in an end-to-end manner using regular VAE loss with a minor difference: we only apply the KL loss to latent variables z since Rols r not necessarily follow a standard Gaussian distribution. We also design our SF-VAE to be as parameter lightweight as possible. Visualizations in Figure3 show that SF-VAE can learn semantic and compact latent RoIs and achieve high reconstruction quality, allowing latents to focus intricate foreground objects.
3.2 Sparse Flow Transformers
Now we have compact latent space defined by latent variables z and RoIs for an image, we describe study how to generate them jointly. Here we first describe the flow matching framework (9] 32) for continuous data modeling, and then delve into our flow formulation and the design of our flow-based generative transformers.
Flow matching. Assume we want model target data distribution p(x), flow matching formulates process starting from a sample drawn from the starting distribution, typically x. NO, 1), to
Compressioo rate is 85 if Rol also counted four values.

---

input recon
Rols
target distribution
more clean
interpolastn ating coefficient
more noiey
C
latent
0.4
0.2
linear
pure noise
0.0
0.5
time
Figure 3: Visualizations of SF-VAE
Figure 4: Asynchronous interpolating schedule for latents and Rols. Rols approach target Rols faster than latents as time increases using 3 = polynomial schedule.
and learned latent Rols (NV = 64). Note different colors are only used for distin- guishing different Rols.
sample from the target distribution X1 p(x) by continuous time process:
X 0x 04X1;
where E [0, 1]. or and on interpolating coefficients characterizing interpolating flow. The flow matching requires do = 1, 00 = 0, σ = 0, and (1 = 1. popular choice is the linear interpolation, Te and a t. The velocity, or the derivation on x; in this process is
dx: d.x0 áxX1-
di
The model F is trained by minimizing the Le loss between the predicted velocity and the true velocity over sampled points and sampled paths:
E xx F(t, vell?.
(9)
During inference, flow matching draws initial sample xo NN 1) and solve the oridinary differential equation (ODE) to get the target sample X1:
dx: di
=(1,x)))
(10)
Flow matching eliminates the noise introduction in the sampling stage [5] and can model most arbitrary distribution, which is suitable for our Rol distribution modeling. Recent SiT models [TO] also shows flow matching as the prediction target with transformers [33 can surpass ones with the diffusion target.
Joint latent and Rol flow. Different from SiT models only to model grid-based latents, we need to model both the joint distribution of latents with their Rols r by SF-VAE in our flow-based generative transformers. The latents z in SF-VAE are not structured encoded in Rols r. Here, we keep most of the transformer architecture in SiT unchanged but retarget take both latents and Rols as input and predict the velocity of both them:
grid, and their positional properties are
(i Ne v..)) = F(t, Z rt).
(11)
where z; and rt are interploated latents and Rols following Equ 7 at time t, Va,t and V,,s are estimated velocity of latents and Rols. We name it as SF-SiT for modeling joint latent and Rol flow. We choose the initial distribution latents and Rols both to standard Gaussian distributions for simplicity.
Given the number of latents N already being highly reduced (e.g., 64), we do not perform any latent "grouping" and "ungrouping" operations in SiT to reduce computation. In other words, latent from SF- VAE directly corresponds to token in the transformer in SF-SiT.
5

---

Rol-based positional encoding. Since we get rid of the grid brid-based latents, it is crucial to inject positional information dependent on Rols r, into the transformer. Otherwise, the latents becomes fully permutation invariant due to the transformer's architecture. We use the sinusoidal positional encoding on four values of Rols, (æ, y, h w), and concatenate them together. We add them as position encoding to corresponding token in the transformer. It is worth noting that this positional encoding is totally based on Rols and therefore changes for the same latent across different training and inference timesteps as Rols move across different timesteps.
Asynchronous interpolation schedule. The time indicates the signal-to-noise ratio (SNR) the interpolation schedule, where = O means pure noise and / 1 means target data. Recall that we need to model both z and r and therefore SF-SiT are required to handle pure noise Rols at time t = as well not-so-clean Rols in early period. This differs from the regular SiT where the position information is always clean, and the model have strong position information given at any t to predict the latent velocity to denoise them. Therefore, we design asynchronous interpolation schedule for Rols compared with latents, where we interpolate Rols to approach the target Rols faster than latents in polynomial way:
ort (1-
Or,t o,, = 1-
(12)
where 3 is control parameter, and reduces to the synchronous linear interpolation when / = 1. We keep the linear interpolation schedule untouched for latents:
2 = 1
Oz22
(13)
This asynchronous interpolation schedule for latents and Rols allows the velocity prediction for z to be position-awes aware with cleaner Rols rı approaches the target Rols faster, as shown in the Fig 4 It is worth noting that the asynchronous interpolation schedule only affects the training phase, i.e., the velocity computation of latents and Rols. The inference logic the same as regular SiT, where we solve the ODE to sample the target latents and Rols. To force the model to learn the accurate Rol distribution, the ground-truth, with a balancing weight WL, as proposed by LayoutFlow [34] to model bounding box flow more accurately in layout generation.
also impose additional I, between the predicted Rol velocity and
Experiments
To verify the feasibility of synthesizing images with sparse non-grid latents, we conduct experiments on the standard ImageNet benchmark (35), mainly on 256 256 images. We first discuss our autoencoders, SF-VAE, and then present the results of our generative model, SF-SiT. Note that our aim is not to achieve new state-of-the-art results on a benchmark, but to explore sparsity in image generation pipeline.
4.1 Sparse Flow Autoencoders
Setup. As discussed in previous section, our SF-VAE consists up of SparseFormer encoder & and neural field-based decoder D, where the latent space defined by latent variables and Rols. To keep as lightweight as possible, we design SparseFormer encoder of 8 blocks with transformer dimension 512 to extract latents and Rols, where the leading 4 blocks extract Rol regional features from the image. The neural field decoder also consists of 8 transformer blocks of 512 dim, and then
Table 1: Reconstruction results on 50K ImageNet validation samples.
resolution
method
latent shape
params
rFID
PSNR
SSIM
LPIPS
256 x 256
SD-VAE-ema-te
32 x 32 x
84M
0.63
24.98
0.804
0.062
DC-AE-in-132e33
8 32
323M
0.77
23.92
0.765
0.086
DC-AE-mix--32633
8 8 32
323M
0.96
23.75
0.763
0.088
SF-VAE
64 x 32
133M
0.70
23.24
0.743
0.085
512 x 512
SD-VAF-emate
64 x x
84M
0.19
27.36
0.849
0.061
DC-AE-in-f32c32
16 16 32
323M
0.20
26.23
0.815
0.078
SF-VAE
256 32
133M
0.29
25.03
0.787
0.088

---

produce dynamic MLP's parameters, where the hidden dimension of two-layer MLP is 64. The output of the neural field decoder is then decoded into raw pixels by D Ppisel of conventional VAE decoder architectures []] but we reduce the dimension of most convolutional layers. The computational cost of the resulting SF-VAE decoder 153GFLOPs for 256 256 image, where the SD-VAE--mm--f decoder needs 31 GFFOP... The latent is 32-dim, and the default number of latents is 64. The loss follows the convention of VAE latent diffusion models, with the L2 loss for pixel reconstruction, perceptual loss 36 and GAN adversarial loss (37 for visual perceptual details, as well as KL divergence [24] for the latent regularization on z. We train the SF-VAE on the ImageNet training set with batch size of 128 for 320K iterations (equivalent 32 epochs) with learning rate of 10* We perform reconstruction FID evaluation 38 on the ImageNet validation 50K samples.
Reconstruction results. We compare our SF-VAE with the popular SD-VA VAE-f8 T that comes with a downsampling rate of 8, resulting in latent shape of 32 x 32 x 4. We also include the recently proposed deep compression autoencoder (DC-AE) 23 with a latent shape of 8 x 32 for comparison, which is exactly the same as our default latent shape. The results are shown in Table
The reconstruction FID of SA-VAE is 0.70, which is competitive with the SD-VAE (rFII) 0.63) with 1024 latents. Also, if counting the latent dimension together, the SE-VAE comes with 64 32 2048 units, much less than SD-VAE's 32 32 4096 units. Compared with DC-AE also trained on ImageNet samples and with the same latent shape, our SF-VAE achieves a lower rFID score while maintaining significantly fewer parameters.
More importantly, the reconstruction results further demonstrate that non non-grid latents are also effective for image reconstruction. These findings align with recent studies, which show that explicit 2D structured latents are not necessary for high-quality reconstructions and token numbers can be extremely compressed [1] However, methods does not necessarily belong the 1D tokenization family, since there are explicit geometric properties, i.e., Rols, associated with each latent. These 2D Rol representations also maintain the translation equivariance property of this pipeline, while ID tokenization methods typically fail to address this point.
Varying number of latents and higher image resolution. As discussed before, SF-VAE en- joys the flexibility of decoupling the number of latents from specified image resolution. Here, we investigate our SF- VAE with more aggres- sive latent reduction for fixed resolution of 256 256 in Table 2] We do not change the architecture of our SF-VAE, but only changes the number of latents from 64 to 48 and 32 and maintains the latent dimension unchanged. To reduce training costs, we initialize SF- VAE with new latent configurations from pre-trained weights of the 64-latents model, and finetune them for 20K iterations. Here, we find 32 to- kens have already been able to reconstruct pix- els with an rFID of 1.70, and 48 tokens give competitive 0.99 rFID.
inpat
latenes
boents
S12u512 ippat
255 latents
Figure 5: SF-VAE reconstructions with different latent numbers. Better zoom in.
Table 2: Token numbers for SF-VAE reconstructions.
Table Effects of asynchronous inter- polating schedule.
Table 4: Effect of L1 loss.
resolution
#latent
rFID
interpolating schedule
FIDL
WL,
FID
128
0.35
linear, 3
44.4
0.0
43.9
256 x 256
64
0.70
async, 8 =
42.0
42.0
48
0.99
async, , =
44.6
0.5
42.6
32
1,70
async, min(2 t, 1)
54.0
1.0
45.1
512 x 512
256
0.66
async, min(4 x t, 1)
55.9
128
1.29
decoupled, Rols first
63.3
7

---

Table 5: Comparison with diffusion//low--orr based transformers with grid-based latents under 400K training iterations without CFG. "SiT reproduced using official code. sampled with ODE Euler sampler with NFE=250 for fair comparison. Flops are calculated with one single NFE.
method
#tokens
#params
flops (G)
FID
1ST
D/T-B/2
256
130M
21.8
43.5
SIT-B/2
256
130M
21.8
33.0
SiT-B/2
256
130M
21.8
37.0
39.6
SF-SiT-B"
64
138M
5.9
33.6
41.4
DiT-XL/2
256
675M
114.4
19.5
SiT-X1-22
256
675M
114.4
17.2
SiT-X1.2"1
256
675M
114.4
18.8
70.9
SF-SIT-XL
64
685M
29.3
18.6
71.6
We also study SF-VAE on higher image resolution of 512 x 512 in Table
2 We first scale up the number of latents to 256 similar dense grid-based VAEs, where latent numbers are increasing quadratically regarding input image size. Then we reduce the number of latents to 128. We find that our SF-VAE can still achieve competitive reconstructions for higher images Figure
4.2 Sparse Flow Transformers
Setup. Our SF-SIT generally follows the design of original SiT transformers [10), where the input and output are retargeted to latents z and Rols r, and the velocity them, v, and v,. We mainly experiment SF-SiT-B and SF-SiT-XI variants, whose configurations strictly follow the original SiT-B and SiT-XL. Since SF-SiT are now required to output Rol velocity besides latent velocity, we add additional AdaLN head 42 to predict Rol velocity, where the output is also initialized zeros similar to latent prediction.
We train SF-SiT variants from scratch on ImageNet training set with 64 latents and Rols from SF-VAE, unless otherwise specified. In SF-SiT, each latent directly corresponds to a token the transformer blocks, as our architecture eliminates extra patchifying operations. The training configurations are the same as the original SiT models, i.c. the global batch 256 and the Adam optimizer [43] with constant learning rate 10 4, We set the default to 2 in the asynchronous interpolating schedule and the weight Wt, to 0.2 for the additional LI loss on Rol velocity prediction. We evaluate our SF-SiT using ADM evaluation pipeline 44 and report FID with 50K samples.
Comparison with grid-based diffusion/flow transformers. We first compare our methods under the 400K iteration budget with grid-based DIT/SiT baseline [T] [0] in Table without classifier-free guidance (CFG). It is worth noting that original SiT models adopt the SDE sampler during inference and report their results. This differs from SF-SIT models, which use simple ODE sampler for latent and Rol inference. Due to the lack of publicly available intermediate SiT checkpoints, we reproduce SiT-B and SiT-XL variants using the official code, and inference samples with ODE Euler sampler. We also align the number of function evaluations (NFE) in SF-SiT and SiT models to 250 to make fair comparison.
As shown in Table 5 SF-SiT models, despite operating on significantly reduced latents, achieve performance on par with SiT models using the Euler ODE sampler. We experimented SF-SiT models with the original SiT's SDE sampler but observed degraded Rol quality and visual distortion in decoded images. We hypothesize this sensitivity arises from that SF- VAE encoded RoIs being deviating from standard Gaussian distributions, making them incompatible with the noise introduced by SDE sampling. Overall, these findings suggest that jointly modeling the flow of latents and Rols have competitive performance paired with ODE-based samplers.
A Ablation studies. For all ablation studies, we resort to SF-SiT-B variant with 200K budget due to the limited computational resource. We also inference samples in ablations with the Heun ODE solver with 50 NFE for efficiency.
Asynchronous interpolation schedule. Our proposed SF-SiT introduces additional RoIs r as additional modeling targets. As discussed before, propose the asynchronous interpolation schedule

---

timestep
0.2
0.4
0.6
0.8
1.0
samples with Rols
Rol
decoded
decoded
Figure 6: Visualization of Rol flows and decoded images with SF-SiT-XL and CFG=4.0.
for be compatible in the joint modeling with the latents Here investigate the effect of the asynchronous interpolation schedule in Table We can find that the asynchronous interpolation schedule for gives the best result. In contrast, synchronous interpolation (3 1) between and z, or purely linear interpolation for both, results the inferior FID. We also try more aggressive asynchronous schedule, where the interpolated Rol reaches the target Rol linearly but at the earlier timestep, e.g., 0.25 for min(4 x 6, 1). These clamped linear schedules are also asynchronous, where we suppose sample Rols first and then sample latents in hard way, leading significant degradation in FID. Finally, the decoupled interpolation schedule indicates that we first schedule Rol denoising separately and individually when in [0, 0.5] and then schedule latent denoising when is in [0.5, 1]. where the denoising processes for r and are fully decoupled and disjoint, leading to the worst FID. These findings show that the joint flow of latents and Rols crucial to good performance in SF-SIT framework but also needs soft asynchronous interpolation between and to have better performance.
Rol velocity L1 loss. We also investigate the effect of L1 loss on the Rol velocity prediction, as proposed by. In line with 34 the L2 loss weight 0.2 gives the best result.
Longer training schedule. We further train the largest SF-SiT variant, SF-SiT-XL, with longer training schedule. To accelerate convergence on limited hardware resource, we use REPA (45) for our SF-SiT-XL training. We use DINOv2-B/14 46 as our REPA align target. Since our SF-SiT
Table Class conditional generation results on 256 256 ImageNet. P refers precision and
refers to recall following 44
method
#tokens
train steps
FID
IS
Pt
R
LDM-4 01
10.56
103.5
0.71
0.62
DIT-XL/2 III
256
7M
9.62
121.5
0.67
0.67
SIT-XL/2 10
256
7M
8.26
131.6
0.68
0.67
SIT-XL/2 w/ REPA 445
256
IM
6.4
SF-SIT-XL w/ REPA
64
1.4M
7.35
131.9
0.70
0.66
DIT-XL/2 (CFG=L.))
256
2.27
278.2
0.57
XL/ (CFG=1.5)
256
7M
2,06
277.5
0.83
0.59
SF-SIT-XL REPA (CFG=L5)
64
1.4M
2.99
279.2
0.54
SF-SIT-XL w/ REPA (CFG=1375))
64
1.4M
2.76
247.7
0.81
0.58

---

does not form grid-based feature maps, we need to transform intermediate representations in SF-SiT transformer blocks into grid using the dimeddependent interpolated r, as positional properties. Here, we use a simple cross attention from dense queries on grids to SF-SIT tokens with the positional embedding based on r, added into keys and values. Dense grid queries are then passed through an MLP to align with DINOv2.
Table shows SF-SiT-XL (w/ 64 latents) results with longer training budget. Note that here we only want to show the potential of SF-SiT-XL on our limited budget with the fast convegence of REPA (it have already took days on A100s to complete 1.4M steps). SF-SiT-XL reaches the competitive performance (7.35 FID w/o CFG and 2.76 w/ CFG) with significantly reduced tokens and computation used. is expected have better results when training for more iterations and we will investigate them further when computational resources are available.
5
Conclusion
In this paper, we proposed a paradigm for visual generative modeling by using non-grid sparse latents with their positional properties, challenging the dense grid convention for image synthesis. We first designed sparse flow variational autoencoder, SF-VAE, which encodes raw image pixels into latents together with their Rols. With SF-VAE, for good reconstruction fidelity. We then propose sparse flow-based generative transformers, SF-SiT, to model the joint flow of latents and Rols with the highly reduced latent space. With carefully- designed asynchronous interpolation schedule and loss for Rols, SF-SiT models have competitive performance compared with diffusion/flow transformers. Although targeting new state-of-the-art results, SF-SiT-XL with just 64 tokens still reaches a competitive 2.76 FID on ImageNet with longer training schedule. We hope that our work facilitate further research directions on non-grid and sparse generative methods, as well as sparse approaches for generation in other domain, c.g., audio and video.
compress 256 x 256 images with down to 32 latents
Limitations. Due to the lack of computational resources, we only train our SF-SiT-XL variant with the help of representation alignment (REPA) for fast convergence, and under fewer tokens settings, it might require more training iterations to converge with RoI modeling compared with grid-based SiT/DiTs for large parameter models. Also, the large scale text-to-image experiments remain exploration in the future. As this paper mainly aims for visual generative modeling, it inherits safety risks regarding its generated content. As the standard ImageNet being the training dataset, which is known to be a relatively safe for academic purposes, these concerns might not be fully addressed in our current framework.
This project is supported by the National Research Foundation, Singapore under its NRFF Award NRF-NRFF1 3-2021--000000
Acknowledgement
10

---

References
[1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolttion image synthesis with latent diffusion models. CVPR, pages 10674-10685, IEEE, 2022.
[2] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text to -maee generation. In ICML, volume 139 Proceedings of Machine Learning Research, pages 8821-8831, PMLR, 2021.
[3] Adam Polyak. Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma. Ching- Yao Chuang. David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang. Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam Tsai, Samanch Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali K. Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng. Dmitry Vengertsev, Edgar Schönfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffiman. Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li. Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, and Yuming Du Movic gen: cast of media foundation models. CoRR, abs/2410.13720 2024.
[4] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR. OpenReview.net, 2023.
[5] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. In ICLR. OpenReview.net, 2024.
[6] Black Forest Labs. Flux. htttps://github..../t www.c//ww.caaacccoo www-toorsstttee labs/tlua 2023.
[7] David Marr. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. The MIT Press, 07 2010.
[8] Charlie Nash, Jacob Menick, Sander Dieleman, and Peter W. Battaglia. Generating images with sparse representations. ICML, volume 139 Proceedings of Machine Learning Research, pages 7958 7968. PMLR, 2021.
[9] Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Lc. Flow matching for generative modeling. In ICLR. OpenReview.net, 2023.
[10] Nanye Ma, Mark Goldstein, Michael S. Albergo, Nicholas M. Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-hased generative models with scalable interpolant transformers. In ECCV (77), volume 15135 of Lecture Notes in Computer Science, pages 23-40. Springer, 2024.
[11] William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, pages 1172-41882 IEEE, 2023.
[12] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CoRR, ahs/2207. 2022.
[13] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, pages 12873-12883, Computer Vision Foundation IEEE, 2021.
[14] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. Maskgit: Masked generative image transformer. In CVPR, pages 11305-11315 IEEE, 2022.
[15] Jonathan Ho, Ajay Jain, and Picter Abbocl. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[16] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. ICLR. OpenReview.net, 2021.
[17] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In NewrIPS, 2022.
[18] Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao. Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In CVPR, pages 22669 22679, IEEE, 2023.

---

[19] Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-end diffusion for high resolution images. In ICML, volume 202 of Proceedings of Machine Learning Research, pages 13213- 13232. PMLR, 2023.
[20] Emiel Hoogehoom, Thomas Mensink, Jonathan Heek, Kay Lamerigts, Ruiqi Gao, and Tim Salimans. Simpler diffusion (sid2): 1.5 FID on imagenet512 with pixel-space diffusion. CoRR, ahs/2410.19922 2024.
[21] Xiaoliang Dai, Ji Hou, Chih- Yao Ma, Sam S. Tsai, Jialiang Wang. Rui Wang. Peizhao Zhang, Simon Vandenhende, Xiaofang Wang, Abhimanyu Dubey, Matthew Yu, Abhishek Kadian, Filip Radenovic, Dhruv Mahajan, Kunpeng Li. Yue Zhao, Vladan Petrovic, Mitesh Kumar Singh, Simran Motwani, Yi Wen, Yiwen Song, Roshan Sumbaly, Vignesh Ramanathan, Zijian He, Peter Vajda, and Devi Parikh. Emu: Enhancing image generation models using photogenic needles in haystack. CoRR, ahs/2339. 2023.
[22] Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. In CVPR, pages 15703-15712. Computer Vision Foundation IEEE, 2025.
[23] Junyu Chen, Han Cai, Junsong Chen, Enze Xie, Shang Yang. Haotian Tang, Muyang Li, Yao Lu, and Song Han. Deep compression autoencoder for efficient high-resolution diffusion models. CoRR, ahs/24 10733, 2024.
[24] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. In ICLR, 2014.
[25] Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS, pages 91-99, 2015.
[26] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV volume 12346 of Lecture Notes in Computer Science, pages 213 2292. Springer, 2020.
[27] Ziteng Gao, Zhan Tong, Limin Wang, and Mike Zheng Shou. Sparseformer: Sparse visual recognition via limited latent tokens. In ICLR. OpenReview.net, 2024.
[28] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV (I), volume 12346 of Lecture Notes in Computer Science, pages 405 421.421.25 Springer, 2020.
[29] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul P. Srinivasan, Edgar Tretschk, Yifan Wang, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, Tomas Simon, Christian Theobalt, Matthias Nießner, Jonathan Barron, Gordon Wetzstein. Michael Zollhöfer, and Vladislav Golyanik. Advances in neural rendering. Comput. Graph. Forum, 41(2):703-77 735, 2022.
[30] Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrlPS, 2020.
[31] N. Ahmed, T. Natarajan, and K.R. Rao. Discrete cosine transform. IEEE Transactions on Computers, C-23(1):90-9)) 93, 1974.
[32] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In ICLR. OpenReview.net, 2023.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pages 5998 6008, 2017.
[34] Julian Jorge Andrade Guerreiro, Naoto Inoue, Kento Masui, Mayu Otani, and Hideki Nakayama. Layout- flow: Flow matching for layout generation. In ECCV (36), volume 15094 of Lecture Notes in Computer Science, pages 56-72. Springer, 2024.
[35] Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
[36] Richard Zhang, Phillip Isola, Alexci A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effec- tiveness of deep features as a perceptual metric. In CVPR, pages 586-595. Computer Vision Foundation / IEEE Computer Society, 2018
[37] Phillip Isola, Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image- tee--to--mma image translation with conditional adversarial networks. CoRR,
2016.
12

---

[38] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge local nash equilibrium. In NIPS, pages 6626-6637, 2017.
[39] Qihang Yu, Mark Weber, Xueqing Deng. Xiaohui Shen, Daniel Cremers, and Liang Chich Chen. An image worth tokens reconstruction and generation. CoRR, abs/2406.07550, 2024.
[40] Shivam Duggal, Phillip Isola, Antonio Torralba, and William T. Freeman. Adaptive length image tokeniza- tion via recurrent allocation. In ICLR. OpenReview.nes 2025.
[41] Wilson Yan, Volodymyr Mnih, Aleksandra Faust. Matei Zaharia, Pieter Abbeel, and Hao Liu. Elastictok: Adaptive tokenization for image and video. ICLR. OpenReview.net, 2025.
[42] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, pages 3942 3951. AAAI Press, 2018
[43] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster). 2015.
[44] Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models heat gans on image synthesis. In NeurIPS, pages 8780-8794, 2021.
[45] Sihyun Yu, Sangkyung Kwak, Huiwon Jang, Jongheon Jeong. Jonathan Huang, Jinwoo Shin, and Saining Xie. Representation alignment for generation: Training diffusion transformers easier than you think. CoRR, abs/2410.06940,, 2024.
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran. Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. Trans. Mach. Learn, Res., 2024, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work performed by the authors.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Ys]
The paper provides sufficient information to reproduce the experimental results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: Now our codebase is not ready for being publicly available. We will release the code upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
//ipss.cc/pbllicguuiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Ye]]
Justification: The paper provides sufficient information to reproduce the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Experiments in this paper are computation-consumiii and due to the limited computational resources, we cannot run the experiments repeatedly.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The paper adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper focuses on improving models in the academic area.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper is investigating the new diffusion models the standard ImageNet dataset from scratch. The ImageNct dataset is commonly used for academic research and is considered relatively safe.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This paper properly credits the creators original owners of assets used in the paper and mentions the license and terms of use explicitly.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
18

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented.
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/coodemmodel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
19

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: No LLMs used in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM for what should or should not be described.
(https: http::/s/eeaarresscc////ooerrere
20


Reviews:
[{"review_id": "UUILWLGnsJ", "reviewer": "Reviewer_Rnro", "summary": "This paper introduce SF-VAE, which encodes an image into a small number of latent vectors along with learnable Regions of Interest, and SF-SiT, a flow-based transformer that jointly models latent and RoI trajectories during generation. This work achieves competitive image synthesis results onImageNet data, reducing computational cost significantly compared to conventional grid-based methods.", "questions": "How does your method perform on dense or structured visual inputs, such as documents, charts, or scenes with many small objects? ImageNet is not sufficient to test this.\n\nWhat happens when the number of RoIs is insufficient for reconstruction or generation? Are there strategies to adaptively allocate more tokens to difficult regions?", "limitations": "The paper does not discuss how the method would generalize to domains with high spatial complexity (e.g., documents or multi-object scenes), nor whether RoIs are semantically meaningful. These are crucial questions given that the method is explicitly designed to model spatial sparsity.", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strenghs:\n1. The introduction of RoI-based latent modeling in a diffusion/flow generation pipeline is an  interesting idea.\n2. This RoI base mehtod achieves strong FID scores using significantly fewer tokens and FLOPs than conventional models (e.g., DiT, SiT).\n\n\nWeaknesses\n1. All experiments are restricted to image generation (reconstruction or synthesis) on ImageNet, without any assessment on image understanding tasks such as classification, retrieval, or layout prediction. This leaves open whether the learned latent representations are semantically meaningful or generalizable.\n\n2. Furthermore, ImageNet mainly contains single-object, center-biased natural images. These are spatially sparse by design and make it easy for the RoI model to leran by locating obvious foregrounds. As such, the proposed method is not stress-tested on structured, multi-object, or text-rich data where adaptive sparsity is most needed.\n\n3. The paper lacks any attempt to quantify the semantic alignment, consistency, or interpretability of the learned RoIs. It is unclear whether RoIs focus on objects, backgrounds, or arbitrary textures. The authors can apply this learned representation on vision-langauge problems.", "was_revised": true, "final_justification": "After reviewing the rebuttal and additional experiments, I maintain a borderline reject recommendation. While the paper presents a promising and efficient approach for sparse image generation using learnable RoIs, key concerns remain unresolved. The method shows limited generalization to complex or dense scenes. Additionally, the learned RoIs lack semantic alignment or interpretability, raising questions about the broader applicability and robustness of the proposed representation.", "rebuttal": "We would like to thank the reviewer's time and comments on our submission. We address the concerns as following:\n\n**Q1. SF-VAE latent representations understanding capabilities**\n\nIt is a great potential direction but differs a lot from this paper's scope and thus we have not yet explored. This paper is mainly for image reconstruction and generation. Variational autoencoders (VAE) for diffusion latent spaces commonly only compress perceptual details into compact latents and do not encode much high-level semantics into latents. We believe SF-VAE latents also follow this property: compress spatial patterns into latents and shiftable RoIs (can be inferred from Fig 3 and Fig 6, RoIs are still based on low-level details). Therefore, this is a very open and challenging topic on understanding capabilities of SF-VAE latents and we may investigate this in the future.\n\n**Q2. SF-VAE on documents, charts, or scenes with many small objects**\n\nWe here evaluate the SF-VAE on MS COCO val 5k images (256x256), which commonly consists of many foreground objects. The results are\n\n| method            | rFID | PSNR | SSIM |\n|-------------------|------|------|------|\n| SD-VAE-ft-mse     | 4.70 | 24.5 | 0.71 |\n| SD-VAE-ft-ema     | 4.42 | 23.7 | 0.69 |\n| **SF-VAE, 64 tokens** | 4.73 | 22.9 | 0.75 |\n\nWe can see SF-VAE is genereally on par with SD-VAE on natural complex scene MS COCO images with a better SSIM score.\n\nWe evaluate our methods on documents and find not-satisfying results (distortion and broken texts), especially small texts. It is mainly due to **a)** training data, as now we only train SF-VAE on ImageNet (texts are very rate in ImageNet training set). SD-VAE-ft-ema also performs poorly on small texts as they are not finetuned on text dataset; **b)** high compression rate. If we lower compression rate of SF-VAE by a large margin, we believe it would be better for high-frequency details. However, increasing latent space is not meaningful for efficient and compact latents for generation, which is the main goal of our paper.\n\n **Q3. Semantics and interpretability of learned RoIs**\n\nIt can be inferred from Fig 3 and Fig 6 that learned RoIs mainly focus on low-level details and are influenced by the pixel frequency within their covered regions. Specifically, RoIs tend to be larger in areas dominated by low-frequency content. There are not many strong semantics in RoIs since we do not impose any semantic alignment loss, and this is expected and pretty common for self-supervised pixel reconstruction task for training to only encode low-level frequency details in RoIs (as discussed in latent diffusion paper).\n\n**Q4. Are there strategies to adaptively allocate more tokens to difficult regions?**\n\nIt is worth noting that our SF-VAE is an end-to-end model and RoIs in SF-VAE can move to high frequency areas by interative shifting by SparseFormer encoding. But handcrafting strategy is a great idea! We have not yet explored in that direction on adaptive allocations of tokens, especially when the number of RoIs is insufficient for reconstruction. A potential simple straightforward way is to iteratively auto-encoding an images with increasing token/RoI numbers on the pixel l1 loss/LPIPS criterion. When the good criterion is met, we stop increasing token numbers. This approach requires training a varying token SF-VAE, which is also simple to implement. Thank you for your advice and we hope can future work explore in this direction further."}, {"review_id": "CpE2eUwkPH", "reviewer": "Reviewer_pbLv", "summary": "This paper proposes to learn a latent generative model for images where, in contrast to most previous work, the latents are not structured in a 2D grid. To learn such model, similarly to other latent space models (e.g., LDM), the paper proposes a two-stage approach: the first stage learns a Sparse Flow VAE (SF-VAE), which encodes an input image into a sparse set of 1D latent codes / tokens and RoI representations; the second stage learns a flow model to sample from the target (joint) distribution of latents and RoIs, by adapting a transformer-based model (SiT) to parameterize the velocity field. The paper refers to the latter as Sparse Flow SiT (SF-SiT).\nExplicitly modeling RoIs introduces additional complexity in learning a joint velocity field, especially at timesteps close to t=0, as the model has to rely on a noisy RoI representation to predict a position-aware latent velocity. To alleviate this issue, the paper proposes an asynchronous interpolating schedule, by designing an interpolation scheme where RoIs approach the target distribution more quickly, compared to the corresponding latents.\nThe paper evaluates the proposed approach on the ImageNet generation benchmark, at two different resolutions (i.e., 256x256 and 512x512), showing competitive results (gFID) when compared with baselines that do employ a grid-structured latent space, while reducing number of latent tokens and training compute.", "questions": "1. Can the authors compare the proposed approach against other 1D image tokenization strategies? One straightforward example could be the already mentioned \"TiTok\" [B], which also evaluates on the ImageNet 256x256 benchmark. \n2. Can the authors compute the total number of FLOPs for sampling an image in pixel space? Which means, also including the necessary operations to convert your 1D tokens into 2D feature maps via the neural field decoder into the total amount of flops.\n3. Can the authors report (cf. Table 6) the result (gFID) obtained with the SiT-XL baseline using REPA for faster convergence and CFG scale > 0? It will be helpful to really show the potential of the proposed SF-SiT-XL with limited compute. \n4. Can the authors provide more insights on why we need to (explicitly) model the distribution of RoIs? This question is also partially motivated by the fact that the RoIs (cf. Figure 6) seem to flow to a fairly even distribution where each token still corresponds to a small patch of your image. Ablating this choice can be a very good starting point.\n5. Can the authors provide a more convincing discussion of the issue of safeguarding their image generation approach?\n\nIn general, I would consider increasing my score upon seeing a comparison with other previous works on 1D image tokenization, which is currently missing from the paper. Also, the issue of safeguarding is not handled well. Thus, (1) and (5) seem to be the most relevant point to address. Addressing other points (2-4) would be helpful to have more insights on the significance of the proposed approach.", "limitations": "The paper is currently missing a paragraph discussing (or briefly summarizing) the limitations of the approach. Although this is already shortly mentioned in the conclusion, it would be helpful to have a separate paragraph mentioning that the goal of this work is not to target SotA generation results, but rather to demonstrate the applicability and scalability of a new image tokenization approach. Another additional point should make it clear that the initial hypothesis has been verified on limited compute (i.e., training time), but it is still to demonstrate whether this approach can scale to more compute and surpass grid-based latent representations, while being faster and less memory-intensive.\n\nI disagree with the answer to question 11 in the questionaire. This is an image generation model and as such the question of safeguards is indeed a relevant issue. While the safeguards needed here are likely no different than those for most of the models compared to, I do believe that this issue cannot be swept under the rug quite this easily. The paper should at least reflect on whether safeguards may be necessary and share these reflections with the readers (and reviewers). References to relevant work on safeguarding of image generators would also improve the paper.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ The paper is quite well-written and easy to follow. All the introduced architectural components are presented in detail, making the technical contributions of the paper easy to grasp.\n+ The idea of encoding an image into a sparse set of latents instead of a lower-resolution latent \"image\" is definitely interesting and worth exploring. In fact, while a 2D latent space can benefit from the inductive bias of convolutions in U-Nets, as discussed in [A], it is reasonable to think that this structure can be dropped when relying on transformers for parameterizing flow or diffusion models.\n+ The evaluation on ImageNet benchmark shows better results for the proposed SF-SiT, compared to presented baselines, on a fixed training budget of 400k iterations and when sampling with an Euler ODE solver. Besides obtaining improved gFID results, sampling requires almost 4 times less FLOPs, thanks to the increased latent compression rate. \n\nWeaknesses:\n- This is not the first work introducing the concept of 1D image (sparse) tokenization.  \nFor instance, [B] develops a highly similar idea, presenting a transformer-based 1D tokenizer for sparse tokenization (e.g., 32 tokens) and a decoder that is also based on a ViT and directly decodes 1D tokens into an image.  \nIndeed, a comparison with the \"TiTok\" approach [B] or other immediate follow-up works is missing. Slot-based auto-encoders [C] (and much follow up work) are, up to my knowledge, yet another way of designing 1D image tokenizers, and hence are also worth a discussion in the related work section at least, or possibly a comparison. \n- The paper does not provide a clear motivation of why we need to explicitly model the distribution of RoIs. I think a motivation / ablation against no explicit modeling, or showing an application (if any) of this \"disentangled\" latent space is missing. This is important also because the shown examples show a rather regular tiling of the image into RoIs, similar to the usual notion of patches.\n- It is not clear whether the reduced number of FLOPs at the sampling / denoising stage (i.e., for SF-SiT) is enough to contrast the increase in FLOPs at the decoding stage, mainly introduced by the proposed neural field decoder, which eventually computes N 2D feature maps (to be blended together) from the N input RoIs, conditioned on the corresponding 1D latents. The evaluation of FLOPs for the entire end-to-end generation would give more insights on the advantage of using the proposed approach. \n- An initial attempt of efficiently scaling the training budget with REPA shows promising results on class-conditional generation (with and without CFG), although the results are still not on par with baselines. A comparison with SiT-XL using REPA for faster convergence under the same training budget and with CFG is also missing, thus it is hard to infer how good the proposed approach can scale.\n\nNote: My assessment is somewhere between 2 and 3 here. I eventually settled for a 2, since the missing comparison to [B] and follow-up work is a significant oversight, as is the missing discussion of safeguards.\n\nReferences:  \n[A] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).  \n[B] Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., & Chen, L. C. (2024). An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37, 128940-128966.  \n[C] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., ... & Kipf, T. (2020). Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33, 11525-11538.", "was_revised": true, "final_justification": "As I already stated in the response to the authors, I am not completely convinced by this paper. I agree with reviewer Rnro that the learned RoIs do not seem to be easily interpretable. The authors are trying to argue that interpretability of the RoIs is not needed for reconstruction or generation, but I do not find this convincing. Even for reconstruction and generation, we do want to know what the role of the main technical innovation of a proposed approach is. In my view, this desire extends beyond applications to scene understanding.\nWithout any precise answer on the role of the RoIs, I believe that a paper that proposes RoIs as its main technical innovation cannot stand on its own. Even if they model low-level details, we would need some analysis to show this quantitatively and some argumentation why this is important. I thus maintain my reject recommendation, upgrading it slightly to borderline reject in light of the (more) convincing points in the rebuttal.", "rebuttal": "We really appreciate your time and efforts for detailed reviews and comments on our manuscript. We would like to address concerns as following:\n\n**Q.1. Can the authors compare the proposed approach against other 1D image tokenization strategies? One straightforward example could be the already mentioned \"TiTok\" [B], which also evaluates on the ImageNet 256x256 benchmark.**\n\nWe show the reconstruction results on our methods and TiTok here on ImageNet 256x256 benchmark.\n| method | token | rFID |\n|--------|-------|------|\n| TiTok  | 32    | 2.21 |\n| TiTok  | 64    | 1.71 |\n| TiTok  | 128   | 1.70 |\n| **SF-VAE** | 32    | 1.70 |\n| **SF-VAE** | 64    | 0.70 |\n| **SF-VAE** | 128   | 0.35 |\n\nIt is worth noting that the comparison is not strictly fair since our SF-VAE uses continuous latents while TiTok uses discrete latents, so we do not include this comparison in the submission. From results, we can see SF-VAE reaches much better performance compared to TiTok and this is expected to a moderate extent. Importantly, we mainly focus on joint modeling of latents and RoIs in reconstruction and generation, which is not explored in TiTok and other 1D tokenization work. We will add more discussion on the difference between our work and TiTok 1D tokenizations/object centric/slot attention methods  in revised manuscript.\n\n**Q.2. Can the authors compute the total number of FLOPs for sampling an image in pixel space? Which means, also including the necessary operations to convert your 1D tokens into 2D feature maps via the neural field decoder into the total amount of flops.**\n\nWe measure the SD-VAE-ft-mae and SF-VAE (64 tokens) decoder's FLOPs. SD-VAE-ft-mae decoder is with 311 GFLOPs and SF-VAE decoder is with 153 GFLOPs. This hugh difference is because we designed SF-VAE decoder, especially pixel decoder, to be light as much as possible, so we configure channel_multipliers to be [1, 1, 2, 4] in SF-VAE decoder while channel_multipliers are [1, 2, 4, 4] in SD-VAE.\n\nWe also compute the FLOPs of our \"neural field\" operations. We use torch.bmm to perform neural field operation:\n```\nShape (64, 1024, 64) -> torch.bmm -> Shape (64, 1024, 64) -> Shape (64, 1024, 512) # 64 is the number of tokens, 1024 is (256/8)*(256/8) grid, 64/512 is the channel\n```\nThe result is 5GFLOPs, which is relatively minor to 153 GFLOPs. So overall, SF-SiT-XL needs 29.3 * 50+153 = 1618 GFLOPs to sample a 256x256 image while  SiT-XL needs 114.4 * 50+311 = 6031 GFLOPs (assume 50 sampling steps)\n\n\n**Q.3. Can the authors report (cf. Table 6) the result (gFID) obtained with the SiT-XL baseline using REPA for faster convergence and CFG scale > 0? It will be helpful to really show the potential of the proposed SF-SiT-XL with limited compute.**\n\nThank you for your advice on that. Till now, there are not publicly-available checkpoints/results of SiT-XL w/ REPA with CFG scale for 1M/1.4M/comparative iterations so we do not report that. If resources available, we might consider reproducing SiT-XL w/ REPA for 1.4M iterations with CFG and report in future version \n\n**Q.4 Can the authors provide more insights on why we need to (explicitly) model the distribution of RoIs? This question is also partially motivated by the fact that the RoIs (cf. Figure 6) seem to flow to a fairly even distribution where each token still corresponds to a small patch of your image. Ablating this choice can be a very good starting point.**\n\nThank you for the advice. We ablation this choice by designing the SF-SiT to model RoI distributions first (t<0.5) and model latents then (t>0.5). It can be implemented by extending the `async, min` interpolating schedule in Table 3. This means in the inference, we first sample RoIs (analogue to \"layouts\") from the distribution and when RoIs are completely generated, latents are then sampled. The resulting gFID result is 63.32, much worse than any entry in Table 3 (best 42.0 for beta=2, worst 55.9 for async min), showing the necessity of modeling the distribution of RoIs, especially jointly with latents.\n\n**Q.5. Can the authors provide a more convincing discussion of the issue of safeguarding their image generation approach?** \n\nWe are sincerely sorry for missing that point. We did not consider safeguarding our image generation approach back then because our all models are trained on standard ImageNet datasets and ImageNet is a relatively \"safe\" dataset. I am not an expert on diffusion safety but I would like to say that intuitively, we can safeguard not only latents but RoIs in the SF-SiT pipeline when generating sensitive contents. The safeguarding techniques could be classifier guiding on either latents or RoIs or both. We can also safeguard SF-SiT in a more finer way by detecting a token to contain sensitive details and throwing it away in SF-VAE decoding. We will do a survey on safeguarding and discuss the relevant work and how to integrate these safeguarding methods in future version."}, {"review_id": "K5p7fXMM2t", "reviewer": "Reviewer_zU5o", "summary": "This paper discusses image synthesis from the perspective of sparsity. Specifically, the latent and position attributes of the image are modeled separately, and flow matching is used to generate the latent and position respectively. Compared with the previous grid-based modeling, it has lower computing resources and competitive performance.", "questions": "See weaknesses, more analysis will help to better understand how it works", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Modeling images as a combination of latents and RoIs from an unstructured view is insightful. This approach could inspire future multimodal learning (e.g., aligning latents with semantics) or efficient video generation, making it valuable to the community.\n\n2. The use of neural rendering to reconstruct images from latents and RoIs is creative. The model achieves 96× compression while maintaining performance close to state-of-the-art VAEs. The proposed RoI-based positional encoding and scheduler are also well-designed, enabling performance comparable to the SiT baseline using only 64 tokens (vs. 256).\n\n3. The paper is clearly written and includes thorough ablations on latent size, resolution, and scheduling.\n\nWeaknesses:\n1. Due to space limits and no supplementary material, many implementation details are missing. For example, as author provides RoI visualization in Figure 3, it's unclear what each RoI-latent pair represents—do they correspond to structured objects or low-level features? Some analysis would help clarify this.\n\n2. Some recent work in discrete image generation explores compressing token representations (e.g., 1D tokenizers [1]). It would be interesting to hear the authors’ views on how their method compares to these approaches.\n\n3. Looking forward to open source and further work.\n\nReference:\n\n[1] An Image is Worth 32 Tokens for Reconstruction and Generation", "was_revised": true, "final_justification": "This paper is overall valuable, with a novel idea, although some experiments may be somewhat incomplete.", "rebuttal": "Thank you for your comments and advice on our submission. We will add more implementation details as much as possible in future version per your advice to make the paper more clear and less confusing. We will open source our work  soon upon acceptance. We  address your concerns as following:\n\n**Q.1. Do RoIs correspond to structured objects or low-level features?**\n\nAs for now, RoIs in SF-VAE mainly corresponding to low-level features (such as textures and edges, mostly frequency based). From Fig 3, we can see RoIs are mainly influenced by the pixel frequency within their covered regions. Generally, RoIs become dense towards high frequency areas while a single RoI becomes large  when covering low frequency content. But overall, RoIs here do not have many strong semantics in RoIs since we do not impose any semantic alignment loss.  We will provide more RoI visualizations and analysis in future version.\n\n**Q.2. How our method compare with discrete image compressing representations**\n\nThank you for rising an insightful discussion here. First of all, our methods (including SF-VAE and SF-SiT) are all continuous-based and designed for diffusion while TiTok and other 1D image representations are discrete tokens are designed for MaskGiT or autoregressive modeling, which is significantly different. These two method are not directly comparative since the space of discrete representations is much smaller than the continuous one. So we'll see a large margin between SF-VAE and TiTok on all token number settings (see response to Q.1. from Reviewer pbLv) and this is expected.\n\nDespite the high-level idea that TiTok and SF-VAE share, they are exploring two different pathways to compress image pixels. TiTok uses transformers to learn compact tokens, which is highly data-driven and lose spatial structures since images are reformatted to 1D representations. SF-VAE exploits latent-and-RoI pairs to learn continuous representations (RoIs are also continous) and maintains the translation equivariance property, which we believe it is suitable and beneficial for image modeling. We will add more discussion on this point"}, {"review_id": "7i0oSa0Lvy", "reviewer": "Reviewer_BxLE", "summary": "The paper proposes to learn sparse non-grid latent representation for image synthesis. A sparse autoencoder learns a set of non-grid latents and RoIs via reconstruction loss, where the decoder is designed with the neural field approach. A flow-based transformer models the joint flow of the latents and RoIs in the denoising process.", "questions": "- The Table 2 ablates the impact of reducing the number of latents from 64 to 32 for resolution 256 x 256. What if the number is increased to 128? It would be interesting to include the #latent=128 result, as it would match the latent units of SD-VAE and allow for a more direct and fair comparison.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is well-motivated. Many real-world images often show non-uniform information density so it is natural to explore new methods with non-grid representation. \n- The exploration of the sparse non-grid latent representations for image synthesis  could make a meaningful contribution to the research community. Ideas such as decoding latents and RoIs in a neural rendering manner and the asynchronous interpolating schedule for RoIs both offer new insights.\n- The paper is well-written with clearly presented figures. The idea flow is also very clear and easy to follow. \n\n**Weaknesses**\n- For results reported in Table 1, there is no explanation of the difference between *DC-AE-in-f32c32* and *DC-AE-mix-f32c32*. And the numbers for DC-AE on ImageNet differ slightly from the numbers presented in the DC-AE paper. \n- The Figure 5 could be more informative. For example, it would be interesting to compare the visualization of the RoIs w.r.t to different latent numbers, along with the corresponding qualitative metrics (e.g., PSNR). \n- The description of the experimental setting of SF-SiT could be clearer. The resolution info is missing for the results reported in Table 5. Since the SF-SiT-XL in Table 6 is also trained with REPA, it could be clearly indicated in the table. Also, it would be good to include the 512 x 512 results to demonstrate the model's high-resolution generative capability.", "was_revised": true, "final_justification": "After considering the rebuttal, I would like to maintain my original score. The additional experimental results show that the method works with more tokens for reconstruction. However, it remains unclear whether it performs well for higher-resolution generation, as results for 512×512 are missing.", "rebuttal": "We would like to thank reviewer's time for reviewing and comments on our work. We would like to address concerns:\n\n**Q.1. The difference between DC-AE-in-f32c32 and DC-AE-mix-f32c32 and DC-AE results differs from the original paper.**\n\nThe difference between DC-AE-in-f32c32 and DC-AE-mix-f32c32 mainly comes from the training dataset. The DC-AE-in-f32c32 is trained on ImageNet training set and DC-AE-mix-f32c32 is trained on their own mixed dataset according to DC-AE paper. Our reported DC-AE results differs from the original paper because we report different models on standard 50K ImageNet **validation** set while DC-AE original paper evaluates models on ImageNet **test** set (from publicly-available code), which is an uncommon practice.\n\n**Q.2. The Figure 5 could be more informative.**\n\nThank you for your advice on Figure 5 and we will revise it to be more informative.\n\n**Q.3. The experimental setting of SF-SiT and Table 6 could be more clearer. It would be good to include 512x512 results**\n\nThank you for your pointing these out and valuable suggestions! Results in Table 5 are on 256x256 resolution. We will add REPA to SF-SiT-XL in Table 6 to clearly indicate it is trained with REPA. If resources available in nearly future, we will finetune SF-SiT-XL on 512x512 resolutions from SF-SiT-XL w/REPA pre-trained on 256x256 with moderately increased latent tokens and compare with other methods.\n\n**Q.4. What if the token number in Table 2 is increased to 128?**\n\nWe finetune SF-VAE tokens number 128 variant for 20K iterations on ImageNet 256x256 per your valuable advice and report results here:\n\n| method                 | rFID | PSNR  | SSIM  | LPIPS |\n|------------------------|------|-------|-------|-------|\n| SD-VAE-ema-f8          | 0.63 | **24.98** | **0.804** | 0.062 |\n| SF-VAE, 64 tokens      | 0.70 | 23.24 | 0.743 | 0.085 |\n| **SF-VAE, 128 tokens** | **0.35** | 24.81 | 0.798 | **0.060** |\n\nWe can see that SF-VAE w/ 128 tokens achieves competitive performance, especially rFID, compared with SD-VAE-ema-f8 on the same compression rate."}, {"review_id": "GostRR7zxT", "reviewer": "Reviewer_Jekt", "summary": "This paper focuses on developing a compact encoding for images that captures their sparse nature. To achieve this, the authors introduce sparse flow autoencoder (SF-VAE) to encode an image using a small number of learnable RoI tokens their bounding boxes, which is in contrast to the typically pixel-aligned grid. Additionally, the author introduce a sparse flow-based transformer (SF-SiT) which achieves competitive performance to recent diffusion/flow-based transformers in terms of image generation while using a more compact representation.", "questions": "Please see above.", "limitations": "The authors answered in the checklist that there is discussion on the limitations, but there does not seem to be one included.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strength\n\nThe general motivation for a more compact representation based on RoI and bounding box makes sense. The reconstruction results seem to support the viability of the RoI encoding approach. Also, the idea of using a neural renderer for making the decoder differentiable w.r.t. the RoI sounds new and interesting. The generation results also demonstrates competitive bench mark results.\n\n## Weakness\n\nThere does not seem to be any major weakness for this paper. But there are a few minor questions:\n\n1. The RoIs are overlapping with each other quite a lot except the ones near the boundary. It would be interesting to learn more about what each RoI is decoded to, are they memorizing different structures in the image (like a silhouette of certain objects)?\n2. The upsampling approach for decoding the MLP output feature seems to be producing some blobby artifact or at least makes the reconstruction result not reproducing finer details in the input fully (e.g., the black blobs at the river bank and on the glass in Figure 3). Do the authors have any proposal for how to improve this?", "was_revised": true, "final_justification": "The authors has provided additional experimental results addressing the other reviewers' requests. The results seem supportive of main claims made by the paper.", "rebuttal": "We would to thank you for your positive comments and review on our submission. We address the reviewers' concerns as following:\n\n**1. It would be interesting to learn more about what each RoI is decoded to, are they memorizing different structures in the image?**\n\nThis is a good suggestion. Indeed we had visualized each RoI individually a lot during developing this approach and we find that each RoI corresponds to low-level spatial patterns and often come across boundaries of different objects. Since we do not impose any semantic loss/alignment, this behavior is expected. Our method only compresses perceptual details (spatial patterns/textures, not semantic) into latents with RoIs. The RoI size also varies when coming into areas of different frequencies, e.g., low-frequency pixel area coverred by large RoIs. We will add single-RoI-decoded visualizations and discussion on these to the revised version.\n\n**2. How to improve blobby artifact or reproducing details fully?**\n\nIt is a great question but also very hard for every researcher on better VAE. If we run today's advanced VAE such as SD-VAE or Flux-VAE, we can find that these reconstructed results also do not reproduce every details in the input fully. VAE commonly is designed for lossy compression since the latent space size is greatly smaller than the input size. Having said that, we can also increase the model's parameters and computations, especially SFVAE's decoder, to lower the gap. In the submission, we design the pixel decoder to be as light as possible (See response to Q.2. from Reviewer pbLv) and this might be the bottleneck for reconstruction capability. Alternatively, we can increase the latent space size to contain more information, which is beneficial for reconstructing pixels accurately. But this way might hinder the convergency the following diffusion model learning since the sample size will become larger."}]

Author rebuttal:

Meta-review: The paper proposes a non-grid latent space with sparse latent vectors and adaptive Regions of Interest (RoIs) for image reconstruction and generation, aiming to improve efficiency. The models show competitive performance while reducing FLOPs compared to grid-based tokenizers and generators.

Reviewers highlighted strengths such as the novelty of non-grid sparse latent modeling, efficiency improvements in both token usage and FLOPs, and a clear technical presentation.

Key concerns centered on the interpretability of RoIs, which seem to capture low-level patterns rather than semantic structures. Reviewers also noted limited comparisons with 1D tokenizers and scaling baselines, as well as missing experimental details in the original submission.

In response, the authors added experiments on MS COCO, provided a FLOPs breakdown, reported gFID comparisons with TiTok, and expanded ablation studies on RoI size and modeling schedules.

Overall, the AC found the paper interesting and inspiring, with a novel contribution on adaptive and sparse image representations that will benefit research in image generation. While the introduced RoI differs from the usual expectation of semantically coherent RoIs, since the paper did not claim to produce semantic RoIs, the AC considered the approach sufficient for the paper’s goals. The authors are encouraged to clarify their definition of RoI.
Decision: Accept (poster)