Title: Sparse Image Synthesis via Joint Latent and RoI Flow
Venue: NeurIPS 2025
Accepted: yes
Keywords: diffusion models

flow matching

sparse diffusion

sparse generation
Abstract: Natural images often exhibit underlying sparse structures, with information density varying significantly across different spatial locations. However, most generative models rely on dense grid-based pixels or latents, neglecting this inherent sparsity. In this paper, we explore modeling visual generation paradigm via sparse non-grid latent representations. Specifically, we design a sparse autoencoder that represents an image as a small number of latents with their positional properties (i.e., regions of interest, RoIs) with high reconstruction quality. We then explore training flow-matching transformers jointly on non-grid latents and RoI values. To the best knowledge, we are the first to address spatial sparsity using RoIs in generative process. Experimental results show that our sparse flow-based transformers have competitive performance compared with dense grid-based counterparts with significantly reduced lower compute, and reaches a competitive 2.76 FID with just 64 latents on class-conditional ImageNet $256\times 256$ generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on developing a compact encoding for images that captures their sparse nature. To achieve this, the authors introduce sparse flow autoencoder (SF-VAE) to encode an image using a small number of learnable RoI tokens their bounding boxes, which is in contrast to the typically pixel-aligned grid. Additionally, the author introduce a sparse flow-based transformer (SF-SiT) which achieves competitive performance to recent diffusion/flow-based transformers in terms of image generation while using a more compact representation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\nThe general motivation for a more compact representation based on RoI and bounding box makes sense. The reconstruction results seem to support the viability of the RoI encoding approach. Also, the idea of using a neural renderer for making the decoder differentiable w.r.t. the RoI sounds new and interesting. The generation results also demonstrates competitive bench mark results.\n\n## Weakness\n\nThere does not seem to be any major weakness for this paper. But there are a few minor questions:\n\n1. The RoIs are overlapping with each other quite a lot except the ones near the boundary. It would be interesting to learn more about what each RoI is decoded to, are they memorizing different structures in the image (like a silhouette of certain objects)?\n2. The upsampling approach for decoding the MLP output feature seems to be producing some blobby artifact or at least makes the reconstruction result not reproducing finer details in the input fully (e.g., the black blobs at the river bank and on the glass in Figure 3). Do the authors have any proposal for how to improve this?\n\nQUESTIONS:\nPlease see above.\n\nLIMITATIONS:\nThe authors answered in the checklist that there is discussion on the limitations, but there does not seem to be one included.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to learn sparse non-grid latent representation for image synthesis. A sparse autoencoder learns a set of non-grid latents and RoIs via reconstruction loss, where the decoder is designed with the neural field approach. A flow-based transformer models the joint flow of the latents and RoIs in the denoising process.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is well-motivated. Many real-world images often show non-uniform information density so it is natural to explore new methods with non-grid representation. \n- The exploration of the sparse non-grid latent representations for image synthesis  could make a meaningful contribution to the research community. Ideas such as decoding latents and RoIs in a neural rendering manner and the asynchronous interpolating schedule for RoIs both offer new insights.\n- The paper is well-written with clearly presented figures. The idea flow is also very clear and easy to follow. \n\n**Weaknesses**\n- For results reported in Table 1, there is no explanation of the difference between *DC-AE-in-f32c32* and *DC-AE-mix-f32c32*. And the numbers for DC-AE on ImageNet differ slightly from the numbers presented in the DC-AE paper. \n- The Figure 5 could be more informative. For example, it would be interesting to compare the visualization of the RoIs w.r.t to different latent numbers, along with the corresponding qualitative metrics (e.g., PSNR). \n- The description of the experimental setting of SF-SiT could be clearer. The resolution info is missing for the results reported in Table 5. Since the SF-SiT-XL in Table 6 is also trained with REPA, it could be clearly indicated in the table. Also, it would be good to include the 512 x 512 results to demonstrate the model's high-resolution generative capability.\n\nQUESTIONS:\n- The Table 2 ablates the impact of reducing the number of latents from 64 to 32 for resolution 256 x 256. What if the number is increased to 128? It would be interesting to include the #latent=128 result, as it would match the latent units of SD-VAE and allow for a more direct and fair comparison.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses image synthesis from the perspective of sparsity. Specifically, the latent and position attributes of the image are modeled separately, and flow matching is used to generate the latent and position respectively. Compared with the previous grid-based modeling, it has lower computing resources and competitive performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Modeling images as a combination of latents and RoIs from an unstructured view is insightful. This approach could inspire future multimodal learning (e.g., aligning latents with semantics) or efficient video generation, making it valuable to the community.\n\n2. The use of neural rendering to reconstruct images from latents and RoIs is creative. The model achieves 96× compression while maintaining performance close to state-of-the-art VAEs. The proposed RoI-based positional encoding and scheduler are also well-designed, enabling performance comparable to the SiT baseline using only 64 tokens (vs. 256).\n\n3. The paper is clearly written and includes thorough ablations on latent size, resolution, and scheduling.\n\nWeaknesses:\n1. Due to space limits and no supplementary material, many implementation details are missing. For example, as author provides RoI visualization in Figure 3, it's unclear what each RoI-latent pair represents—do they correspond to structured objects or low-level features? Some analysis would help clarify this.\n\n2. Some recent work in discrete image generation explores compressing token representations (e.g., 1D tokenizers [1]). It would be interesting to hear the authors’ views on how their method compares to these approaches.\n\n3. Looking forward to open source and further work.\n\nReference:\n\n[1] An Image is Worth 32 Tokens for Reconstruction and Generation\n\nQUESTIONS:\nSee weaknesses, more analysis will help to better understand how it works\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to learn a latent generative model for images where, in contrast to most previous work, the latents are not structured in a 2D grid. To learn such model, similarly to other latent space models (e.g., LDM), the paper proposes a two-stage approach: the first stage learns a Sparse Flow VAE (SF-VAE), which encodes an input image into a sparse set of 1D latent codes / tokens and RoI representations; the second stage learns a flow model to sample from the target (joint) distribution of latents and RoIs, by adapting a transformer-based model (SiT) to parameterize the velocity field. The paper refers to the latter as Sparse Flow SiT (SF-SiT).\nExplicitly modeling RoIs introduces additional complexity in learning a joint velocity field, especially at timesteps close to t=0, as the model has to rely on a noisy RoI representation to predict a position-aware latent velocity. To alleviate this issue, the paper proposes an asynchronous interpolating schedule, by designing an interpolation scheme where RoIs approach the target distribution more quickly, compared to the corresponding latents.\nThe paper evaluates the proposed approach on the ImageNet generation benchmark, at two different resolutions (i.e., 256x256 and 512x512), showing competitive results (gFID) when compared with baselines that do employ a grid-structured latent space, while reducing number of latent tokens and training compute.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The paper is quite well-written and easy to follow. All the introduced architectural components are presented in detail, making the technical contributions of the paper easy to grasp.\n+ The idea of encoding an image into a sparse set of latents instead of a lower-resolution latent \"image\" is definitely interesting and worth exploring. In fact, while a 2D latent space can benefit from the inductive bias of convolutions in U-Nets, as discussed in [A], it is reasonable to think that this structure can be dropped when relying on transformers for parameterizing flow or diffusion models.\n+ The evaluation on ImageNet benchmark shows better results for the proposed SF-SiT, compared to presented baselines, on a fixed training budget of 400k iterations and when sampling with an Euler ODE solver. Besides obtaining improved gFID results, sampling requires almost 4 times less FLOPs, thanks to the increased latent compression rate. \n\nWeaknesses:\n- This is not the first work introducing the concept of 1D image (sparse) tokenization.  \nFor instance, [B] develops a highly similar idea, presenting a transformer-based 1D tokenizer for sparse tokenization (e.g., 32 tokens) and a decoder that is also based on a ViT and directly decodes 1D tokens into an image.  \nIndeed, a comparison with the \"TiTok\" approach [B] or other immediate follow-up works is missing. Slot-based auto-encoders [C] (and much follow up work) are, up to my knowledge, yet another way of designing 1D image tokenizers, and hence are also worth a discussion in the related work section at least, or possibly a comparison. \n- The paper does not provide a clear motivation of why we need to explicitly model the distribution of RoIs. I think a motivation / ablation against no explicit modeling, or showing an application (if any) of this \"disentangled\" latent space is missing. This is important also because the shown examples show a rather regular tiling of the image into RoIs, similar to the usual notion of patches.\n- It is not clear whether the reduced number of FLOPs at the sampling / denoising stage (i.e., for SF-SiT) is enough to contrast the increase in FLOPs at the decoding stage, mainly introduced by the proposed neural field decoder, which eventually computes N 2D feature maps (to be blended together) from the N input RoIs, conditioned on the corresponding 1D latents. The evaluation of FLOPs for the entire end-to-end generation would give more insights on the advantage of using the proposed approach. \n- An initial attempt of efficiently scaling the training budget with REPA shows promising results on class-conditional generation (with and without CFG), although the results are still not on par with baselines. A comparison with SiT-XL using REPA for faster convergence under the same training budget and with CFG is also missing, thus it is hard to infer how good the proposed approach can scale.\n\nNote: My assessment is somewhere between 2 and 3 here. I eventually settled for a 2, since the missing comparison to [B] and follow-up work is a significant oversight, as is the missing discussion of safeguards.\n\nReferences:  \n[A] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022). High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10684-10695).  \n[B] Yu, Q., Weber, M., Deng, X., Shen, X., Cremers, D., & Chen, L. C. (2024). An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37, 128940-128966.  \n[C] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., ... & Kipf, T. (2020). Object-centric learning with slot attention. Advances in Neural Information Processing Systems, 33, 11525-11538.\n\nQUESTIONS:\n1. Can the authors compare the proposed approach against other 1D image tokenization strategies? One straightforward example could be the already mentioned \"TiTok\" [B], which also evaluates on the ImageNet 256x256 benchmark. \n2. Can the authors compute the total number of FLOPs for sampling an image in pixel space? Which means, also including the necessary operations to convert your 1D tokens into 2D feature maps via the neural field decoder into the total amount of flops.\n3. Can the authors report (cf. Table 6) the result (gFID) obtained with the SiT-XL baseline using REPA for faster convergence and CFG scale > 0? It will be helpful to really show the potential of the proposed SF-SiT-XL with limited compute. \n4. Can the authors provide more insights on why we need to (explicitly) model the distribution of RoIs? This question is also partially motivated by the fact that the RoIs (cf. Figure 6) seem to flow to a fairly even distribution where each token still corresponds to a small patch of your image. Ablating this choice can be a very good starting point.\n5. Can the authors provide a more convincing discussion of the issue of safeguarding their image generation approach?\n\nIn general, I would consider increasing my score upon seeing a comparison with other previous works on 1D image tokenization, which is currently missing from the paper. Also, the issue of safeguarding is not handled well. Thus, (1) and (5) seem to be the most relevant point to address. Addressing other points (2-4) would be helpful to have more insights on the significance of the proposed approach.\n\nLIMITATIONS:\nThe paper is currently missing a paragraph discussing (or briefly summarizing) the limitations of the approach. Although this is already shortly mentioned in the conclusion, it would be helpful to have a separate paragraph mentioning that the goal of this work is not to target SotA generation results, but rather to demonstrate the applicability and scalability of a new image tokenization approach. Another additional point should make it clear that the initial hypothesis has been verified on limited compute (i.e., training time), but it is still to demonstrate whether this approach can scale to more compute and surpass grid-based latent representations, while being faster and less memory-intensive.\n\nI disagree with the answer to question 11 in the questionaire. This is an image generation model and as such the question of safeguards is indeed a relevant issue. While the safeguards needed here are likely no different than those for most of the models compared to, I do believe that this issue cannot be swept under the rug quite this easily. The paper should at least reflect on whether safeguards may be necessary and share these reflections with the readers (and reviewers). References to relevant work on safeguarding of image generators would also improve the paper.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce SF-VAE, which encodes an image into a small number of latent vectors along with learnable Regions of Interest, and SF-SiT, a flow-based transformer that jointly models latent and RoI trajectories during generation. This work achieves competitive image synthesis results onImageNet data, reducing computational cost significantly compared to conventional grid-based methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghs:\n1. The introduction of RoI-based latent modeling in a diffusion/flow generation pipeline is an  interesting idea.\n2. This RoI base mehtod achieves strong FID scores using significantly fewer tokens and FLOPs than conventional models (e.g., DiT, SiT).\n\n\nWeaknesses\n1. All experiments are restricted to image generation (reconstruction or synthesis) on ImageNet, without any assessment on image understanding tasks such as classification, retrieval, or layout prediction. This leaves open whether the learned latent representations are semantically meaningful or generalizable.\n\n2. Furthermore, ImageNet mainly contains single-object, center-biased natural images. These are spatially sparse by design and make it easy for the RoI model to leran by locating obvious foregrounds. As such, the proposed method is not stress-tested on structured, multi-object, or text-rich data where adaptive sparsity is most needed.\n\n3. The paper lacks any attempt to quantify the semantic alignment, consistency, or interpretability of the learned RoIs. It is unclear whether RoIs focus on objects, backgrounds, or arbitrary textures. The authors can apply this learned representation on vision-langauge problems.\n\nQUESTIONS:\nHow does your method perform on dense or structured visual inputs, such as documents, charts, or scenes with many small objects? ImageNet is not sufficient to test this.\n\nWhat happens when the number of RoIs is insufficient for reconstruction or generation? Are there strategies to adaptively allocate more tokens to difficult regions?\n\nLIMITATIONS:\nThe paper does not discuss how the method would generalize to domains with high spatial complexity (e.g., documents or multi-object scenes), nor whether RoIs are semantically meaningful. These are crucial questions given that the method is explicitly designed to model spatial sparsity.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}