Title: Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior
Venue: NeurIPS 2025
Accepted: yes
Keywords: Image Coding for Machines

Diffusion Model

Generative Image Compression
Abstract: Image compression methods are usually optimized isolatedly for human perception or machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks.
Based on this insight, we propose Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training.
Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags to stimulate the pre-trained diffusion model's generative capabilities, requiring minimal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through a single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose that image compression meant for downstream tasks should be built accordingly to emphasize more semantic fidelity. They propose Diff-ICMH, a generative image compression method based on pretrained diffusion models that tries to do just that, while also optimizing for perceptual quality. Specifically, they train a new encoder-decoder to compress the latent representation of a given image (incorporating a semantic consistency loss), and on the client side decompress the image by guiding the diffusion model with image tags (also transmitted) and by using controlNet to condition the model on the decompressed latent vector. They show strong performance on downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- Using tags instead of automatically generated sentences (e.g. by using BLIP) is a good idea.\n- The proposed method exhibits strong performance across a diverse range of downstream tasks, demonstrating its adaptability and effectiveness.\n- I believe using pre-trained diffusion models is a good direction for compression algorithms, leveraging their generative capacity to enhance visual quality. \n- Faire reporting on PSNR comparing non-latent based methods (showing that using a VAE induces limitations).\n\n## Weaknesses\n- The novelty is somewhat unclear to me. The claim that combining perceptual optimization and Image Compression for Machines (ICM) is novel is somewhat questionable. The concept of using a single codec for multiple downstream tasks has been previously explored (e.g., DDCM [1]). Furthermore, traditional codec-based methods arguably serve similar roles and should not be excluded from such categorization. \n- Motivation: In L110 the authors argue that traditional image compression methods harm semantic integrity, but I see no evidence that they perform poorly on that front. On the contrary, perceptually oriented methods inherently maintain semantic accuracy, making this assertion debatable.\n- **Clarity Issues and Structural Organization:**\n  - Figure 1 is unclear to me, conveying very limited information (and the caption does not explain as well). I don't see why it should occupy so much space in the main paper. Instead, the authors should use that space for the image examples they lack. \n  - Section 4.1 – It's unclear if the generative process follows the standard diffusion process (does $z_t$ starts from random noise here)?\n  - It's not explicitly stated what information is transmitted and compressed, and the reader must infer it consists of the compressed latent space representation alongside the tag bits. This should be clearly written.\n   - Figure 7 lacks organization. Metrics are arranged seemingly at random, making it difficult to follow ((E.g., Kodak PSNR is at [0,0], its LPIPS is in [1,0], it's MS-SSIM is [0,2], and DISTS in [1,2]). Not all metrics mentioned in the caption are shown for each DB, e.g. Kodak lacks FID. If these are omitted due to the number of images in the DB, patches should be used instead (this is quite commonplace, see e.g. [1])\n  - Figure 5: the arrow comes out from the bottleneck of the diffusion model, which suggests the semantic space you consider is the h-space of the diffusion model. The text however implies it is the VAE latent space. Please clarify. Additionally, Eq 1 seems to suggest this is not happening per time step, which confuses me about when the loss is performed.\n  - Ablations in Section 5.4: The mAP metric lacks context—was it calculated on a specific task or averaged across multiple tasks? L276 mentions better rate-distortion performance, but the evidence supporting this claim is unclear.\n- The paper includes multiple comparisons to PerCo. However, their internal model was not released. Did the authors train their own PerCo version, or used the open sourced version, \"Perco (SD)\"? If the latter, the corresponding paper [2] should be cited. \n- The DDIM step count differs between methods—Diff-ICMH utilizes 50 steps, whereas PerCo is restricted to 20 steps. PerCo’s perceptual quality may suffer from this shorter inference process. A fairer comparison using equivalent step counts would be preferable.\n- The ablations are lacking with regards to the necessity of TGM. Recent methods seem to not need text guidance for similar bitrates ([1]).\n- Visualizations: The decision to limit visual examples to two and relegate perceptual comparisons to the appendix is problematic, to say the least. Given the claim of perceptual superiority, a more thorough presentation in the main paper is warranted, along with additional examples in the appendix. Furthermore, these perceptual comparisons (Fig 12) do not compare to SOTA diffusion based methods such as DiffEIC, PerCo or DDCM, raising concerns regarding the fairness of comparisons to similar generative priors. Also, in the first image the proposed method's BPP is one of the highest in the figure, leading to unfair comparison, for a fair comparison the proposed method should have the same BPP roughly or lower. Finally, the zoomed region in Figure 12 would be better positioned as an additional row rather than obscuring portions of the original image.\n- The results in Figure 7 appear identical to those reported in DiffEIC. If data reuse occurred, this should be explicitly acknowledged and cited. Is that the case? If so, the authors should verify their evaluation protocol to ensure the reuse did not affect the validity of the reported performance.\n- The paper lacks discussion on trained, partially trained, and zero-shot compression methods, which would provide a more balanced perspective on result comparisons (L251)\n- Table 2 reports speed comparisons across different hardware settings, which does not make sense. \n- This method requires training (or fine-tuning) for every new desired bit-rate.\n\n[1] Compressed Image Generation with Denoising Diffusion Codebook Models. G Ohayon, H Manor, T Michaeli, M Elad - arXiv preprint arXiv:2502.01189, 2025\n\n[2] PerCo (SD): Open Perceptual Compression. N Körber, Workshop on Machine Learning and Compression, NeurIPS 2024.\n\nQUESTIONS:\n- L251: The paper suggests that tag-based conditioning incurs minimal overhead, but explicit numbers would help quantify this claim. 100 bits for a 40K words dictionary would be less than 10 words? The authors mention they set a limit, this should be reported.\n- The authors use Kodak images, but considering their size (768×512), how well do they align with the 768x768 SD 2.1?\n- L183: From my knowledge and experience, the VAE latent space isn't very smooth (e.g. [3]). It baffles me that optimizing similarity in this latent space helps enforce actual similarity. How do you explain that? Also, how does this affects the robustness of the downstream tasks results?\n\n[3] Norm-guided latent space exploration for text-to-image generation. D Samuel, R Ben-Ari, N Darshan, H Maron, G Chechik - Advances in Neural Information Processing Systems, 2023.\n\nLIMITATIONS:\nNo, the authors mention in the checklist they do in the appendix, but this is very briefly touched upon only in a complexity analysis. No mention at all about the different training needed for different bit-rates.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-07"}

comments: SUMMARY:
This paper proposes a diffusion-based ICMH framework, which is implemented by a control-net and proposed *TGM*. The network is trained by rate-distortion loss, diffusion loss and proposed *SC loss*. The main contribution is that the proposed framework reserves both semantic and perceptual information, appending the generated information to obtain better down-stream task performance. This method achieves SOTA performance across various downstream tasks.

STRENGTHS AND WEAKNESSES:
### Strengths

- The motivation is clear and convincing.
- This paper is well experimented with ablation studies on proposed network and SOTA comparisons.
- The proposed method achieves SOTA performance across various downstream tasks.

### Weaknesses

- The contributions presented in this paper are a little insufficient: Most of the modules in the framework (e.g., autoencoder, tag extractor and stable diffusion) are pretrained.

- Poor inference efficiency: ICMH should be a very practical application-oriented field, as it is often designed for downstream tasks. However, the reported encoding/decoding time is much longer than existing methods.

QUESTIONS:
- Could you provide more ablation studies evaluated on FID and LPIPS metrics ?
- Why do you adopt the middle feature in the diffusion model to calculate the *SC Loss* ? The network architecture of diffusion models should be a U-net. The middle feature of a U-net only contains the lowest scale information, which is clearly NOT like the bottleneck of an autoencoder that contains full compressed semantic information.

LIMITATIONS:
No discussion on limitations.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Diff-ICMH proposed in this paper is a generative image compression framework that utilizes a diffusion-based decoder and a semantic bootstrap module. The core claim is that Diff-ICMH can support a wide range of downstream intelligence tasks - including traditional computer vision tasks and multimodal reasoning - without the need for task-specific adaptation. To support this claim, the authors have experimented on several benchmarks, including object detection, instance segmentation, keypoint estimation, and full-view segmentation on COCO; image text retrieval on Flickr30K; finger representation understanding on RefGTA; and open vocabulary segmentation on ADE20K.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe authors claim that Diff-ICMH supports a wide range of tasks, including both traditional computer vision tasks and multimodal reasoning, without the need for any task-specific adaptation. This is a compelling and practically valuable goal, especially in the context of compressing images for general-purpose downstream usage. The authors provide empirical evidence across several tasks (e.g., object detection, instance segmentation, keypoint estimation, and panoptic segmentation on COCO; image-text retrieval on Flickr30K; referring expression comprehension on RefGTA; and open-vocabulary segmentation on ADE20K). These evaluations span different types of downstream models and include tasks involving both vision and language.\n\nWeaknesses and Questions:\n1. Limitation: A notable limitation in this manuscript is the lack of discussion of the limitations of the proposed approach. Diffusion-based decoders are known to be computationally intensive. Could the authors provide further information on the considerations and issues regarding trade-offs in terms of inference efficiency, cost and resources. Providing quantitative metrics or qualitative insights on these aspects would allow reviewers and future readers/users to better understand the practical deployment implications of Diff-ICMH.\n2. Task-Independent Support: The claim of “task-independent support” remains somewhat under-tested. First, the experiments do not indicate whether any downstream tasks or their semantics were encountered during the training or labelling phases. This raises the question of whether the model truly generalizes in a zero-point manner or benefits from task-aware conditioning implicit in the training process..\n3. Tag Guidance Module (TGM): The authors have conducted experiments using a Tag Guidance Module (TGM) to improve performance at negligible bitrate cost. While this design is somehow efficient, it is still too simple—relying on static tag extraction and direct prompt injection. Why doesn’t the paper explore TGM's robustness and effectiveness under more complex semantic conditions? Why not compare it with more powerful alternatives such as CLIP-based conditioning or learned prompt optimization methods? Any of these trade-offs or comparisons could significantly enhance the paper’s quality. Without such analyses, the current TGM contribution, while practical, may be perceived more as an ad-hoc engineering choice than a principled or generalizable advance. Providing such evaluations would not only strengthen the paper but also offer clearer guidance for future researchers considering whether and how to apply TGM in their own works. Since prompt-based testing is not difficult to conduct experimentally, the authors could consider including their TGM evaluations under more complex scenarios as well as comparisons with alternative prompt-based guidance methods, depending on their available resources.\n4. Generalizability: The generalizability of the model to completely unseen domains or tasks (e.g., medical imaging, real-time video comprehension, or other non-visual language settings) was not explored. Thus, while the current empirical results are encouraging, the strength of the task-independence statement would be greatly enhanced by including evaluations under unseen or out-of-distribution tasks, comparisons to a task-adaptation baseline, and explicit measurements of the cost of adaptation (e.g., training time, number of fine-tuning parameters, or latency).\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nSee weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity.\n\nSTRENGTHS AND WEAKNESSES:\n**Pros:**\nThe paper addresses an important and timely research problem. Traditional lossy image compression methods are typically designed to preserve visual fidelity for human perception or storage/transmission efficiency. However, in many real-world applications, compressed images are used as inputs for downstream vision tasks. The authors correctly identify this gap and motivate their work around optimizing compression for task performance, which is a valuable direction.\n\n---\n\n**Mixed:**\nThe proposed architecture is built upon ControlNet, and the Tag Guidance Module appears to function similarly to existing tagging models. This raises some concerns about novelty. That said, the integration of a semantic consistency loss is a noteworthy aspect that could contribute to the uniqueness of the approach.\n\nIt would be helpful if the authors elaborated on their choice of similarity metric for enforcing semantic consistency—e.g., whether they used cosine similarity, Euclidean distance, or another measure—and discussed its impact on performance.\n\n---\n\n**Cons:**\nThe empirical performance of the proposed method appears limited. As shown in Figure 6, on widely used datasets such as COCO and Flickr, the method performs on par with existing perceptual compression baselines. Furthermore, Figure 7 indicates no significant improvement in perceptual quality, which weakens the empirical claims of the paper.\n\nAdditionally, the paper brings to mind related work such as [Matsubara et al., WACV 2022](https://openaccess.thecvf.com/content/WACV2022/papers/Matsubara_Supervised_Compression_for_Resource-Constrained_Edge_Computing_Systems_WACV_2022_paper.pdf), which explores supervised compression for multiple downstream tasks using a model distillation framework. This prior work seems highly relevant, yet the current paper only compares against reconstruction-based baselines. Including comparisons with task-specific or multitask compression approaches would strengthen the experimental analysis.\n\n---\n\n**Recommendation:**\nI believe this paper sits on the borderline for acceptance. The motivation is strong, and the proposed semantic consistency idea has potential, but concerns about novelty and limited empirical gains weaken the overall contribution. I would consider increasing my score if the rebuttal addresses these concerns effectively.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nNo. The authors should include a discussion of the model’s limitations. For example, perceptual-quality-oriented compression methods may introduce hallucinated details that were not present in the original image. While this may improve visual appeal, it can be problematic in domains where accuracy and fidelity are critical—such as medical imaging or scientific analysis—where hallucinated content could lead to misleading or incorrect interpretations. Highlighting this trade-off would strengthen the paper by providing a more balanced perspective on the method’s applicability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}