Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior
Authors: Ruoyu Feng
Yunpeng Qi
Jinming Liu
Yixin Gao
Xin Li
Xin Jin
Zhibo Chen
Venue: NeurIPS 2025 poster ()
Keywords: Image Coding for Machines
Diffusion Model
Generative Image Compression
Primary area: applications

Abstract: Image compression methods are usually optimized isolatedly for human perception or machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks.
Based on this insight, we propose Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training.
Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags to stimulate the pre-trained diffusion model's generative capabilities, requiring minimal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through a single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception.

Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior
Ruoyu Fengl: 1
Yunpeng Qi
Jinming Liu? Zhibo Chen't
Yixin Gao
Xin Li
Xin Jin?
University of Science and Technology of China Eastern Institute of Technology, Ningbo
Abstract
Image compression methods are usually optimized isolatedly for human perception machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks. Based on this insight, we propose Diff-ICMH, a gen- erative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training. Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags stimulate the pre-trained diffusion model's 's generative capabilities, requiring mini- mal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception.
Introduction
The digital era has driven explosive growth in network data. Compression algorithms crucial in the storage and transmission. Image compression, a cornerstone of visual signal processing, is essential for both technological advancements and the operational efficiency of numerous digital systems. In terms algorithmic development, traditional compression standards such as JPEG (I. JPEG2000 (2), H.264/AVC (3), H.265/HEVC [4], and H.266/VVC (5) have been widely applied. More recently, learned image compression methods [6/23] have emerged, demonstrating remarkable performance. However, these methods are primarily optimized for human visual perception. Concur- rently, the rapid development of artificial intelligence technology means that the scale of visual data consumed by downstream intelligent tasks growing substantially. This shift in application scenarios highlights the urgent need for developing compression methods tailored for intelligent tasks.
Representative existing approaches to image compression for machines (ICM) are depicted in Fig. [ (c)- Traditional codec-based methods optimize compression for machine tasks through quantization parameter tuning [44 strategic bit allocation 33] by integrating neural
network-based pre/post-processing modules [4 37. While these approaches enable a single codec
"Equal contribution. Conrespnninng authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Traditional Compression
Learned-based Compression
81
Task T1
8
Task T22
82
Task 72
8
Task
T3
93
Task T33
(a] Traditional codec based methods
(b) End-to-end optimized methods
Feature Extraction
Feature Comaression
Semantic Preserved
f1
Task 71
Generative Compression
Task
x
f2
Task
8
Task 72
fo
Task T'3
Task T3
(c) Feature compression-based methods
(d) Diff-ICMH
Figure 1: Comparison of compression and reconstruction workflows between differentt methods.
to support multiple tasks, their generalization and performance are often constrained due to the codec's inherent fidelity-oriented design and its non-differentiable nature. Task-driven end-to-end optimized methods [38 45) typically achieve superior performance on specific tasks but exhibit limited generalization at both the bitstream and codec levels. Feature compression bbsed methods [46]- 6I directly compress intermediate features from specific task models, often yielding a more favorable rate-distortion trade-off, while also reducing the computational burden the server-side. However, such methods still face generalization challenges across different intelligent task models.
Optimizing image compression solely for specific intelligent tasks limits generalization across diverse tasks and human perception. To overcome this, we introduce Diff-ICMH (Fig. (d)), a conditional generation approach targeting both versatile machine task support and human visual quality. From novel perspective, we identify that performance degradation in intelligent tasks using compressed images primarily stems from two information losses: (1) compromised semantic integrity (loss of core intelligible meaning). which directly impairs task analytics; and (2) reduced perceptual realism (textures and details deviating from natural distributions). leading to distribution mismatches that hinder feature extraction and cause inaccurate semantic analysis. Diff-ICMH tackles these issues by first employing diffusion model-based generative framework, leveraging pre-trained models like Stable Diffusion [62). to ensure perceptual realism and mitigate domain shifts. Crucially, to preserve semantic integrity, introduce Semantic Consistency loss (SC loss), which aligns features extracted by the pre-trained diffusion models. Furthermore, a Tag Guidance Module (TGM) utilizes efficiently coded, word-level image tags activate generative priors, thereby enhancing both the subjective quality and semantic clarity of the reconstructed images.
The contributions of this paper
follows:
We introduce an innovative perspective for designing a versatile codec that jointly serves multiple intelligent tasks and human visual perception, identifying semantic fidelity and perceptual realism critical determinants for this unification.
Building on this insight, we introduce Diff-ICMH, which integrates generative priors with robust semantic information preservation, realized through proposed Semantic Consistency loss and Tag Guidance Module.
Extensive experiments validate our approach, showcasing state-of-the-an performance across 10 diverse downstream intelligent tasks. All results are achieved without task-specific adaptation training, while concurrently delivering high-quality reconstructions for human visual perception.
Related work
2.1 Image compression for machines
Research in image compression for machines (ICM) has predominantly explored three primary directions. First, traditional codec-based methods adapt standardized formats like JPEG [] and

---

Semannti Information
Sem. Info. Perc. Info.
Lossy Compression
Info.
Decade
Perceptual
Perc.
Information
Info.
(a) Firelity-Oriented Image Compression
Sem.
Sem. Info. Perc. Info.
Information
Semantic-Preseene Preseeiin
Info.
Generative
Generated Info,
Perceptual
Compression
Perc. Info.
Reconstruction
Information
(b) Diff-ICMH
Figure Illustration of the information transformation during compression processes in signal fidelity-oriented image compression and Diff-ICMH.
H.265 4 by optimizing quantization parameters 24 29 employing strategic bit allocation 6D
While these methods offer the advantage of supporting multiple tasks with a single codec, their inherent signal fidelity-oriented design often limits their generalization capabilities and can lead to performance bottlenecks. Task-driven end-to- end optimized methods utilize learned codecs directly optimize for specific machine tasks [38 43). However, task-driven optimization often leads to poor generalization across different tasks.
or applying neural network pre/post-processing 34 37
To mitigate this limitation, recent works 44 45 4
implement adaptation mechanisms aimed at enabling efficient task adaptation with minimal trainable parameters in the codec. Feature compression methods (46-6]) directly encode intermediate neural network features instead of images, offering efficiency gains especially in cloud-edge scenarios, though they remain tightly coupled with specific feature extractors and cannot support human viewing.
2.2 Generative image compression
The basic objective of lossy compression to optimize the trade-off between bit rate and quality. However, because traditional fidelity-oriented metrics like Peak Signal-to-Noiss Ratio (PSNR) often correlate poorly with human visual perception, generative codecs have emerged as a promising direction for enhancing perceptual quality. Foundational theoretical work on rate- dinte-distttion-pp perteppton relationships (63-65] has catalyzed practical advances in this field. Two predominant approaches have gained prominence, First, GAN- based [66] methods 67 72 typically employ adversarial training to enhance perceptual quality. Second, diffusion model-based approaches [73 83] optimize for rate, distortion, and generation quality, either by training models from scratch or by leveraging large pre-trane tre-trained models like Stable Diffusion (22]. Both approaches can be seen using decoded features as conditional signals guide perceptually realistic reconstructions. The ongoing advancement of generative models holds significant promise for their integration into image compression, paving the way for high-fidelity visual reconstructions at low bit rates.
Motivation
As shown in Figure 2 (a). traditional signal fidelity-orienedd image compression methods 21 typically focus on indiscriminately minimizing pixel-wise errors, such as Mean Squared Error (MSE). This indiscriminate optimization, however, often incurs both semantic distortion and pixel-level perceptual mismatch. Semantic distortion directly impacts the information completeness crucial for downstream tasks. Furthermore, perceptual mismatch manifests as domain shifts, leading to error propagation during feature extraction and hindering the accurate mapping of inputs to semantic features. Generative image compression, by its very nature of reconstructing images to mimic natural distributions, is inherently well-suited to mitigating such domain shifts.
To investigate this, we compared feature divergence (1 minus cosine similarity) between fidelity- oriented codecs (VTM-18.2. ELIC) and a GAN-based generative one (MS-ILLM (72). As illustrated in Figure 3 while orieete codecs exhibited smaller feature differences at shallow network layers (stem layer) due superior signal fidelity (higher PSNR), they suffered significantly larger divergence at deeper layers (layer2 and layer4). This confirms that the realistic textures produced

---

VTM-18.2
bpp:0.044
PSNR:24.7
Averago Offfrecce: 01:10
Averaee Difference 0.197
Arerage Difference: 0.464)
ELIC
bpp:0.046
PSNR:24.9
Ararage Offerecce 03225
Average DiPfrrnner 2135
Arerage Differncee 0-4577
MS-ILLM
bpp:0.043
64
PSNR:23.6
Diffearce 1990
Everage Differemen 0.2642
Onerage Diflrrnze 0.2354
Network Depth
Reconstruction
stem layer
layer2
layer4
Figure 3: Variation in the difference (1 minus cosine similarity) between features extracted from reconstructed images by different codecs and those from original images when input to pre-trained ResNet50 84) shown against increasing network depth.
by generative image compression effectively mitigate error accumulation arising from domain shift. However, realistic textures alone are insufficient for optimal intelligent task performance semantic information integrity is compromised. Accurate extracted representations require both realistic textures and complete semantic information.
Diff-ICMH addresses these dual challenges based on a novel design philosophy: robust semantic information prioritized and preserved during encoding, which then guides a generative reconstruc- tion of perceptually realistic details during decoding, illustrated in Figure[] (b). We realize this by developing generative compression method leveraging a pre-trained Stable Diffusion model [62], further enhanced by our proposed Semantic Consistency loss (SC loss) and Tag Guidance Module (TGM). The subsequent sections will detail the overall framework of Diff-ICMH along with the specifics of the SC loss and the TGM.
Diff-ICMH
4.1 Overall framework
The Diff-ICMH framework depicted Figure The input image is compressed and reconstructed latent feature N. targeting the VAE latent space of the pre-trained Stable Diffusion model. Concurrently, the tag extractor derives word-level tags e from x to capture coarse-graine searse-raaine semantics. In practical usage, the bitstream contains the compressed latent variables and the extracted tag IDs.
key design choice in Diff-ICMH the decoding of the bitstream directly into the VAE latent space of the pre-trained diffusion model instead of pixel space. This is motivated by several compelling properties of the latent space. It inherently optimized for feature-level perceptual quality and realism, result of its training objectives that include perceptual-oriente losses [66]. Furthermore, this space provides a compact and perceptually rich representation (e.g., 8 x 8 spatial downsampling in Stable Diffusion) which effectively filters semantically irrelevant redundancy from the pixel domain. Consequently, optimizing for fidelity within this latent space enables the bitstream to prioritize perceptually salient and semantically coherent information, crucial for robust performance in downstream machine tasks.
LPIPS loss
(85) and adversarial
Generative reconstruction is then conducted with & as condition. Specifically, 2 is fed to the control module, while the noisy latent z, (at timestep input to the diffusion model. The control module adapts ControlNet [86] -ike architecture for conditioning on the downsampled latent feature 2. Subsequently, generative reconstruction performed using the control module and the pre-trained Stable Diffusion model. The reconstructed feature & and the latent feature 2, at timestep are input the control module and diffusion model, respectively. The diffusion model then predicts the noise Eg (z 2, c, t). Following the standard reverse diffusion process for I' steps, obtain the denoised

---

Codec
Control Module
Zero Conv
D
"doy, sat, human,
33
eat, human,
Lt
Eo (7i, 2, c,
Tag Guidance Module
Diffusion Model
Encode and Decode
Denoise (Repeot T Iterations)
Figure 4: Overview of Diff-ICMH. Diff-ICMH consists of two parts: (left) image encoding/decoding and tag extraction; (right) condition-based image reconstruction. For simplicity, skip connections are omitted in the diagram.
latent zn. This 2o then passed through the VAE decoder D(.) )) to yield the final reconstructed image x. See Appendix for the detailed architecture of control module.
4.2 Semantic Consistency loss
To effectively preserve crucial semantic information during lossy coding, which is often compromised in traditional approaches, we propose the Semantic Consistency loss (SC loss). The core idea is to enforce semantic alignment between representations derived from the ground truth and the decoded signal. This is achieved by framing it as a pretext task: both signals are projected into shared, high-quality semantic space where their representations are encouraged be consistent.
The choice of this semantic space is critical for ensuring that the preserved semantics are both rich and generalizable for diverse downstream applications. Recent works [87- have demonstrated that large-scale pre-trained diffusion models possess strong inherent capabilities for image understanding and semantic feature extraction. Inspired by this, leverage features extracted by these pre-trained diffusion models to instantiate our semantic space and guide the SC loss.
Specifically, shown in Figure 5 (a), the ground truth latent variable E(x) and the decoded feature = (e(EE(x)) are separately input into the pre-trained diffusion model, where & indicates the VAE's encoder and Ec, De corresponds the the codec's 's encoder and decoder. We utilize the model's forward propagation f()) as a bridge mapping from the original latent space to the semantic space, and align the two in this semantic space, optimizing the encoder-decoder through backpropagation.
This alignment enforced by maximizing the similarity between the semantic representations fz) and f2)). The SC loss, Leem, is therefore formulated as:
N
Liem - -Ez
N
where represents the number spatial positions in the feature, n indicates the spatial position index of the feature, and sim(.,-) is predefined similarity measurement function. In this paper, instantiate the sim(,.) function using cosine similarity:
x² 2
sim z, 2)
|z|2 2/2
4.3 Tag Guidance Module
To activate generative priors in pre-trained diffusion models with minimal bitrate overhead, introduce the Tag Guidance Module (TGM). As depicted in (b) a pre-trained tag extractor E, (c.g., Recognize Anything [93] is used in this paper) first generates instance-level semantic tags for the
5

---

800
Dif/usion Model
Tag Extractor
Consistency
"cat,
367.
367.
"cat,
washbasin,
287,
0 -
287.
washbasin, ground"
2/(5(0))
ground"
1276
1276
Queryling and mapping
Diffusion Model
Dictionory
f(2g))
(a) Semantic Consistency Loss
(b) Tag Guidance Module
Figure 5: (a) Semantic Consistency loss. The latent variable z and the decoded latent variable 2 are projected through the pre-trained diffusion model, resulting semantic representations that are then aligned. (b) Tag Guidance Module. Tags are extracted via tag extractor and losslessly compressed.
input image x. These tags are subsequently mapped to numerical indices using a predefined dictionary and then losslessly encoded. At the decoder, these indices converted back to their corresponding word-level tags via the dictionary. Finally, these tags are formatted as comma-separated text strings and input conditioning to both the diffusion model and the control module.
This tag-based guidance mechanism incurs very low bitrate overhead (approximately 100 bits per image) due to the typically small number of tags per image and compact predefined dictionary. We therefore employ simple fixed-length coding for the tag indices. During the inference stage, Classifier-Free Guidance (CFG) (94] is utilized with these text tags to steer the diffusion model towards generating reconstructions with more distinct and accurate semantic content.
4.4
Loss function
Our final optimization objective, Limat combines four components: a rate loss Crates latent space reconstruction loss Laias diffusion model noise prediction loss Lair. and the SC loss Lyem-
The rate loss Lrae penalizes the estimated entropy of the quantized primary latent variables ý and the quantized hyperprior latents 2n, following previous methods [7]
Lrate = R(ý) + R(2x).
(3)
During training, quantization is approximated by adding uniform noise 6 and we use the straight through estimator for the input of synthesizer.
The latent space reconstruction loss Luss measures the distortion between the target VAE latent (x) and reconstructed latent 2 D.) We define it as the Mean Squared Error (MSE):
Laist 2|/ [xvv D.(5)
where EYae is the encoder the pre-trained VAE (associated with the diffusion model) and D. is the codec's decoder operating ý.
The diffusion model's 's noise prediction loss Lai follows the standard formulation:
Lait Fgt.cc..eN..... [||e - c0(2, à, c, [[]]
(5)
where z is the clean VAE latent (serving as Zo for the diffusion process), z, its noised version at timestep t, is the sampled Gaussian noise, eg is the network's predicted noise, à is the reconstructed latent from our codec acting as a condition, and c represents tags from TGM.
The final composite loss function is weighted sum of these components:
Linal Crate Caist + Aair + Grems
(6)
where Arate Adists Agitt and Asem scalar weights balancing each term.

---

VTM-18.2
FTIC
MS-ILLM
DITIEIC
Adapt-ICMH
(Ours)
Object Detection Bbox AP COCO2017
Instance Segmentation
Pose Estimation
Panoptic Segmentation
Mask COCO2017
Keypoint AP COCO2017
PQ T COCO2017
35
25
25
20
10
10
15
0.0
0.1
0.2
0.00
0.05
0.10
0.15
0.20
0:25
0.39
0.00
0.05
0.10
0.15
0.29
0.25
0.30
0.0
9.2
0.3
bpp Retrieval
bpp
hpp
Accuracy Comprehension
R@81
70
80
73
80
70
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.50
2.05
0.10
0.15
3.20
0.25
0.30
0.500 0.025 20500 0.675 0.15 5.125 6.15 0.15 0200
bpp
bop
bop
Open-Vocabulary Semantic Segmentation
Open-Vocabulary Instance Segmentation
Open-Vocabulary Panoptic Segmentation
on
PQ ADEZOK
28
25
35
24
25
30
22
20
25
0,000 0.025 0.050 0.075
0.125 0.250 9.175 0.200 0.000 5.025 0.050 0.075 5.100
0.125
0.150 0.1/5 0.200
0.000 0.025 3,000 0.875
0:100
6.125 0.150 0.175 0.200 bpp
Figure Performance comparison of Diff-ICMH with other methods across diverse intelligent tasks.
5 Experiments
5.1 Implementation details
Dataset. We use LSDIR (95] as our training dataset. During training, images from the dataset are randomly cropped to size of 512 x 512.
Model setup. The experiments Stable Diffusson 2.1 as the pre-trained diffusion model. The control module initialized the same way as ControlNet [8). by copying parameters from the Stable Diffusion model for initialization, and initializing the weights of the Zero Convolution (86) to O. During training, the pre-trained Stable Diffusion model and the tag extractor & remain frozen, while the encoder-decoder and control module parameters learnable. For inference, we follow the standard DDPM process 196] 62 and start denoising from pure Gaussian noise.
Hyper-parameters In equation (6). Adist- Aums and A are empirically set to 1, 1, and 2 respectively, while Avate is set to 2, 4, 8, 16, 32 to obtain codec models at different rates. We use the Adam optimizer [97] with 31 and B2 set to 0.9 and 0.999 respectively, and training batch size of 16. Training is conducted in two stages. First, we train for 200K iterations with >mme = and a learning rate of le - obtain high bit rate model. Then, fine-tune for another 200,000 iterations with learning rate of 5c across all Amme values to obtain models for all bit rate points. For Classifier-Fres Guidance (CFG), text tags dropped with probability of 0.1 during training. During inference, the CFG Scale is set to 5.0. Inference uses DDIM sampling [98) with 50 steps.
5.2 Eyaluation protocol
We conduct comprehensive evaluation of Diff-ICMH across key dimensions: performance diverse set of downstream intelligent tasks and the perceptual quality of image reconstruction.

---

BPG
VTM-18.2
ELIC
HINI
MS-ILLM
PerCo
DIFFEI
Diff ICMH (Ours)
PSWR on Kodak
PSNR T on CLICZ020
MS-SSIM 1 on Kodak
MS-S5IM on CLIC2020
20
1.00
1.00
30
0.05
20-
44-
22-
0.75
0.65
0.10
5.0
0.01
0.0 6.55
LPIPS Kodak
LPIPS Tecnick
LPIPS CLIC2020
Tecnick
0.30
0.1-
100
0.4
50-
0.2-
60-
0.10
0.3
3.04
0.29
0.55
0.0
8.1 0.34
092
0.54
0.06
5/M 632 0.14
0.52
0.04
5/09
0.55
0.0
6.12
042
004
0.08
038
6.19
032
1eo
100
300
FID 1 on CLIC2020
DISTS on Kodak
DISTS on Tecnick
DISTS on CLIC2020
120
0.32
0.30
0.25
0.25
60-
0.20
60
0.20
20
0.10
Figure 7: Comparison of Diff-ICMH and other methods on signal fidelity-oriented (PSNR1, MS- SSIM?) and perception-orientie (LPIPS), FID4, DISTS() metries.
Intelligent tasks. To evaluate the generalizability and effectiveness of our approach for machine vision, we conducted extensive experiments on a diverse range of intelligent tasks, encompassing variety of task categories, model architectures, and backbone networks. These tasks includes traditional computer vision (e.g., detection and segmentation on COCO 2017 (99]), multimodal retrieval (Flickr30K (100),, and advanced Multimodal Large Language Model (MLLM) based understanding tasks (such as referring expression on RefGTA 101 and open-set segmentation on ADE20K [022 ). Comprehensive details regarding these tasks are available in the Appendix.
Perceptual quality of reconstruction. Three public datasets utilized: Kodak (103), Tecnick [104). and CLIC2020 [105]. We assess performance using metrics that measure signal fidelity (PSNR and MS-SSIM) as well those that correlate with human perceptual quality (LPIPS, FID, and DISTS).
Compared methods for intelligent task support. We conduct comparative experiments with multiple hiigh-errfrmanee codecs, including the powerful traditional image codec VVC (V) 18.2), learned codecs ELIC [106] and FTIC [20]. as well as codecs focusing on perceptual quality optimization, including HiFiC 71 PerCo 73/ MS-ILLM [72], DiffEIC [74], and task-specific optimized codecs TransTIC.
and Adapter-ICMH
43 as comparative methods.
Compared methods for perceptual quality. We conduct comparisons with the traditional codec BPG, VVC (5) (VTM-18.2 2), the learned codec ELIC [106), and perceptual quality optimization codecs including HiFiC 711, PerCo (73 [07]. MS-ILLM 772), and DiffEIC (74). Numerical results
of compared methods are obtained from the DifIBIC repository.
5.3 Results
Multiple intelligent task supporting. Giilusttil Diff-ICMH's performance across diverse intelligent tasks on COCO, Flickr30K, RefGTA, and ADE20K datasets, compared against existing methods. Overall, Diff-ICMH consistently achieves state-of-the (SOTA) or highly competitive results. On COCO, it generally excels in object detection, instance, and panoptic segmentation, outperforming traditional, perception--rreented, and several task-specific codecs. For Flickr30K
htttps://iithub. http:://ggg.tuuococc/aaaaacccaas
8

---

cross-modal retrieval and ADE20K open neen-vocabulary segmentation tasks, Diff-ICMH demonstrates significant advantages, particularly at very low bitrates (e.g.. 0.01-0.05 bpp on Flickr, 0.02-0.1 bpp on ADE20K), underscoring its effective semantic preservation capabilities attributed to our SC loss and TGM. This highlights strong adaptability to both traditional vision and advanced MILLM-based understanding tasks. While generally leading, performance is comparable to some methods like DiffEI specific scenarios, such as COCO pose estimation or RefGTA referring expression, potentially due to inherent VAE latent space limitations for fine details domain shift from synthetic data, respectively. Despite these minor trade-offs,, the extensive evaluations confirm Diff-ICMH's broad generalizability and excellent overall rate-distortion performance across multitude of tasks without requiring any task-specific finettuning emphasizing its practical value.
Perception-oriented reconstruction. Figure 7 presents the compared results of reconstruction quality. Consistent with its generative nature, Diff-ICMH's PSNR/MS-SSS MS-MSMM scores trail behind traditional fidelity-optimized codecs, common characteristic when prioritizing perceptual realism. Besides, is comparable DiffEIC and outperforms PerCo in these fidelity metrics. Conversely, Diff-ICMH demonstrates substantial advantages in perceptual quality. It significantly surpasses fidelity-focused codecs (e.g., BPG, VTM-18.2) and other perception-orient
methods (e.g.. HiFiC, PerCo) across LPIPS, FID, and DISTS. Notably, Diff-ICMH achieves SOTA performance in FID and DISTS scores among all compared methods, exhibiting particular strength at extremely low bitrates. These perceptual gains, especially low BPP, are attributed to the effectiveness of our SC loss and TGM in preserving semantic integrity and guiding realistic reconstruction.
Additional results. For brevity, visualizations, feature difference analysis, and computational
complexity analysis are provided Section B.2 B.3 B.4
5.4 Ablation study
We conducted ablation studies to validate the effectiveness the proposed Semantic Consis- tency loss (SC loss) and Tag Guidance Mod- ule (TGM), along with an analysis of the SC loss setup. Models are trained on the LSDIR dataset and evaluated using object detection on COCO 2017. More details provided in the Appendix.
35.0
32.5
27.5
Effectiveness of SC loss and TGM. Figure 9 presents ablation results for the SC loss and TGM. Both modules significantly improve the task performance, with their combination (blue curve) yielding the best performance (e.g., around 4 mAP gain over the baseline at approx- imately 0.025 Bpp). This confirms their individ- ual contributions and synergistic effect. These results validate that SC loss maintains semantic similarity while TGM activates generative priors, complementarily boosting performance.
22.5
Web Syen wio tag guidance
Crem lag guidence
20.0
w/o Erees w/ tag guidance
Gee w/ tag guidance
0.000
0.025
0.050
0.075 0,100
0.150 0.175
Rate (Bpp)
Figure 9: Ablation of Semantic Consistency loss and Tag Guidance Module.
Setup of SC loss. Figure 8 details ablations for SC loss hyperparameters.
Weight Asem- As shown in Figure g (a), Asem = 2.0 achieves the optimal balance, effectively preserving semantic information while maintaining low bitrate.
Position of SC loss application. Applying SC loss to deeper features within the U-Net yields better rate-distortion performance (Figure 8 (b)). The middle block found be optimal, as
features at this depth better capture abstract semantic content compared to shallower layers.
Input noise level for SC loss. We compared using clean latent features (z, 2) versus noised versions (z,, 21) as inputs to the semantic feature extractor f(-) for the SC loss. Figure (c) demonstrates that noise-free inputs (1 0) result in the optimal rate-distortion performance. This suggests that while the diffusion model is trained with noise, its capability for semantic feature extraction (for our SC loss) maximized with clean signals. Notably, all tested noise levels still improved performance over the bascline, confirming the robustness of the SC loss.

---

30
30
28
28
26
26
Baseline
+
Baseline
24
Baseline
22
4,0
Layer
22
2.0
Encoder Layer
t-U(0,02)
20
Asam11
20
Layer
20
L~U(0,04;
18
0.5
18
Middle Block
18
t- U(0,00066
0.02
0.03
0.04
0.05
0.06
0.02
0.03
0.04
0.05
0.06
0.02
0.03
0.04
0.05
0.06
Rate (Bpp)
Rata (Bpp)
Rate (Bpp)
(a) Weight Arm of SC loss.
(b) Position SC loss application.
(c) Noise level SC loss inputs.
Figure 8: Ablation of paraeetrss of SC loss. "Baseline" indicates not including SC loss.
Limitation
The primary limitation of Diff-ICMH is computational complexity, discussed in Section B.4 Diff-ICMH requires relatively significant decoding time due to the iterative nature of the diffusion denoising process, where each step necessitates complete forward pass through the neural network. However, we emphasize that the primary focus of this paper is to demonstrate the generalizability and significant potential of Diff-ICMH for harmonizing diverse intelligent tasks with human perceptual quality. Therefore, computational complexity analysis is not the central emphasis of our current work. In future work, incorporating efficient sampling methods [TO8 and distillation techniques [1T]- [33 to reduce sampling steps would make Dift-ICMH more practical for real-world applications.
7 Conclusion
In this paper, we introduced Diff-ICMH, a verstile generative image codec designed to effectively serve both intelligent tasks and human visual perception. The design philosophy of Diff-ICMI departs from isolated optimization by revealing and leveraging fundamental commonalities between these objectives: Preserving semantic information integrity is paramount for both machine analysis and human understanding. Concurrently, enhanced perceptual quality, which is built upon this semantic foundation, not only improves the visual experience but also benefits machine feature extraction. Diff-ICMH embodies this by integrating a diffusion model-based generative framework with a novel Semantic Consistency loss and efficient Tag Guidance Module. Comprehensive evaluations demonstrate the effectiveness and generalization of our proposed method, which efficiently supports diverse tasks with single codec and aasc-specific adaptation, alongside excellent visual quality. Our work thus presents promising pathway towards truly versatile image compression.
Acknowledgements This work was supported in part by NSFC under Grant 62371434, 623B2098, and 6202 1001, the Postdoctoral Fellowship Program CPSF under Grant Number GZC20252293, and the China Postdoctoral Science Foundation-Anin Joint Support Program under Grant Number 2024T017AH. This work was also funded by Anhui Postdoctoral Scientific Research Program Foundation (0.2202A1115)
10

---

References
[1] G. K. Wallace, "The jpeg still picture compression standard," IEEE Transactions on Consumer Electronics, vol. 38, no. I. pp. xviii-Xxxix 1992.
[2] C. Christopoulos, A. Skodras, and T. Ebrahimi, "The jpeg2000 still image coding system: overview," TEEE Transactions Consumer Electronics, vol. 46, no. pp. 1103-1127, 2000.
[3] T. Wiegand, G. J. Sullivan, G. Bjontegaard, and A. Luthra, "Overview of the h. 264/avc video coding standard," IEEE Transactions Circuits and Systems for Video Technology, vol. 13, no. 7, pp. 460-576, 2003.
[4] G. J. Sullivan, J.-R. Ohm, W.-J. Han, and T. Wiegand, "Overview of the high efficiency video coding (heve) standard," JEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1649-1666 2012.
[5] B. Bross, Y.-K. Wang. Y. Ye, S. Liu, J. Chen, G. J. Sullivan, and J.-R. Ohm, "Overview of the versatile video coding (vve) standard its applications." IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 10, pp. 3736-3764, 2021.
[6] Ballé, Laparra, and Simoncelli, "End-to-end optimized image compression," in International Conference on Learning Representations, 2017.
[7] J. Ballé, Minnen, S. Singh, S. J. Hwang, and N. Johnston, "Variational image compression with scale hyperprior," in International Conference on Learning Representations, 2018.
[8] D. Minnen, J. Ballé, and G. D. Toderici, "Joint autoregressive and hierarchical priors for learned image compression." " Advances Neural Information Processing Systems, vol. 31, 2018.
[9] Z. Guo, Z. Zhang. R. Feng, and Z. Chen, "Causal contextual prediction for learned image compression." IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 4, pp. 2329 2341, 2021.
[10] D. He, Y. Zheng, B. Sun, Y. Wang, and H. Qin, "Checkerboard context model for efficient learned image compression," " IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14771-14 780.
[11] D. Minnen and S. Singh, "Channel-wis autoregressive entropy models for learned image compression." IEEE International Conference on Image Processing. IEEE, 2020, pp. 3339-3343.
[12] Z. Cheng. H. Sun, M. Takeuchi, and J. Katto, "Learned image compression with discretized gaussian mixture likelihoods and attention modules," IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7939-7948.
[13] Y. Xie, K. L. Cheng, and Q. Chen, "Enhanced invertible encoding for learned image compression." in ACM International Conference on Multimedia, 2021, pp. 162-170.
[14] J. Liu, H. Sun, and Katto, "Learned image compression with mixed transformer-enn architectures." in IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023, pp. 14 488811433 397.
[15] Y. Zhu, Y. Yang, and T. Cohen, "Transformer-base based transform coding." in International Conference Learning Representations, 2022.
[16] R. Zou, C. Song, and Z. Zhang, "The devil is in the details; Window-baset attention for image compres-
sion," in IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022, pp. 17 492-17
[17] Y. Qian, X. Sun, M. Lin. Z. Tan, and R. Jin, "Entroformer: A transformer-based entropy model for learned image compression." in International Conference Learning Representations, 2022.
[18] A. B. Koyuncu, H. Gao, A. Boev, G. Gaikov, E. Alshina, and E. Steinbach. "Contextformer: A trans- former with apatio-channe, attention for context modeling in learned image compression." in European Conference Computer Vision. Springer, 2022, pp. 447-463.
[19] J. Kim, B. Heo, and J.-S. Lee, "Joint global and local hierarchical priors for learned image compression," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5992-6001.
[20] H. Li, S. Li, W. Dai, C. Li, J. Zou, and H. Xiong, "Frequency-awart transformer for learned image compression." in International Conference on Learning Representations, 2024.
[21] F. Zeng, H. Tang, Y. Shao, S. Chen, L. Shao, and Y. Wang. "Mambaic: State space models for high- performance learned image compression," arXiv preprint arXiv:2503.12461 2025.
11

---

[22] S. Qin, J. Wang. Y. Zhou, B. Chen, T. Luo, B. An. T. Dai, S. Xia, and Y. Wang. "Mambave: Learned visual compression with selective state spaces," arXiv preprint arXiv:2405.154: 2024.
[23] S. Iwai, T. Miyazaki, and S. Omachi, "Controlling rate, distortion, and realism: Towards single comprehensive neural image compression model," in Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, 2024, pp. 2900-2909.
[24] Z. Liu, T. Liu, W. Wen, L. Jiang, J. Xu, Y. Wang, and G. Quan, "Deepn-jpeg: A deep neural network favorable jpeg-based image compression framework," in Annual Design Automation Conference, 2018, pp. 1-6.
[25] D. Chamain, S.-c, S. Cheung, and Z. Ding, "Quannet: Joint image compression and classification over channels with limited bandwidth," in IEEE International Conference on Multimedia and Expo. IEEE, 2019, pp. 338-343.
[26] Z. D. Li, "Optimizing jpeg quantization for classification networks." Resource-Construinge Machine Learning, 2020.
[27] J. Choi and B. Han, "Task-aware quantization network for jpeg image compression," in European Conference on Computer Vision. Springer, 2020, pp. 309-324.
[28] L.-Y. Duan, X. Liu, Chen, T. Huang, and W. Gao, "Optimizing jneg quantization table for low bit rate mobile visual search," in Visual Commanccaions and Image Processing. IEEE, 2012, pp. 1-6.
[29] X. Luo, H. Talebi, F. Yang, M. Elad, and P. Milanfar, "The rate- pall-distortion-accraaa tradcoff: Jpeg case study," 2020. [Online], Available: https://arxiw.rrg/bb///// 00666
[30] Q. Cai. 7. Chen, D. O. Wu, S. Liu, and X. Li, "A novel video coding strategy in heve for ohject detection," IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 12. pp. 4924 4937, 2021.
[31] Huang, Jia, S. Wang, and S. Ma, "Visual analysis motivated rate-distortion model for image coding," " in IEEE International Conference on Multimedia and Expo. IEEE, 2021, pp. 1-6.
[32] X. Li, J. Shi, and Z. Chen, "Task-driven semantic coding via reinforcement learning," IEEE Transactions Image Processing, vol. 30. pp. 6307-63073 6320, 2021.
[33] G. Xie, Li. S. Lin, Z. Chen, L. Zhang, K. Zhang, and Y. Li, "Hierarchical reinforcement learning based video semantic coding for segmentation," IEEE International Conference on Visual Communicutions and Image Processing. IEEE, 2022, pp. 1-5.
[34] S. Suzuki, M. Takagi, K. Hayase, T. Onishi, and A. Shimizu, "Image pre-transformation for recognition- aware image compression," in IEEE International Conference on Image Processing. IEEF 2019, pp. 26600
[35] G. Lu, X. Ge, T. Zhong, Hu, and Geng, "Preprocessing enhanced image compression for machine vision," IEEE Transactions on Circuits and Systems for Video Technology. 2024.
[36] H. Zhang, J. I. Ahonen, N. Le, R. Yang. and F. Cricri, "Competitive learning for achieving content-specific filters in vidco coding for machines," in JEEE International Conference on Image Processing. IEEE, 2024, pp. 1877-1882.
[37] M. Yang. F. Yang. L. Murn, M. G. Blanch, J. Sock, S. Wan, F. Yang. and L. Herranz, switchble pre-processor for image compression for multiple machine vision tasks," IEFE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 7, pp. 6416-6429, 2024.
[38] S. Luo, Y. Yang, Y. Yin, C. Shen, Y. Zhao, and M. Song. "Deepsic: Deep semantic image compression," " in International Conference on Neural Information Processing, 2018, pp. 106.
[39] N. Patwa, N. Ahuja, S. Somayazulu, O. Tickoo, S. Varadarajan, and S. Koolagudi. "Semmaniic-reserviiii image compression." in IEEE International Conference Image Processing. IEEE, 2020, pp. 1281- 1285.
[40] K. Fischer, F. Brand, and A. Kaup, "Boosting neural image compression for machines using latent space masking," 2021. [Online]. Available:
[41] L. D. Chamain, F. Racapé, J. Bégaint, A. Pushparaja, and S. Feltman, "End-to to- optimized image compression for machines, a study," in Data Compression Conference. IEEE, 2021, pp. 163-172
12

---

[42] N. Le. H. Zhang. F. Cricri. R. Ghaznavi- Youvalari, and E. Rahtu, "Image coding for machines: end-to- end learned approach," in IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2021, pp. 1590-1594.
[43] H. Li, S. Li, S. Ding, W. Dai, M. Cao, C. Li, J. Zou, and H. Xiong, "Image compression for machine and human vision with pretuay-requencyy adaptation," " Europan Conference on Computer Vision. Springer, 2024, pp. 382-399.
[44] Y.HH. Chen, Y.-C. Weng, C.-H. Kao, C. Chien, W.-. Chiu, and W. W. Peng, "Transtic: Transferring transformer-bas based image compression from human perception to machine perception." in IEEE/CVF International Conference Computer Vision, 2023, pp. 23 297-23 307.
[45] J. Liu, X. Jin, R. Feng. 7. Chen, and W. Zeng, "Composable image coding for machine via task-oriented internal adaptor and external prior," in IEEE International Conference on Visual Communications and Image Processing. JEEE, 2023. pp. 1-5.
[46] H. Choi and I. Bajic, "Deep feature compression for collaborative object detection," " IEEE Interna- tional Conference on Image Processing. TEFE, 2018, pp. 3743-3747.
[47] Chen, K. Fan, S. Wang, Duan, W. Lin, and A. C. Kot, "Toward intelligent sensing: Intermediate deep feature compression." IEEE Transactions on Image Processing, vol. 29, pp. 2230-2243, 2019.
[48] A. E. Eshratifar, A. Esmaili, and M. Pedram, "Bottlenet: A deep learning architecture for intelligent mobile cloud computing services," in IEEF/ACM International Symposium on Low Power Electronics and Design. IEEF, 2019, pp. 1-6.
[49] S. Suzuki, M. Takagi, S. Takeda, R. Tanida, and H. Kimala, "Deep feature compression with spatio- temporal arranging for collaborative intelligence," in IEEE International Conference on Image Processing. IEEE, 2020, pp. 3099-3102
[50] Suzuki, S. Takeda, M. Takagi, R. Tanida, H. Kimata, and H. Shouno, "DDepp feature compression using spatio-temporal arrangement toward collaborative intelligent world," IEFE Transactions on Circuits and Systems for Video Technology. vol. 32, no. 6, pp. 3934-3946, 2021.
[51] J. Shao and J. Zhang. "Bottlenet++:: An end-to-end approach for feature compression in device-edge co- co-inference systems," in IEEE International Conference on Communications Workshops. IEEE, 2020,
pp. 1-6.
[52] S. Singh, S. Abu-EI-Haija, N. Johnston, J. Ballé, A. Shrivastava, and G. Toderici, "End-to-end learning of compressible features," in IEEE International Conference Image Processing. IEEE, 2020, pp. 3349-3353.
[53] S. R. Alvar and I. V. Bajic, "Bit allocation for multi-task collaborative intelligence," in IEEE International Conference on Acoustics, Speech and Signal Processing. IERE, 2020, pp. 4342-4346.
[54] "Pareto-optimal bit allocation for collaborative intelligence," IEEE Transactions on Image Prucess- ing. vol. 30, pp. 3348-3361, 2021.
[55] Henzel, Misra, and Ji, "Efficient feature compression for the object tracking task," in IEFE International Conference on Image Processing. IEFF, 2022, pp. 3505-3509.
[56] H. Tu, L. Li, W. Zhou, and H. Li, "Reconstruction-fees image compression for machine vision via knowledge transfer." ACM Transactions on Multimedia Computing, Communications and Applications, vol. 20, 10. pp. 19, 2024.
[57] N. Ahuja, P. Datta, B. Kanzariya, V. S. Somayazulu, and O. Tickoo, "Neural rate estimator and unsuper- vised learning for efficient distributed image analyties in split-dnn models," in IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2023, pp. 2022-2030.
[58] Kim, Jeong. J. Yu, Kim, J. Lee, S. Y. Jeong, and Y. Kim, "End-to-ent learnable multi scate feature compression for vem," IEEE Transactions on Circuits and Systems for Video Technology. 2023.
[59] D. Nazir, Bartels, J. Piewek, T. Bagdonat, and T. Fingscheidt, "Distributed semantic segmentation with efficient joint source and task decoding." in European Conference on Computer Vision. Springer, 2024, pp. 195-212.
[60] L. Xiong, X. Luo, Z. Wang, C. He. S. Zhu, and B. Zeng, cexxture-giide coding for deep features," 2024. [Online]. Available:
19669
13

---

[61] S. Guo, Z. Chen, Y. Zhao, N. Zhang. X. Li. and L. Duan, "Toward scalable image feature compression: content-adaptive and diffusion-based approach." in ACM International Conference Multimedia, 2023, pp. 1431-1442.
[62] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, "High-resolution image synthesis with latent diffusion models," in IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022,
pp. 10 684-10695.
[63] Y. Blau and T. Michaeli, "Rethinking lossy compression: The rate rate-distortion percettionn tradeoff," in International Conference on Machine Learning. PMLR, 2019. pp. 675-685.
[64] Z. Yan, F. Wen, R. Ying. C. Ma, and P. Liu, "On perceptual lossy compression: The cost of perceptual reconstruction and an optimal training framework," " in International Conference on Machine Learning. PMLR, 2021, pp. 11 682 11 692.
[65] G. Zhang, J. Qian, J. Chen, and A. Khisti, "Universal atte-ddstortion-peecppionn representations for lossy compression," " Advances Neural Information Processing Systems, vol. 34, pp. 11 517-11 529, 2021.
[66] I. Goodfellow, J. Pouget-Abadice M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," Advances in Neural Information Processing Systems, vol. 27, 2014.
[67] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. Van Gool, "Extreme learned image compression with gans," in IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 2587-2590.
[68] E. Agustsson, M. Tschannen, F. Mentzer, R. Timofte, and L. V. Gool, "Generative adversarial networks for extreme learned image compression." in IEEE/CVF International Conference on Computer Vision, 2019, pp. 221-231.
[69] S. Iwai, T. Miyazaki, Y. Sugaya, and S. Omachi, generative adversarial networks." in International Conference Pattern Recognition. IEEE, 2021, pp. 2242-
extreme image compression with
[70] L. Wu, K. Huang, and H. Shen, "A gan-based tunable image compression system," in IEEF/CVF Winter Conference Applications Computer Vision, 2020. pp.
[71] F. Mentzer, G. D. Toderici, M. Tschannen, and E. Agustsson, "High-fidelity generative image compres- sion," Advances in Neural Information Processing Systems, vol. 33, pp. 11 913 11 924, 2020.
[72] M. J. Muckley, A. El-Nouby, K. Ullrich, H. Jégou, and J. Verbeek, "Improving statistical fidelity for neural image compression with implicit local likelihood models," in International Conference Machine Learning. PMLR, 2023, pp. 25 424426-22 1066-25440 443.
[73] M. Careil, M. J. Muckley, J. Verbeek, and S. Lathuilière, "Towards image compression with perfect realism ultra-low bitrates," in International Conference on Learning Representations, 2023.
[74] Z. Li, Y. Zhou, H. Wei, C. Ge, and J. Jiang. "Towards extreme image compression with latent feature guidance and diffusion prior." IEEE Transactions on Circuits and Systems for Video Technology. 2024.
[75] Yang and Mandt, "Lossy image compression with conditional diffusion models," Advances in Neural Information Processing Systems, vol. 36, pp. 64971-64 995, 2023.
[76] Relic, Azevedo, M. Gross, and C. Schroers, "Lossy image compression with foundation diffusion models," in European Conference on Computer Vision Springer, 2024, pp. 303-319.
[77] Theis, T. Salimans, M. D. Hoffman, and Mentzer, "Lossy compression with gaussian diffusion," 2022. [Online]. Available: httpss//axivvorr//////26......
[78] Z. Pan, X. Zhou, and H. Tian, "Extreme generative image compression by learning text embedding from diffusion models," 2022. [Online]. Available: https//amiv.on//a//2211.111
[79] T. Xu, Z. Zhu, D. Нe. Li, L. Guo, Y. Wang. Z. Wang, II Qin, Y. Wang, J. Liu et al., "Idempotence and perceptual image compression." in International Conference Learning Representations, 2024.
[80] E. Hoogehoom, E. Agustsson, F. Mentzer, L. Versari, G. Toderici, and L. Theis, "High- fidelity image compression with score-based generative models," 2023. [Online]. Available: https:/annxivooyaab/22555..2.
[81] N. F. Ghouse, J. Petersen, A. Wiggers. T. Xu, and G. Sautiere, "A residual diffiusion model for high perceptual quality codec augmentation." 2023. [Online]. Available: https://axivorg////223.....
14

---

[82] Y. Ma, W. Yang. and J. Liu. "Correcting diffusion- based perceptual image compression with privileged end-to-end decoder," in International Conference on Machine Learning. PMLR, 2024, pp. 34 075- 34 093.
[83] H. Kuang, Y. Ma. W. Yang, Z. Guo, and J. Liu, "Consistency guided diffusion model with neural syntax for perceptual image compression." in ACM International Conference on Multimedia, 2024, pp. 1622-1631.
[84] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition." in IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770-778.
[85] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. "The unreasonable effectiveness of deep features as a pereeptual metric," in IEEE/CVF Conference Computer Vision and Pottern Recognition, 2018, pp. 586-595.
[86] Zhang, A. Rao, and M. Agrawala, "Adding conditional control to text-to-image diffusion models," in I.FF//CV International Conference on Computer Vision, 2023, pp. 3836-3847.
[87] Mukhopadhyay, M. Gwilliam, Agarwal, N. Padmanabhan, A. Swaminathan, Hegde, T. Zhou, and A. Shrivastava, "Diffusion models beat gans on image classification," " 2023. [Online]. Available: http://aaxi.orr///b/22277..00000
[88] J. Zhang, C. Herrmann, J. Hur, L. Polania Cabrera, V. Jampani, D. Sun, and M.-H. Yang. "A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence," Advances in Neural Information Processing Systems, vol. 36, pp. 45 533-45 547, 2023.
[89] S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie, "Representation alignment for generation: Training diffusion transformers is easier than you think," 2024. [Online]. Available: http:///arivoor///b///4100000440
[90] A. C. Li, M. Prabhudesai, S. Duggal, E. Brown, and D. Pathak, "Your diffusion model is secretly aero-shottan classifier," in IEEE/CVF International Conference on Computer Vision, 2023, pp. 2206 2217.
[91] W. Xiang. H. Yang. D. Huang, and Y. Wang, "Denoising diffusion autoencoders are unified supe-supe learners." in IEEE/CVF International Conference on Computer Vision, 2023, pp. 15 802-15 812.
[92] X. Chen, Z. Liu, S. Xie, and K. He. "Deconstructing denoising diffusion models for self sulf-super learning," " 2024. [Online]. Available: jutpss//aaxxvvor///b/////// 14404
[93] Y. Zhang, X. Huang, J. Ma, Z. Li, Z. Luo, Y. Xie, Y. Qin, T. Luo, L. S. Liu et al., "Recognize anything: A strong image tagging model," in JEEE/CVF Conference Computer Vision and Pattern Recognition, 2024, pp. 1724 1732.
[94] J. Ho and T. Salimans, "Classifier-free diffusion guidance," in Advances in Neural Information Processing Systems Workshop on Deep Generative Models and Downstream Applications, 2021.
[95] Y. Li, K. Zhang, J. Liang. J. Cao. C. Liu, R. Gong. Y. Zhang. H. Tang, Y. Liu, D. Demandolx et al., "Lsdir: A large scale dataset for image restoration," in IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 1775-1787
[96] J. Ho, A. Jain, and P. Abbeel, "Denoising diffusion probabilistic models," Advances Neural Information Processing Systems, vol. 33, pp. 6840-6851, 2020.
[97] D. P. Kingma and J. Ba, "Adam: method for stochastic optimization," 2014. [Online]. Available: httpa/iww..or/bb///1/..6666
[98] J. Song, Meng, and S. Ermon, "Denoising diffusion implicit models," in International Conference on Learning Representations, 2021.
[99] T.-Y. Lin, M. Maire, S. Belongie, Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft coco: Common objects in context," European Conference on Computer Vision. Springer, 2014, pp. 740-755.
[100] A. B. Plummer, Wang. M. C. Cervantes, J. Caicedo, J. Hockenmaier, and Lazebnik, "Flickr30k entities: Collecting region-to-phrase correspondences for richer mnage-to-sentence models," International Journal of Computer Vision. pp. 74-93, 2017.
[101] M. Tanaka, T. Itamochi, K. Narioka, I. Sato, Y. Ushiku, and T. Harada, "Generating easy- to-deer-nddd referring expressions for target identifications." in /EEE/CVF International Conference on Computer Vision, 2019, pp.
15

---

[102] B. Zhou, H. Zhao, X. Puig, S. Fidler, A. Barriuso, and A. Torralba, "Scene parsing through ade dataset," in Proceedings the IEEE conference on computer vision and pattern recognition, 2017, pp. 633-641.
[103] E. Kodak, "Kodak lossless true color image suite (photocd pcd0992), http://rOOk.us/graphicsskoo 1993.
[104] N. Asuni, A. Giachetti et al., "Testimages: a large-scale archive for testing visual devices and basic image processing algorithms." " in STAG, 2014, pp. 63-70.
[105] G. Toderici, I.. Theis, N. Johnston, F. Agustsson, F. Mentzer, J. Ballé, W. Shi, and R. Timofte, "Clic 2020: Challenge learned image compression." Retrieved March, vol. 29, p. 2021, 2020.
[106] D. He, Z. Yang. W. Peng, R. Ma, H. Qin, and Y. Wang, "Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding," in IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2022, pp. 5718-5727.
[107] N. Körber, E. Kromer, A. Siebert, S. Hauke, D. Open perceptual compression," Workshop on Machine Learning and Compression, NeurIPS 2024, 2024. [Online]. Available: https://openreiiww.eeeleee wwwpevvviwiwwe//ooummmmm Sudlrrrgdundddd
and B. Schuller, "Perco (SD):
[108] Lu, Zhou, F. Bao. Chen, Li, Zhu, "Dpm-solver: A fast ode solver for difftusion probabilistic model sampling in around 10 steps," Advances in Neural Information Processing Systems, vol. 35, pp. 5775-5787. 2022.
[109] "Dpm-solvert+: Fast solver for guided sampling of diffusion probabilistic models," Machine Intelligence Research, 1-22, 2025.
[110] K. Zheng, C. Lu, J. Chen, and J. Zhu, "Dpm-solver-v': Improved diffusion ode solver with empirical model statistics," Advances in Neural Information Processing Syxtems, vol. 36, pp. 55 502-55 542, 2023.
[111] T. Salimans and J. Ho, "Progressive distillation for fast sampling of diffusion models," in International Conference on Learning Representations, 2023.
[112] A. Sauer, D. Lorenz, A. Blattmann, and R. Rombach, "Adversarial diffusion distillation," in European Conference on Computer Vision. Springer, 2024, pp. 87-103.
[113] A. Sauer, F. Boesel, T. Dackhorn, A. Blattmann, P. Esser, and R. Rombach, "Fast high-resolution image synthesis with latent adversarial diffusion distillation," in SIGGRAPH Asia 2024 Conference Papers, 2024, pp. 1-11.
[114] T-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection." in JEEE Conference Computer Vision and Pattern Recognition, 2017, pp. 2117-2125.
[115] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som et al., "Image foreign language: Beit pretraining for vision and vision-language tasks," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 19 175-19 186.
[116] H. Bao, W. Wang, L. Dong, Q. Liu, O. K. Mohammed, K. Aggarwal, S. Som. S. Piao, and F. Wei. "Vlmo: Unified languae pre-training with
modd-momtt--eeperrsstt Advances in Neural Information Processing Systems, vol. 35, pp. 32 897-32 2022.
[117] Bai, K. Chen, X. Liu, J. Wang. W. Ge, S. Song, K. Dang, Wang. S. Wang. Tang et al., "Qwen2.5-vl technical report," 2025. [Online]. Available: https//aaxii.og//bb//50....
[118] Yuan, W. Li, J. Liu, D. Tang. X. Luo. C. Qin, L. Zhang, J. Zhu, "Osprey: Pixel understanding with visual instruction tuning." in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28 202-2822222 211.
[119] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, "A convnet for the 2020s," in IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 11 976--1 986.
16

---

Appendix
Contents
Implementation Details
17
A. Architecture of the control module
17
A.2
Implementation in ablation studies
17
A.3 Hardware device
18
B
Experiments
18
B.1
Details of evaluation protocol
18
Visualization results
19
B.3
Feature difference analysis
19
B.4
Complexity analysis
B,5
Ablation study of distortion loss calculation space
A
Implementation Details
A.1
Architecture of the control module
The control module described in Section 4.1 is modified from original ControlNet (86). Key modifi- cations include: (i) Replacing ControlNet's s initial three stride-2 convolutions with a single stride-1 convolution to establish dimensional compatibility with 2. (ii) Implementing bilateral feature in- jection through cero-convolutions from both encoder and decoder pathways of the diffusion model (unlike ControlNet's decoder-oni approach) to establish Additionally, the extracted word-level tags с are strategically injected as text prompts into both the control module and the diffusion model effectively activate relevant generative priors.
precise control mechanisms from 2.
A.2
Implementation in ablation studies
Basic setup. As described in Section 5.4, we designed ablation experiments to verify the effective- ness of the SC loss and tag guidance module, along with the hyper-parameter configuration of SC loss. the experimental setup, training was conducted on the LSDIR dataset with batch size of 8, total of 80, 000 iterations, and learning rate of 1e - 5. The experimental results were rigorously evaluated on the COCO 2017 object detection benchmark dataset.
Semantic Consisitency loss with noisy input. Section 4.2, for SC loss calculation, clean latent features and â utilized as inputs to the diffusion model. Recognizing that diffusion models are typically trained on noisy signals, we conducted a series of comparative experiments (illustrated in Figure 1Op to investigate the implications of using noisy latent features z, and é, as inputs, corresponding to results Section 5.4, "Input noise level for SC loss". Specifically, this variant, the same sampled noise first added the latent variable and the reconstructed latent variable & using the same timestep sampled from U (O, mmax) where bmax is a predefined threshold used control the maximum noise intensity. A value of = corresponds to the complete 1000-step noising process. These are then separately fed into the trained diffusion model for feature mapping to obtain the corresponding semantic features, which are subsequently aligned.
Bits of Tag Guidance. We employ RAM++ from Recognize Anything 933, which a maximum default vocabulary of 4585 tags. For simplicity, we utilize fixed-length encoding without entropy coding, requiring 13 bits per tag to accommodate maximum of 8192 tags. Based on our analysis of 500 randomly sampled COCO images, RAM++ predicts average of 8.7 tags per image under its
17

---

f(a))
-a,z+ ((1 8 + 2,
Diffusion Model
E (L, fm G .VO..)
Semontic Consistency
2 + (1 6 +
Diffusion Model
f(2))
Figure 10:
Illustration of the variant of SC loss calculation. Noisy latent variable 2, and 2, are utilized as input into the pre-trained diffusion model.
default settings. Consequently, the average bit overhead for tag guidance amounts to 13 x 8.7 = 113.1 bits per image.
A.3
Hardware device
All training and inference experiments are conducted on 4 NVIDIA A100 Tensor Core GPUs.
B
Experiments
B.1 Details of evaluation protocol
We conduct comprehensive evaluation from two dimensions: the performance various intelligent tasks and the perceptual quality of image reconstruction.
Intelligent tasks. As mentioned in Section 5.2, the efficacy and generalization capabilities of our proposed method are rigorously evaluated across a diverse spectrum of intelligent tasks spanning different domains, datasets, task-specific models, and vision backbones. To the best of our knowledge, this is the first comprehensive study to present such extensive experimental validation in the context of machine-oriented image coding. We believe this thorough empirical analysis not only substantiates our contributions but also establishes new benchmarks that may accelerate advancement in this emerging field.
The specific information about datasets and task models is shown in Table I where datasets include COCO 2017 199], Flickr30K RefGTA [1OI], and ADE20K The intelligent tasks cover three major categories: traditional perception-
computer vision tasks, multimodal retrieval tasks, and multimodal understanding tasks based large language models. For task models, traditional perception tasks are evaluated through series of models the Detectron2 toolkit; multimodal retrieval tasks evaluated through the BEiT-3 model; and for multimodal understanding capabilities, we specifically selected two multimodal large language models (MLLMs) based on different backbone networks, processing different data domains, and executing tasks of varying granularities.
This comprehensive experimental framework aims to thoroughly verify the generalizability and superior performance of the Diff-ICMH approach across different data types, datasets, intelligent tasks, and task models. For evaluation metrics, we employ standard measures appropriate to each task: mAP (mean Average Precision) of bounding boxes for object detection, mAP of segmentation masks for instance segmentation, AP of keypoints for pose estimation, PQ (Panoptic Quality) for panoptic segmentation, Recall @ I for multimodal retrieval tasks, and accuracy for referring expression comprehension. The open-set pixel-domain understanding tasks based on MILMS are evaluated using mIoU for semantic segmentation, mAp for instance segmentation, and PQ for panoptic segmentation.
bttpe://hockemmaierr wiwiinniii......mmi.i.......ii.t hetihnooie.ii//DDoontttttodggg
https://ade20k.csa cswp://dddeeokcckl...aaaaa mit. edu/
18

---

Table I: Overview of experimental datasets, tasks, and models.
Dataset
Task Type
Specifie Task
Task Model
Backbone
Object Detection
Faster R-CNN
Instance Segmentation
Mask R-CNN
COCO 2017
Traditional Perception
R50-FPN IE4
Pose Estimation
Keypoint R-CNN Panoptic-FPN
Panoptic Segmentation
Image- Retrieval
Flickr30K
Multi-Modal Retrieval
BETT- 115
MW-Transformer 116
Text-Image Retrieval
RefGTA ADE20K
Referring Expression
Qwen2.5-VL 117
WA-ViT T17
MLLM Understanding
Open-Set Segmentation
Ospray 118
ConvNext 119
Image
VTM-18.2
ELIC
Diff-ICMH (Ours)
0.0388bpp
0.0315bpp
0.0316bpp
Figure 11: Visualized results instance segmentation.
This paper uses three public datasets: Kodak 103 Tecnick 104], and CLIC2020 [105]. During the experiments, images from Tecnick and CLIC2020 datasets are rescaled to 768 pixels the short side, and samples of size 768 768 are cropped from the center of the images for evaluation, following previous method [74]. Evaluation metries include PSNR and
Image reconstruction.
MS-SSIM for measuring signal fidelity, as well LPIPS, FID, and DISTS for evaluating perceptual quality.
B.2 Visualization results
Task supporting. Figure II presents visualization results comparing our method against VTM- 18.2 (5). ELIC (106), and FTIC 20 instance segmentation tasks. The comparison reveals that VTM-18.2, despite operating at its highest bit rate, fails to correctly identify the two distinct objects within the image. Similarly. ELIC erroneously classifies the "bird" object on the right "person," demonstrating significant recognition inaccuracy. In contrast, only the reconstruction output generated by our Diff-ICMI method successfully preserves the critical semantic information necessary for accurate object identification, thereby effectively supporting the completion of this intelligent task with precision.
Perception-orienteed reconstruction. Figure shows the visual comparison reconstruction quality between Diff-ICMH and other compression methods. From the comparison, it can be observed that methods optimized for signal fidelity (VTM-18.2, ELIC, FTIC) produce reconstructed results with obvious blurring, significantly reducing visual quality. Although MS-ILLM shows some improvement in texture clarity, its reconstructed textures lack authenticity and are accompanied by obvious noise interference. In contrast, our proposed Diff-ICMH demonstrates the best visual realism reconstruction results while maintaining clear boundary contour features.
B.3
Feature difference analysis
To further evaluate Diff-ICMH's performance advantages in semantic information protection and intelligent task support, we designed comparative experiments based on feature difference analysis. Specifically, we calculated the differences between feature representations of reconstructed images from various compression methods and those of original images at different layers of ResNet50 feature extractor (using minus cosine similarity the metric, where lower values represent higher feature similarity), and plotted the curves of these differences across network depths.
19

---

Original
VTM-18.2
ELIC
FTIC
MS-ILLM
Diff-ICMH (Ours)
bop LPIPS 1 1
0.0441/ / 0.465 0.246
0.0279 0 0.282
0.0248 / 0.480 00.290
0.0385 / 0.147 0.303
0.0347 0.224 0.101
bpp LPIPS 1 I
0.0600 / 0.1.1/1/.222 00.201
0.0596/0.ur /0.45/ 0.49//00.///2215 0.221
0.0656 / 0.407 0.207
0.0775 / 0.093 0.080
0.0655 / 0.154 0.086
Figure 12: Reconstruction of Diff-ICMH and other compression methods.
0.5
VTM-18.2
ELIC
Average Feature Difference
MS-ILLM
DIffEIC
(1-Cosine Similarity)
Diff-ICMH (Ours)
0.3
0.2
conv1
layer1.0 0
layer1.1
layer1.2
layer2.0 0
layer2.1
layer2.2
layer2.3
layer3.0 0
layer3.1 1
layer3.2 2
layer3.3
layer3.4 4
layer3.5 5
layer4
layer4.1 1
layerd.2
Network Depth
Figure 13: The evolution curves showing feature differences between reconstructed images from Diff-ICMH and other compression methods compared original images across various layers of the ResNet50 feature extractor. Lower values represent higher feature similarity.
The experiments are conducted the Kodak dataset, analyzing the average results of all images. As shown in Figure [3, the experimental results exhibit significant hierarchical characteristics: in shallow network layers, generative compression methods (MS-ILLM, DiffEIC, Diff- Diff-ICMH) show relatively higher difference values, while methods optimized for signal fidelity (VTM- 18.2, ELIC) maintain minimal differences. However, the network hierarchy and semantic level deepen, this pattern changes significantly- the latter methods' difference values increase rapidly, reaching a peak of approximately 0.5 the deepest layer, far exceeding generative compression methods.
Among generative compression methods, Diff-ICMH's advantage demonstrated its performance from the middle network layers (layer3.0) the deepest layers (layer4.2), consistently maintaining the
20

---

Table 2: Encoding and decoding time on Kodak dataset. (Second)
Method
NFE
Encoding Time
Decoding Time
Hardware
VVC
13.862
0.066
13th Core 19-13900K
ELIC
0.056
0.081
RTX4090
HiFiC
0.038
0.059
RTX4090
MS-ILLM
0.038
0.059
RTX4090
PerCo
0.080
0.665
A100
PerCo
20
0.080
2.551
A100
DIffEIC
20
0.128
1.964
RTX4090
DIMEIC
50
0.128
4.574
RTX4090
Diff-ICMH
20
0.232
5.456
A100
Diff-ICMH
50
0.232
13.14
A100
lowest feature difference values in this range. This result confirms the superiority of our approach in semantic information protection. More importantly, considering that the feature extractor (ResNet50) used in the experiment and the feature mapping (Stable Diffusion) used in the SC loss belong to different network architectures and different pre-training purpose, this performance advantage also highlights the excellent model generalization capability of our method.
B.4 Complexity analysis
Table illustrates the encoding and decoding time of different methods. The data reveals that diffusion-basee methods (such as DifEEIC and Diff-ICMH) perform well in terms of encoding speed, showing significant advantages compared traditional VVC methods (13.862 seconds), with DiffEIC taking 028 seconds and Diff-ICMH taking 0.232 seconds.
However, there is room for improvement in our method's decoding time. Particularly, when the number of denoising steps increases, the decoding time increases significantly. For instance, Diff- ICMH requires 5.456 seconds and 13.14 seconds for decoding 20 steps and 50 steps, respectively. The decoding time is considerably longer than traditional methods like VVC (0.066 seconds) and GAN-based methods such HiFiC (0.038 seconds) and MS-ILLM (0.059 seconds).
The main reason for this phenomenon is that the progressive denoising process of diffusion models inherently an iterative computational process, with each step requiring complete network forward pass. When the number of steps increases, the computational cost increases linearly. Optimizing encoding and decoding speeds will be important direction for future work: (i) Investigating more efficient sampling strategies, such as implementing samplers with less steps [08]] [09]; (ii) exploring knowledge distillation techniques to distill multi-step denoising networks into lighter single-stey step or few-step networks [III;; (iii) and optimizing network structures to reduce the computational complexity of each denoising step.
While diffusion-basse based image methods currently face computational efficiency challenges, this work primarily serves to demonstrate the fundamental potential of this paradigm. We anticipate that ongoing advancements in sampling algorithm efficiency and targeted engineering optimizations will substantially reduce encoding and decoding latency, thereby enhancing the practical utility of diffusion models in this research field.
B.5 Ablation study of distortion loss calculation space
In Equation (4) of the main text, the distortion loss fisist are calculated in the VAE latent space:
Laia 1z & 2 (x) (9)
(7)
24
Here conduct ablation study of calculating loss in the pixel space:
Caist = Xx x11 = ||x 13)
(8)
where D' maintains the same architectural foundation D. but incorporates additional upsampling blocks
28
Istent space
21
Figure 14: Ablation of distortion loss calcula- tion space.

---

to reconstruct signals at pixel-level resolution. The re- constructed pixel-space output x is subsequently fed into the control module for generative reconstruction. Figure 14] demonstrates the substantial performance advantage of calculating distortion in the latent space rather than pixel space. This finding confirms that the VAE latent space provides a more compact and perceptually meaningful representation that effectively filters semantically irrelevant information from the pixel domain. Through optimizing fidelity the latent space, the compressed bitstream effectively prioritizes semantically salient infor- mation, thereby achieving superior distoritootion performance across various downstream machine vision applications.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The contributions of this paper are claimed in the abstract and the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this paper in the Appendix.
23

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We descrihe the implementation details in the Section Experiments. Besides, code will be released.
Guidelines:
The answer NA means that the paper does not include experiments.

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
Code will be released when published.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeeuiiiiinoooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
25

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The full training and test details are described Section 5.1
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Rate-distortion curves are illustrated verify the effectiveness of proposed method, which are commonly used in the area of compression.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-iimma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the computer resources used in our experiments in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttps:/neeurips http://aaarrr...cooaaaaa/cccccca
Answer:
[Yes]
Justification:
We conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: This work is about the foundamental techniques in terms of image compression, with no societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
27

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite original paper and code used in this paper.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This submission does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
28

---

Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our L.M policy http:s//eaarrtppecc//o/eeeeeeeee for what should or should be described.
29


Reviews:
[{"review_id": "RgUNdYkPz8", "reviewer": "Reviewer_s4Y9", "summary": "The authors proposed Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity.", "questions": "N/A", "limitations": "No. The authors should include a discussion of the model’s limitations. For example, perceptual-quality-oriented compression methods may introduce hallucinated details that were not present in the original image. While this may improve visual appeal, it can be problematic in domains where accuracy and fidelity are critical—such as medical imaging or scientific analysis—where hallucinated content could lead to misleading or incorrect interpretations. Highlighting this trade-off would strengthen the paper by providing a more balanced perspective on the method’s applicability.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Pros:**\nThe paper addresses an important and timely research problem. Traditional lossy image compression methods are typically designed to preserve visual fidelity for human perception or storage/transmission efficiency. However, in many real-world applications, compressed images are used as inputs for downstream vision tasks. The authors correctly identify this gap and motivate their work around optimizing compression for task performance, which is a valuable direction.\n\n---\n\n**Mixed:**\nThe proposed architecture is built upon ControlNet, and the Tag Guidance Module appears to function similarly to existing tagging models. This raises some concerns about novelty. That said, the integration of a semantic consistency loss is a noteworthy aspect that could contribute to the uniqueness of the approach.\n\nIt would be helpful if the authors elaborated on their choice of similarity metric for enforcing semantic consistency—e.g., whether they used cosine similarity, Euclidean distance, or another measure—and discussed its impact on performance.\n\n---\n\n**Cons:**\nThe empirical performance of the proposed method appears limited. As shown in Figure 6, on widely used datasets such as COCO and Flickr, the method performs on par with existing perceptual compression baselines. Furthermore, Figure 7 indicates no significant improvement in perceptual quality, which weakens the empirical claims of the paper.\n\nAdditionally, the paper brings to mind related work such as [Matsubara et al., WACV 2022](https://openaccess.thecvf.com/content/WACV2022/papers/Matsubara_Supervised_Compression_for_Resource-Constrained_Edge_Computing_Systems_WACV_2022_paper.pdf), which explores supervised compression for multiple downstream tasks using a model distillation framework. This prior work seems highly relevant, yet the current paper only compares against reconstruction-based baselines. Including comparisons with task-specific or multitask compression approaches would strengthen the experimental analysis.\n\n---\n\n**Recommendation:**\nI believe this paper sits on the borderline for acceptance. The motivation is strong, and the proposed semantic consistency idea has potential, but concerns about novelty and limited empirical gains weaken the overall contribution. I would consider increasing my score if the rebuttal addresses these concerns effectively.", "was_revised": true, "final_justification": "The author has resolved most of my concern.", "rebuttal": "**Rebuttal to Mixed**\n\n\"It would be helpful if the authors elaborated on their choice of similarity metric for enforcing semantic consistency—e.g., whether they used cosine similarity, Euclidean distance, or another measure—and discussed its impact on performance.\" Thanks for your constructive suggestion. Below are the ablative experiments of the choice of distortion metric to calculate SC loss, including cosine similarity and Euclidean distance. Experiments are conducted on MS COCO object detection. We follow the default training setting for both metrics. It can be observed that the results of cosine similarity achieve a better rate-distortion trade-off. We conjecture that this is because Euclidean distance is sensitive to both the direction and magnitude of vectors, making it susceptible to being dominated by large-magnitude dimensions in the features, which introduces bias in loss computation and hinders overall feature alignment. In contrast, cosine similarity eliminates the influence of vector magnitude through normalization and focuses solely on the directional relationship between vectors, thereby better capturing structural similarities between features and achieving more balanced cross-dimensional feature alignment. We will add the experimental results and corresponding analysis to the Appendix.\n| Distance Metric | bpp | mAP |\n|-----------------|-----------|-------------|\n| Cosine            | 0.0213    | 21.89       |\n|                        | 0.0345    | 26.82       |\n|                        | 0.0628    | 30.80       |\n|                        | 0.1059    | 34.01       |\n|                        | 0.1505    | 35.66       |\n| Euclidean       | 0.0251    | 21.15       |\n|                        | 0.0412    | 26.91       |\n|                        | 0.0698    | 30.33       |\n|                        | 0.1169    | 33.72       |\n|                        | 0.1633    | 35.44       |\n\n\n**Rebuttal to Cons**\n1. About \"The empirical performance of the proposed method appears limited\". First, a major motivation of this paper is to explore a codec that can uniformly adapt to diverse machine vision tasks and human visual perception requirements. Moreover, we must emphasize that Figure 6 includes ten different tasks in total. Our method only performs comparably to the second-best approach on the third and seventh tasks, while demonstrating significant advantages across all other tasks.\n2. About \"Figure 7 indicates no significant improvement in perceptual quality\". First, the core objective of this paper is to develop a general codec that unifies human vision and machine vision. Therefore, the inherent goal is to achieve state-of-the-art or comparable performance simultaneously in both perceptual quality and intelligent tasks. Second, our method achieves superior perceptual quality at lower bitrates compared to other approaches, such as DiffEIC.\n3. About relationship to \"Matsubara et al., WACV 2022\" and \"comparisons with task-specific or multitask compression approaches\". First, regarding the relationship between the two papers, the WACV paper performs adaptive optimization for multiple downstream tasks to achieve better rate-distortion performance on those specific tasks, while our work does not perform any task-specific adaptive optimization and additionally supports human visual perception, making it a generalized codec. We will incorporate the citation of this work in the main text and provide a clear clarification. Furthermore, regarding comparisons with task-specific optimized codecs, we emphasize that our evaluation includes TransTIC and Adapt-ICMH—task-specific methods specifically designed and optimized for particular tasks, as demonstrated in Figure 6 and the corresponding analysis. Notably, Diff-ICMH achieves significantly superior performance compared to these specialized approaches while requiring no task-specific adaptive optimization. This demonstrates not only the exceptional performance of our method but also its superior generalization capability and practical ease of deployment.\n\n**Feedback to Limitations**\n\nWe greatly appreciate your valuable suggestions. We will add a dedicated Limitations section to comprehensively address these concerns, including the trade-offs you mentioned regarding hallucinated details and applications in medical imaging and scientific analysis."}, {"review_id": "Y60WJUKBCA", "reviewer": "Reviewer_NvM5", "summary": "The Diff-ICMH proposed in this paper is a generative image compression framework that utilizes a diffusion-based decoder and a semantic bootstrap module. The core claim is that Diff-ICMH can support a wide range of downstream intelligence tasks - including traditional computer vision tasks and multimodal reasoning - without the need for task-specific adaptation. To support this claim, the authors have experimented on several benchmarks, including object detection, instance segmentation, keypoint estimation, and full-view segmentation on COCO; image text retrieval on Flickr30K; finger representation understanding on RefGTA; and open vocabulary segmentation on ADE20K.", "questions": "See weaknesses above.", "limitations": "See weaknesses above.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThe authors claim that Diff-ICMH supports a wide range of tasks, including both traditional computer vision tasks and multimodal reasoning, without the need for any task-specific adaptation. This is a compelling and practically valuable goal, especially in the context of compressing images for general-purpose downstream usage. The authors provide empirical evidence across several tasks (e.g., object detection, instance segmentation, keypoint estimation, and panoptic segmentation on COCO; image-text retrieval on Flickr30K; referring expression comprehension on RefGTA; and open-vocabulary segmentation on ADE20K). These evaluations span different types of downstream models and include tasks involving both vision and language.\n\nWeaknesses and Questions:\n1. Limitation: A notable limitation in this manuscript is the lack of discussion of the limitations of the proposed approach. Diffusion-based decoders are known to be computationally intensive. Could the authors provide further information on the considerations and issues regarding trade-offs in terms of inference efficiency, cost and resources. Providing quantitative metrics or qualitative insights on these aspects would allow reviewers and future readers/users to better understand the practical deployment implications of Diff-ICMH.\n2. Task-Independent Support: The claim of “task-independent support” remains somewhat under-tested. First, the experiments do not indicate whether any downstream tasks or their semantics were encountered during the training or labelling phases. This raises the question of whether the model truly generalizes in a zero-point manner or benefits from task-aware conditioning implicit in the training process..\n3. Tag Guidance Module (TGM): The authors have conducted experiments using a Tag Guidance Module (TGM) to improve performance at negligible bitrate cost. While this design is somehow efficient, it is still too simple—relying on static tag extraction and direct prompt injection. Why doesn’t the paper explore TGM's robustness and effectiveness under more complex semantic conditions? Why not compare it with more powerful alternatives such as CLIP-based conditioning or learned prompt optimization methods? Any of these trade-offs or comparisons could significantly enhance the paper’s quality. Without such analyses, the current TGM contribution, while practical, may be perceived more as an ad-hoc engineering choice than a principled or generalizable advance. Providing such evaluations would not only strengthen the paper but also offer clearer guidance for future researchers considering whether and how to apply TGM in their own works. Since prompt-based testing is not difficult to conduct experimentally, the authors could consider including their TGM evaluations under more complex scenarios as well as comparisons with alternative prompt-based guidance methods, depending on their available resources.\n4. Generalizability: The generalizability of the model to completely unseen domains or tasks (e.g., medical imaging, real-time video comprehension, or other non-visual language settings) was not explored. Thus, while the current empirical results are encouraging, the strength of the task-independence statement would be greatly enhanced by including evaluations under unseen or out-of-distribution tasks, comparisons to a task-adaptation baseline, and explicit measurements of the cost of adaptation (e.g., training time, number of fine-tuning parameters, or latency).", "was_revised": true, "final_justification": "I'd like to thank the authors for their response. I've raised the overall score.", "rebuttal": "**Rebuttal to Weaknesses and Questions**\n1. \"lack of discussion of the limitations of the proposed approach\". Thank you very much for your feedback. Indeed, we have included experimental results on computational complexity and a discussion of limitations in Section B.4 of the Appendix. To enhance clarity, we will reorganize this content into a dedicated section in the main text to more clearly present the limitations of our approach.\n2. \"Concerns of Task-Independent Support\". We sincerely apologize for any misunderstanding. We must emphasize that the entire training process does not involve any explicit or implicit optimization targeting any downstream task. Our model is optimized solely on the LSDIR dataset using diffusion-based ε-prediction loss, rate optimization, and our proposed semantic consistency loss, then directly tested on different downstream intelligent tasks. Therefore, Diff-ICMH is a general and generalizable codec that can zero-shot support different intelligent tasks once training is completed.\n3. \"Tag Guidance Module (TGM) is simple\". Thank you very much for your suggestion. This is indeed a highly valuable question for discussion. In fact, the design of TGM follows a simple yet efficient philosophy, grounded in two core design principles: **First, leveraging the strong generative prior of T2I models.** As is well known, T2I models are inherently designed to generate images conditioned on text. Therefore, choosing text as the conditioning modality is a natural and reasonable choice. Methods such as CLIP image representations or learned prompt optimization approaches would deviate from the original input format of T2I models, potentially hindering the effective activation of the model's prior knowledge. We have experimented with using the CLIP grid image features of the image as the condition instead of tags. However, we found that even with CLIP features free from any compression distortion, their performance was still inferior to that of tags. As a result, we abandoned this approach. **Second, achieving negligible bitrate overhead**. Another core design principle of TGM is encoding/decoding tags, which requires extremely minimal bitrate to compress the corresponding tags as stimulation factors for activating the generative prior. As described in the paper, each image requires approximately only 100 bits. Specifically, we utilize the RAM++ version of Recognize Anything (https://github.com/xinyu1205/recognize-anything), which has a maximum default vocabulary of 4,585 tags. We employ fixed-length encoding to compress tag IDs, requiring 13 bits per ID (with a maximum capacity of 8,192). Our statistics on COCO show that each image predicts an average of 8.7 tags under RAM++'s default settings, resulting in an average bitrate of 13×8.7=113.1 bits. In contrast, compressing text would require significantly more bits, not to mention CLIP-based representations or learned prompt optimization methods, which typically involve high-dimensional floating-point continuous representations (e.g., 2,048, 3,072, or 4,096 channels). However, the CLIP-based conditioning or learned prompt optimization methods you mentioned are indeed promising directions for future research, though they would necessarily require targeted designs to address the high bitrate requirements of such representations and how to well stimulate the generative prior, which exceeds the scope of this paper. The core motivation of our work is to demonstrate that diffusion-based perceptual coding methods can effectively unify human perception and machine vision tasks. For this motivation, our designed TGM serves as a simple and practical module.\n4. \"Concerns of Generalizability\". First, this work primarily validates the effectiveness of our method through common vision benchmarks, which is consistent with how other methods (Omni-ICM[1], TransTIC[2], Adapt-ICMH[3]) conduct their testing and validation. To our knowledge, we are the first to encompass such a diverse range of vision tasks and datasets, totaling 10 intelligent tasks. Other domains such as medical imaging, real-time video comprehension, and non-visual language settings are beyond the scope of this paper, though they could be explored in future work. Additionally, we have also tested on out-of-distribution (OOD) datasets, specifically RefCOCO, corresponding to the seventh subplot in Figure 6 and the related discussion at L241, and demonstrated the generalization ability of our method on OOD data.\n\n[1] Feng R, Jin X, Guo Z, et al. Image coding for machines with omnipotent feature learning[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 510-528. \n\n[2] Chen Y H, Weng Y C, Kao C H, et al. Transtic: Transferring transformer-based image compression from human perception to machine perception[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 23297-23307. \n\n[3] Li H, Li S, Ding S, et al. Image compression for machine and human vision with spatial-frequency adaptation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 382-399."}, {"review_id": "FksAIugLrn", "reviewer": "Reviewer_vy5U", "summary": "This paper proposes a diffusion-based ICMH framework, which is implemented by a control-net and proposed *TGM*. The network is trained by rate-distortion loss, diffusion loss and proposed *SC loss*. The main contribution is that the proposed framework reserves both semantic and perceptual information, appending the generated information to obtain better down-stream task performance. This method achieves SOTA performance across various downstream tasks.", "questions": "- Could you provide more ablation studies evaluated on FID and LPIPS metrics ?\n- Why do you adopt the middle feature in the diffusion model to calculate the *SC Loss* ? The network architecture of diffusion models should be a U-net. The middle feature of a U-net only contains the lowest scale information, which is clearly NOT like the bottleneck of an autoencoder that contains full compressed semantic information.", "limitations": "No discussion on limitations.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n- The motivation is clear and convincing.\n- This paper is well experimented with ablation studies on proposed network and SOTA comparisons.\n- The proposed method achieves SOTA performance across various downstream tasks.\n\n### Weaknesses\n\n- The contributions presented in this paper are a little insufficient: Most of the modules in the framework (e.g., autoencoder, tag extractor and stable diffusion) are pretrained.\n\n- Poor inference efficiency: ICMH should be a very practical application-oriented field, as it is often designed for downstream tasks. However, the reported encoding/decoding time is much longer than existing methods.", "was_revised": true, "final_justification": "Although the novelty appears somewhat inadequate, the experimental results show the robustness of the method. Overall, I think this is a borderline paper.", "rebuttal": "**Rebuttal to Weaknesses**\n1. \"Most of the modules in the framework (e.g., autoencoder, tag extractor and stable diffusion) are pretrained\". We sincerely apologize for any confusion. As shown in Figure 4 and described in L199-L200, the tag extractor and stable diffusion model are indeed pre-trained and frozen, but the autoencoder and control module are completely trained from scratch.\n2. \"Poor inference efficiency\". Yes, you are absolutely right, and this is a very important question. We have also provided corresponding experimental results and related analysis in the Appendix Section B.4. The slow decoding speed is mainly caused by the iterative denoising process. To more clearly address this concern, we will add a dedicated Limitations section in the main text to elaborate on this problem. However, it is worth noting that the core motivation of this work is to demonstrate the significant value of Diff-ICMH in unifying human perception and machine vision. The earliest autoregressive-based codecs [1,2] also required tens of seconds to decode a single image, but later, with the development of better entropy models, decoding speed improved from tens of seconds to tens of milliseconds. In recent years, inference speed optimization algorithms for diffusion models have also been rapidly developing, with numerous model distillation [3] and inference step optimization [4,5] methods emerging continuously. We believe that in the near future, combining diffusion models for image encoding and decoding will no longer be constrained by complexity and lengthy encoding/decoding times.\n\n**Answers to questions**\n1. \"More ablation studies evaluated on FID and LPIPS metrics\". Thank you for your constructive suggestions. We conducted ablation studies of LPIPS and FID respectively on CLIC2020 test set to explore the impact of our proposed SC loss and TGM (tag guidance module) on subjective quality metrics. The results are shown in the table below. It can be observed that SC loss and TGM have minimal impact on LPIPS. We speculate this is because LPIPS primarily focuses on perceptual accuracy of reconstructed images, particularly low-level features, while SC loss and TGM mainly optimizes semantic consistency, which is not directly correlated with the LPIPS metric. On the other hand, SC loss and TGM significantly help reduce FID, as they implicitly and explicitly enhance semantic information in the bitstream respectively, thereby better activating the pretrained generative priors.\n| Method | bpp | LPIPS | FID |\n|--------|-----|-------|-----|\n| w/o SC loss, w/o TGM | 0.1073 | 0.1013 | 7.439 |\n|  | 0.0319 | 0.1979 | 14.81 |\n| w/ SC loss, w/o TGM | 0.1069 | 0.0967 | 7.293 |\n|  | 0.0322 | 0.1949 | 13.95 |\n| w/o SC loss, w/ TGM | 0.1071 | 0.1007 | 6.947 |\n|  | 0.0315 | 0.1986 | 13.31 |\n| w/ SC loss, w/ TGM | 0.1066 | 0.0961 | 6.821 |\n|  | 0.0318 | 0.1966 | 12.73 |\n2. \"Why do you adopt the middle feature in the diffusion model to calculate the SC Loss ?\" Thank you for your constructive feedback. This is an excellent question. We initially also believed that shallower representation spaces might preserve more comprehensive information, which would be more beneficial for model training and supporting subsequent intelligent analysis. However, as shown in Figure 8(b), we found that computing the SC loss only at the middle features yields the best results, which we analyzed in L276-L278. Overall, this is likely because deeper features, i.e., middle block features, can capture higher-level semantic information while filtering out some noisy low-level perceptual information. Computing the SC loss in this representation space is more conducive to preserving high-level semantics while filtering out irrelevant information that should not be compressed and transmitted, thereby achieving better rate-distortion performance.\n\n​[1] Cheng Z, Sun H, Takeuchi M, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 7939-7948. \n\n[2] Minnen D, Ballé J, Toderici G D. Joint autoregressive and hierarchical priors for learned image compression[J]. Advances in neural information processing systems, 2018, 31. \n\n[3] Sauer A, Lorenz D, Blattmann A, et al. Adversarial diffusion distillation[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 87-103.\n\n[4] Sauer A, Boesel F, Dockhorn T, et al. Fast high-resolution image synthesis with latent adversarial diffusion distillation[C]//SIGGRAPH Asia 2024 Conference Papers. 2024: 1-11. \n\n[5] Meng C, Rombach R, Gao R, et al. On distillation of guided diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 14297-14306."}, {"review_id": "mhHTTjwQg0", "reviewer": "Reviewer_fiP4", "summary": "The authors propose that image compression meant for downstream tasks should be built accordingly to emphasize more semantic fidelity. They propose Diff-ICMH, a generative image compression method based on pretrained diffusion models that tries to do just that, while also optimizing for perceptual quality. Specifically, they train a new encoder-decoder to compress the latent representation of a given image (incorporating a semantic consistency loss), and on the client side decompress the image by guiding the diffusion model with image tags (also transmitted) and by using controlNet to condition the model on the decompressed latent vector. They show strong performance on downstream tasks.", "questions": "- L251: The paper suggests that tag-based conditioning incurs minimal overhead, but explicit numbers would help quantify this claim. 100 bits for a 40K words dictionary would be less than 10 words? The authors mention they set a limit, this should be reported.\n- The authors use Kodak images, but considering their size (768×512), how well do they align with the 768x768 SD 2.1?\n- L183: From my knowledge and experience, the VAE latent space isn't very smooth (e.g. [3]). It baffles me that optimizing similarity in this latent space helps enforce actual similarity. How do you explain that? Also, how does this affects the robustness of the downstream tasks results?\n\n[3] Norm-guided latent space exploration for text-to-image generation. D Samuel, R Ben-Ari, N Darshan, H Maron, G Chechik - Advances in Neural Information Processing Systems, 2023.", "limitations": "No, the authors mention in the checklist they do in the appendix, but this is very briefly touched upon only in a complexity analysis. No mention at all about the different training needed for different bit-rates.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n- Using tags instead of automatically generated sentences (e.g. by using BLIP) is a good idea.\n- The proposed method exhibits strong performance across a diverse range of downstream tasks, demonstrating its adaptability and effectiveness.\n- I believe using pre-trained diffusion models is a good direction for compression algorithms, leveraging their generative capacity to enhance visual quality. \n- Faire reporting on PSNR comparing non-latent based methods (showing that using a VAE induces limitations).\n\n## Weaknesses\n- The novelty is somewhat unclear to me. The claim that combining perceptual optimization and Image Compression for Machines (ICM) is novel is somewhat questionable. The concept of using a single codec for multiple downstream tasks has been previously explored (e.g., DDCM [1]). Furthermore, traditional codec-based methods arguably serve similar roles and should not be excluded from such categorization. \n- Motivation: In L110 the authors argue that traditional image compression methods harm semantic integrity, but I see no evidence that they perform poorly on that front. On the contrary, perceptually oriented methods inherently maintain semantic accuracy, making this assertion debatable.\n- **Clarity Issues and Structural Organization:**\n  - Figure 1 is unclear to me, conveying very limited information (and the caption does not explain as well). I don't see why it should occupy so much space in the main paper. Instead, the authors should use that space for the image examples they lack. \n  - Section 4.1 – It's unclear if the generative process follows the standard diffusion process (does $z_t$ starts from random noise here)?\n  - It's not explicitly stated what information is transmitted and compressed, and the reader must infer it consists of the compressed latent space representation alongside the tag bits. This should be clearly written.\n   - Figure 7 lacks organization. Metrics are arranged seemingly at random, making it difficult to follow ((E.g., Kodak PSNR is at [0,0], its LPIPS is in [1,0], it's MS-SSIM is [0,2], and DISTS in [1,2]). Not all metrics mentioned in the caption are shown for each DB, e.g. Kodak lacks FID. If these are omitted due to the number of images in the DB, patches should be used instead (this is quite commonplace, see e.g. [1])\n  - Figure 5: the arrow comes out from the bottleneck of the diffusion model, which suggests the semantic space you consider is the h-space of the diffusion model. The text however implies it is the VAE latent space. Please clarify. Additionally, Eq 1 seems to suggest this is not happening per time step, which confuses me about when the loss is performed.\n  - Ablations in Section 5.4: The mAP metric lacks context—was it calculated on a specific task or averaged across multiple tasks? L276 mentions better rate-distortion performance, but the evidence supporting this claim is unclear.\n- The paper includes multiple comparisons to PerCo. However, their internal model was not released. Did the authors train their own PerCo version, or used the open sourced version, \"Perco (SD)\"? If the latter, the corresponding paper [2] should be cited. \n- The DDIM step count differs between methods—Diff-ICMH utilizes 50 steps, whereas PerCo is restricted to 20 steps. PerCo’s perceptual quality may suffer from this shorter inference process. A fairer comparison using equivalent step counts would be preferable.\n- The ablations are lacking with regards to the necessity of TGM. Recent methods seem to not need text guidance for similar bitrates ([1]).\n- Visualizations: The decision to limit visual examples to two and relegate perceptual comparisons to the appendix is problematic, to say the least. Given the claim of perceptual superiority, a more thorough presentation in the main paper is warranted, along with additional examples in the appendix. Furthermore, these perceptual comparisons (Fig 12) do not compare to SOTA diffusion based methods such as DiffEIC, PerCo or DDCM, raising concerns regarding the fairness of comparisons to similar generative priors. Also, in the first image the proposed method's BPP is one of the highest in the figure, leading to unfair comparison, for a fair comparison the proposed method should have the same BPP roughly or lower. Finally, the zoomed region in Figure 12 would be better positioned as an additional row rather than obscuring portions of the original image.\n- The results in Figure 7 appear identical to those reported in DiffEIC. If data reuse occurred, this should be explicitly acknowledged and cited. Is that the case? If so, the authors should verify their evaluation protocol to ensure the reuse did not affect the validity of the reported performance.\n- The paper lacks discussion on trained, partially trained, and zero-shot compression methods, which would provide a more balanced perspective on result comparisons (L251)\n- Table 2 reports speed comparisons across different hardware settings, which does not make sense. \n- This method requires training (or fine-tuning) for every new desired bit-rate.\n\n[1] Compressed Image Generation with Denoising Diffusion Codebook Models. G Ohayon, H Manor, T Michaeli, M Elad - arXiv preprint arXiv:2502.01189, 2025\n\n[2] PerCo (SD): Open Perceptual Compression. N Körber, Workshop on Machine Learning and Compression, NeurIPS 2024.", "was_revised": true, "final_justification": "The authors answered most of my concerns, importantly:\n- clarifying figures 7 & 5\n-  Will add ablations over different tasks\n- fix details such as referring to PerCo (SD), and consistent step counts, referring to DiffEIC metrics, speed on same devices\n- promised to add more visualization and move at least one to the main text\n- limitations will be added to the paper concretely.\n\nHowever, the novelty is still somewhat weak in my eyes. Fig 1 is also still a bit redundant in my opinion, and the authors should at least make it more compact.", "rebuttal": "**Rebuttal to Weaknesses**\n- About \"Novelty\". To our knowledge, we are the first to explore a single codec with a single bitstream that adapts to different intelligent tasks and human perception. By combining generative priors, we designed SC Loss and TGM to achieve this goal. Besides, we have carefully reviewed the DDCM paper. This work primarily focuses on compression algorithms for perceptual quality and does not include any exploration on intelligent tasks. We will cite this paper and provide distinction and discussion. Finally, we show the pipeline of adapting traditional codecs to different tasks in the first diagram of Figure 1 and discuss it in L30-L35. Although the traditional codec itself is general, the bitstream is task-specific. This is because they often optimize for specific tasks, such as some RoI bit allocation methods, whereas Diff-ICMH does not require any additional design or separate bitstreams.\n- About \"Motivation\". We sincerely apologize for any misunderstanding. We must emphasize that in the paper, we do not make any claim that traditional image compression methods \"harm\" semantic fidelity; they simply do not focus on semantic fidelity. Nor do we claim that perceptual-oriented methods can inherently better preserve semantic accuracy. In the motivation section, our claim is that two key points for intelligent tasks are semantic fidelity and perceptual realism. The former directly relates to whether tasks can be completed normally, while the latter can cause error accumulation and further semantic distortion due to training-inference mismatch. Traditional signal fidelity-oriented image compression methods only optimize for signal fidelity itself and do not focus on semantic fidelity and perceptual realism. These methods include both traditional codecs and learning-based codecs. Perceptual-oriented methods can achieve better performance compared to pure signal fidelity-oriented image compression methods because they better ensure perceptual realism. Diff-ICMH aims to simultaneously address both semantic fidelity and perceptual realism mentioned above. We will further emphasize this statement in the paper.\n- About \"Clarity Issues and Structural Organization\".\n  - \"Necessity of Figure 1\". We must emphasize that Figure 1 is a very important illustration. As we introduce in L30-L40, previous codecs for intelligent tasks either require multiple codecs to achieve this purpose, or different intelligent tasks need different bitstreams. Therefore, the original starting point of this work, as shown in the last diagram of Figure 1, is to use a unified codec with a generalized bitstream to adapt to different intelligent tasks as well as human vision.\n  - \"generative process follows the standard diffusion process?\" Yes. We will describe this more clearly in the paper.\n  - \"not explicitly stated what information is transmitted\". As illustrated in Figure 4, the transmitted bitstream consists of the compressed latent space representation and tag bits. We will accurately describe this linguistically in Section 4.1.\n  - \"Figure 7 lacks organization.\" Thank you for your suggestion. We will reorganize the arrangement of these figures to present the results more clearly.\n  - \"where the SC loss is conducted\". Thank you very much for your careful review. The semantic space refers to the feature space of the diffusion model, as illustrated in Figure 5. We will revise the wording for greater clarity in the paper.\n  - \"Ablations in Section 5.4\". Regarding the experimental setup for ablation studies. As stated in L262, the experimental details for the ablation study correspond to section A.2 in the appendix: \"The experimental results were rigorously evaluated on the COCO 2017 object detection benchmark dataset.\" Regarding the results, as illustrated in Figure 8 (b), results of the middle block achieves the best RD performance (in the Rate-Distortion performance graph, closer to the upper left indicates better performance), which demonstrates our claim.\n- \"Did the authors train their own PerCo version, or used the open sourced version?\" We adopted Perco(SD): https://github.com/Nikolai10/PerCo. We will cite and acknowledge it.\n- \"The DDIM step count differs between methods\". Experiments with steps=20 of Diff-ICMH and PerCo are shown below. It can be observed that our method also significantly outperforms PerCo with the same denoising steps.\n| Method    | bpp    | mAP   |\n|-----------|--------|-------|\n| Diff-ICMH | 0.1092 | 34.68 |\n|   | 0.0350 | 27.88 |\n| PerCo  | 0.1274 | 33.53 |   \n|  | 0.0332 | 21.50 |\n\n- \"The ablations are lacking with regards to the necessity of TGM\". We sincerely apologize for any misunderstanding. We must emphasize that we did not omit the ablation study for TGM. Figure 9 and L263-L272 correspond to the ablation study experimental results and analysis for TGM and SC Loss. We will change \"tag guidance\" to \"TGM\" in Figure 9 to avoid similar misunderstandings.\n- \"Visualizations\"\n  - \"A more thorough presentation in the main paper ... along with additional examples in the appendix\". Excellent suggestion. We will reorganize to include key perceptual results in the main paper and expand the Appendix with additional examples.\n  - \"perceptual comparisons (Fig 12) do not compare to SOTA diffusion based methods such as DiffEIC, PerCo or DDCM\". Thanks for your suggestion. We will add comparison results with those SOTA diffusion based methods.\n  - \"in the first image the proposed method's BPP is one of the highest in the figure\". Thank you for your insightful feedback. We will adjust the bitrates in our comparisons for improved clarity.\n  - \"zoomed region in Figure 12 ... additional row\". We appreciate this constructive suggestion. We will add additional rows of zoomed-in images for each comparison group to enable more precise visual analysis.\n- \"data reuse\". Yes, the experimental results in Figure 7 are consistent with DiffEIC. We will explicitly acknowledge and cite it.\n- \"lacks discussion on trained, partially trained, and zero-shot compression methods\". We seek clarification on the comment about \"trained, partially trained, and zero-shot compression methods.\" Standard neural codec evaluation requires fully trained models for meaningful comparison. Partially trained or zero-shot approaches would significantly degrade performance and provide uninformative comparisons.\n- \"speed comparisons across different hardware settings\". Yes, the evaluation is conducted across different hardware configurations. However, we emphasize that the primary focus of this paper is to demonstrate the generalizability and significant potential of Diff-ICMH for diverse intelligent tasks and human perceptual quality assessment. Therefore, computational complexity analysis was not the central emphasis of our current work.\n- \"requires training for every new desired bit-rate.\" Yes, each rate point currently requires a separate trained model. However, most methods not specially designed for variable-rate compression also use one model per bit-rate, which is the standard validation approach for novel neural compression techniques. Our primary contribution is demonstrating Diff-ICMH's generalizability across diverse intelligent tasks and human perceptual quality assessment. While codecs equipped with variable-rate adjustment can achieve multiple rates with a single model, such engineering implementations are beyond our current research scope.\n\n**Answers to Questions**\n- \"details of explicit numbers of tag conditions\". Thanks for your constructive suggestion. We are using the RAM++ version from Recognize Anything (https://github.com/xinyu1205/recognize-anything). Its maximum default vocabulary is 4585, and we do not perform any entropy coding but use fixed-length encoding, so each ID requires 13 bits for compression (max as 8192). Additionally, we counted that each image on COCO predicts 8.7 tags on average under RAM++'s default setting, so the final average bit count is 13×8.7=113.1 bits. We will supplement these specific details in the article and supplementary materials. Thank you again for your suggestion.\n- \"resolution mismatch\". Thank you for this insightful observation. You're correct that stable diffusion models trained on fixed resolutions can suffer from training-inference mismatch when generating different resolutions, often leading to quality degradation and structural collapse. However, our approach incorporates ControlNet, which provides additional control signals that effectively address these structural issues. This enables robust adaptation across different resolutions. As demonstrated in the original ControlNet paper, the method successfully handles various aspect ratios and resolutions while using Stable Diffusion 1.5 (pretrained at 512×512) as the base model.\n- \"why optimizing similarity in this latent space helps enforce actual similarity\". Essentially, the approach can be viewed as compressesing a compact representation that filters out irrelevant textural details. Specifically, Stable Diffusion's 4-channel VAE, optimized under MSE+LPIPS+GAN loss, functions as a perceptual-oriented lossy compression process rather than simple dimensionality reduction. As discussed in the appendix (L131-L136), VAE latent space provides a compact, perceptually meaningful representation that filters semantically irrelevant information. Comparing to compressing the original pixel-level signal, compressing VAE's latent variables allows the compressed bitstream to prioritize semantically salient information and avoid wasting bits on those hard-to-code and complex textures, achieving superior rate-distortion performance across downstream machine vision applications. The results in Appendix Section B.5 and Figure 14 also confirm our statement. Besides, to address potential convergence issues from extreme values of VAE latent variables, we compress the latents after normalization. We will add this implementation detail to the Appendix."}]

Author rebuttal:

Meta-review: This paper develops a framew, called Diff-ICMH, which is a generative image compression framework designed to align machine and human vision in image compression. It achieves perceptual realism through the use of generative priors while also maintaining semantic accuracy. 

I thank the authors for their detailed responses and additional empirical results, which effectively addressed the reviewers’ major concerns. All reviewers are now in favor of accepting the paper. I kindly ask the authors to ensure that the paper is updated with the new results and that the reviewers’ comments are thoroughly addressed. I believe the authors have summarized the necessary revisions very clearly in their “final remarks” message.
Decision: Accept (poster)