Consistency Guided Diffusion Model with Neural Syntax for Perceptual Image Compression

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:42:50

Abstract: Diffusion models show impressive performances in image generation with excellent perceptual quality. However, its tendency to introduce additional distortion prevents its direct application in image compression. To address the issue, this paper introduces a Consistency Guided Diffusion Model (CGDM) tailored for perceptual image compression, which integrates an end-to-end image compression model with a diffusion-based post-processing network, aiming to learn richer detail representations with less fidelity loss. In detail, the compression and post-processing networks are cascaded and a branch of consistency guided features is added to constrain the deviation in the diffusion process for better reconstruction quality. Furthermore, a Syntax driven Feature Fusion (SFF) module is constructed to take an extra ultra-low bitstream from the encoding end as input, guiding the adaptive fusion of information from the two branches. In addition, we design a globally uniform boundary control strategy with overlapped patches and adopt a continuous online optimization mode to improve both coding efficiency and global consistency. Extensive experiments validate the superiority of our method to existing perceptual compression techniques and the effectiveness of each component in our method.

Main paper: 
 introduction: 
 Image compression technology plays a pivotal role in diverse fields, e.g., multimedia, communications, and computer vision. Its objective is to efficiently minimize the storage space and bandwidth requirements of digital images for their efficient storage and transmission, while preserving the main content of the original images and maintaining the visual quality. In today's digital age, with the ever-increasing demand for high-resolution and high-quality images from multimedia devices, the challenge of managing storage and bandwidth resources has become paramount, and the demand for more efficient and high-performance image compression methods is also growing rapidly.Over the past decades, conventional image compression techniques like JPEG [52] , BPG [5] and JPEG2000 [37] have become the common choice in image processing. These methods or standards exhibit excellent encoding capabilities, adopting the route of transform/hybrid coding framework for Rate-Distortion Optimization (RDO) with key modules such as transformation, quantization, and entropy coding, while being complemented by additional complex predictive modes. Moreover, numerous efforts have been made to improve the RDO of each input image by projecting the image signals into a manually designed specific subspace for more compact representations, e.g. intra-prediction based on various directions [52] . Nonetheless, these optimizations rely on human design, lacking the capability for global optimization. As the number of manually designed strategies continues to grow, the framework of the compression model becomes increasingly complicated, gradually revealing its performance bottlenecks.In recent years, deep learning technologies have made remarkable advancements, prompting many researchers to explore image compression methods with the immense capabilities of neural networks [4, 18, 24, 35, 53] . By leveraging vast datasets to train neural networks, these methods excel in discovering the latent relationships and underlying structures embedded within image data. As a result, they attain a remarkable compression efficiency, while ensuring minimal distortion, thereby surpassing traditional techniques in terms of performance. While these methods have achieved a notable level of image compression and reconstruction, they gradually encounter performance bottlenecks: further improving the compression ratio resulting in significantly degraded quality.Some researches [6, 55] indicate that the distortion of images does not align with human perception of subjective image quality. Due to the inherent trade-off between image quality and storage efficiency, perceptual image compression techniques [38, 42, 56] are proposed. These technologies strive to protect the quality in terms of human visual perception rather than focusing on fidelity measurement. To enhance the subjective visual quality of images, many generative models [17, 32, 51] skill at producing visually appealing details such as GANs are seamlessly integrated into image compression methods [38, 42, 58] resulting in a significant improvement in perception quality. However, the development of these methods is constrained by the inherent limitations of generative models, including the frequent lack of diversity in the images produced by GAN models and so on, thereby posing significant challenges in their further advancement.In recent years, diffusion models [21, 34, 48] have emerged as a powerful tool in the field of image generation, exhibiting remarkable capabilities in producing images with exceptional perceptual quality. These models, developed according to the formulation of diffusion processes, have demonstrated their ability to capture intrinsic details and generate realistic images. However, despite their remarkable success in image generation, it has been established in many practices that vanilla diffusion models tend to reconstruct images with richer visual details at a cost of significantly impaired fidelity [45] , due to the random nature of the process of progressive adding or removing noise. Applying the diffusion models directly to the image compression task may take on risks of a synchronous drop of both visual quality and fidelity. Thus, until now, how to apply diffusion models to image compression remains under-explored.To address the issue of utilizing the power of diffusion models while avoiding their generated artifacts, this paper proposes to regularize the diffusion models with global consistency guidance. In detail, we propose a novel approach called the Consistency Guided Diffusion Model (CGDM), which incorporates additional consistent guidance into the network structure of the diffusion model. This approach aims to constrain deviations in the diffusion process for improving the quality of the reconstructed image. Furthermore, we propose a Syntax driven Feature Fusion (SFF) strategy. This strategy encodes an additional ultra-low bitstream obtained from the encoding stage, providing semantic prior information about the image. By leveraging this prior information, we can reduce the ambiguity in the inference target during the post-processing phase, leading to more accurate and faithful reconstructions. To achieve the same objective of reducing randomness in the diffusion process and effectively leveraging image semantic information, we apply a globally consistent edge control strategy into our model's inference phase. Additionally, we adopt a continuous online optimization mode to further enhance the model's performance. These efforts not only reduce the stochasticity associated with diffusion processing but also entourage the model to capture rich semantics from the input images, thereby leading to improved overall performance.Our contributions are summarized as follows:• We develop a Consistency Guided Diffusion Model (CGDM) for perceptual image compression, which incorporates an additional consistent guidance with a diffusion-based network, aiming to constrain the deviation in the diffusion process for learning richer detail representations with less fidelity loss. • We devise a Syntax driven Feature Fusion (SFF) module that takes an extra ultra-low bitstream from the encoding end as input, guiding the adaptive fusion of information from the two branch. • We design a globally uniform boundary control strategy, and adopt continuous online optimization mode to further improve both coding efficiency and global consistency. Experimental results show that our proposed method achieves a BD-rate [16] savings of 9.227% in perception and 6.251% in distortion compared to the current state-of-the-art perceptual image compression method ILLM [42] . 
related works 2.1 generative model: 
 Generative models aim to learn the overall distribution of data and generate data within the same distribution, which have been a central focus of research in recent years, leading to significant advancements in multimedia generation and processing. With the rapid development of deep learning, one of the milestone and the most notable generative models is the Generative Adversarial Networks (GANs) [17] , which consist of two competing networks, a generator and a discriminator. The adversarial training process results in highly realistic and diverse generations. Subsequent work built upon GANs such as Conditional GAN [41] , StyleGAN [25, 26] further enhances its ability to generate high-quality images based on given conditions. Beyond GANs, there has also been a surge of interest in other types of generative models, including autoregressive models [15] , variational autoencoders (VAEs) [29] , normalizing flow models [11] , energy-based models [12] , score-based model [49] , flow-based model [28] and so on.Recently, diffusion models [21, 34, 48] have emerged as powerful generative models that define the forward and reverse diffusion processes for data noise addition and removal, respectively. Their generations often exhibit superior quality and diversity, and there have been many studies attempting to use diffusion-based models for image generation [9, 59] , enhancement [13, 36, 43, 57] and so on. In our work, we apply the diffusion-based generation model to image compression, and obtain the image with higher quality through the guidance of a semantic stream.With the significant advancements in deep learning, recent years have witnessed deep learning based image compression methods outperforming classical methods in striking a balance between bit rate and reconstruction quality. Initially, Ballé et al. [2, 3] pioneered the utilization of neural network to establish lossy image compression autoencoders, sparking a surge in learning-based image compression methods [39, 40] . In addition to transformations, numerous studies have focused on entropy coding of latent representations based on learned probability models, including hyperpriors [4] and context models [7, 33] . Furthermore, the employment of Gaussian Mixture Models and attention-based modules in transformations has further enhanced image compression performance [8] .Facing the trade-off between image quality and storage efficiency, a range of perceptual image compression methods have been proposed, which aim to enhance the perceptual quality of compressed images and align them more closely with human perception. Agustsson et al. [1] introduced the concept of using GANs [17] as decoders for image compression. This approach allows for the generation of reconstructed images with rich details. Subsequently, He et al. [19] further enhanced these GAN-based methods by incorporating advanced perception models. Recently, with the great success of diffusion models, some efforts [22, 56] have been made to study perceptual image compression. However, as we have stated previously, due to the lack of fidelity caused by the uncertainty of the diffusion process, this area needs to be further explored. In this paper, we propose a solution for this issue through an additional consistent guidance and a neural syntax driven strategy. 
method: 
 In this part, we first describe general information of diffusion models while outlining our motivations in Section 3.1, followed by a detailed elaboration on our proposed consistency guided diffusion model in Section 3.2. Then, we further propose our syntax driven feature fusion module in Section 3.3. The globally uniform boundary control strategy and continuous online optimization during inference is introduced in Section 3.4. Finally, our training strategy is described in Section 3.5.We start with the characteristic analysis of the diffusion model. As a generative model, diffusion models have been demonstrated to effectively create images with excellent perceptual quality by leveraging a conditional model that incorporates latent features.Simultaneously, there are numerous works [13, 36, 43, 57 ] that employ the diffusion models as a post-processing or enhancement module. Generally, these works utilize the degraded image x as a condition and construct a conditional model that aims to learn the data distribution 𝑝 (𝑥 | x) through a fixed multi-step chain of length 𝑇 . The diffusion process is defined by a forward process 𝑞 through adding Gaussian noise. Formally, the distribution of the forward process can be expressed as:EQUATIONwhere 𝛼 𝑡 and 𝜎 2 𝑡 are hyper-parameter functions of 𝑡 [34] . Meanwhile, the inference process can be conducted as a reverse process from Gaussian noise 𝑞(𝑥 𝑇 )∼ N (0, I) to a target 𝑥 0 , which can be expressed as:EQUATIONwhere the 𝜇 𝜃 ( x, 𝑥 𝑡 , 𝑡) denotes the mean value of the conditional distribution 𝑝 𝜃 (𝑥 𝑡 -1 |𝑥 𝑡 , x), and the diffusion model is trained to learn the conditional distributions by parametric approximation to the distribution. For the neural network, mostly existing diffusion post-processing frameworks directly feed the condition x and noise 𝑥 𝑡 , along with the timestamp 𝑡, into the U-Net backbone, similar to the vanilla DDPM and output the predicted noise ε𝑡 at each step. However, there are two notable issues with this paradigm:• This approach often leads to the final reconstructed image 𝑥 0 deviating from the condition x, amplifying the distortion of x and results in simultaneous degradation of both fidelity and perceived quality. • As the condition x represents a degraded image with some information loss, there can exist multiple images that lead to the same x. That means, the optimized probability distribution target 𝑝 (𝑥 | x) is ambiguous, making it challenge to ensure that the reconstructed image is more similar to the original image. In our work, we aim to adopt a novel framework to address both of these issues:• For the first issue, we propose to incorporate an additional consistent guidance into the network structure of the diffusion model, called consistency guided diffusion model, constraining the deviation in the diffusion process and improving the quality of the reconstructed image. • For the second issue, we propose a syntax driven feature fusion strategy to encode an additional ultra-low bitstream 𝑠 from the encoding stage to provide semantic prior information of the image, thereby alleviating the ambiguity in the inference target of post-processing. In the following sections, we describe our method in detail.On a high level, our compression process consists of two parts, an end-to-end image compression model and a diffusion-based postprocessing model called consistency guided diffusion model, with structure shown in Fig. 2 . End-to-end image compression model. Firstly, we utilize a standard end-to-end image compression network to perform lossy compression on the original image 𝑥, obtaining an image with some detailed information lost x:EQUATIONwhere Q() means quantizer and E(), D() represents the pre-trained autoencoder. Here, we utilize the recently proposed ILLM architecture [42] as the end-to-end autoencoder, which is the current stateof-the-art perception-oriented end-to-end compression method. Diffusion-based post-processing model. Our diffusion-based post-processing model follows the encoder-decoder architecture with skip connections [44] as the denoising model similar to [14] , which includes two encoders and one decoder. The upper branch in Fig. 2 encodes the noisy image into 𝑁 multiresolution diffusion feature maps 𝑑 𝑖 with different scales, where 𝑁 is the depth of the U-Net backbone and 𝑖 ∈ {1, ..., 𝑁 }, while the lower branch extracts feature maps 𝑒 𝑖 of corresponding scales from the image x. Then, we introduce a syntax driven feature fusion module guided by an ultra-low semantic information bitstream 𝑠 in the decoder part of the U-Net, which is expounded in the following section. Initially, the feature 𝑑 𝑁 and 𝑒 𝑁 are fused to obtain the feature 𝑢 𝑁 . Then, at each layer, the corresponding layer's feature 𝑢 𝑖+1 and 𝑒 𝑖 are adaptively fused to produce 𝑢 𝑖 , which can The entire framework of our proposed method. For an image 𝑥 to be encoded, we first perform lossy compression using a standard end-to-end image compression network, resulting in an output degraded image x. Then, we extract a syntax vector from the original image 𝑥 using a syntax generator. This syntax vector is then used to guide the fusion of consistency features 𝑒 and diffusion features 𝑑 in a Consistency Guided Diffusion Model with Neural Syntax. After a complete diffusion process, we obtain a higher-quality reconstructed image 𝑥 0 . The consistent guidance architecture, neural syntax driven mechanism lead the diffusion model to stably reconstruct high-quality images, making the final output excellent in terms of perception and fidelity.be expressed as:EQUATIONwhere SFF i means the syntax driven feature fusion module and 𝑠 is a compact syntax vector. This process is repeated layer by layer, with adaptive fusion and upsampling of features performed at each step, ultimately generating the predicted output noise. By taking this approach, during the denoising diffusion process of the diffusion post-processing model, we consistently inject a constant guidance feature derived from the degraded image, guiding its inference process to stay close to the conditional distribution x. This approach enables the final output to enhance the perceptual quality while maintaining the similarity to the original image, achieving a better trade-off between fidelity and perceptual quality.As we mentioned in section 3.1, the target optimized probability distribution of the post-processing module 𝑝 (𝑥 | x) is ambiguous. To mitigate the ambiguity to ensure the reconstructed image 𝑥 0 similar to original image 𝑥, we proposed to send a compact syntax vector extracted from the original image 𝑥 and decoded image x which costs ultra-low bitstream to provide the syntax information of the original image. Based on this idea, inspired by [53] , we encode the syntax information of the original image into a compact and discrete one-dimensional vector by a syntax generator which serve as a dynamic convolution kernel in the syntax driven feature fusion module. By decoding the syntax vector into dynamic convolution kernels and performing convolution operations on the features to be fused, syntax information is transmitted in a neural representationlike manner. The structure of the syntax generator module and syntax driven feature fusion module is illustrated in Fig. 3 . Syntax Generator. The syntax generator's structure follows the design of [53] , which contains a multi-scale network on the basis of hyper-priors entropy model [4, 23] . During syntax extraction, the features at each scale are globally average pooled and concatenate to a compact one-dimensional vector. This approach effectively utilizes multi-scale information while ensuring global consistency of the semantic information. Syntax Driven Feature Fusion. The syntax driven feature fusion module takes the features 𝑑 𝑖 , 𝑒 𝑖 , the obtained syntax vector 𝑠, and the timestamp 𝑡 of the current step as inputs. After getting the syntax vector 𝑠 and timestamp 𝑡, we concatenate them and utilize a fully connected network to map them to two convolutional kernels 𝑊 𝑖 𝑒 , 𝑊 𝑖 𝑑 . These two sample-adaptive dynamic convolutional kernels separately perform convolution on the two input features, achieving adaptive fusion of features at each layer:EQUATIONwhere * denotes convolution. Since the semantic vector used for generating these two convolutions is highly dependent on the input original image, the fusion process can effectively capture the characteristics of the original image, enabling the more adaptive fusion of the two streams of features during the generation process to obtain a better reconstructed image.Furthermore, we propose that due to the excellent sample adaptability of our method, a more refined design during the inference process can more fully tap into the performance potential of our proposed method and achieve better performance. Specifically, during the inference process, we employ a globally uniform boundary control strategy and a sample-adaptive continuous online optimization mechanism for different resolutions and styles of images to be compressed, respectively, to further enhance the performance.Globally Uniform Boundary Control. During model training, we used patches of a fixed size. However, directly feeding images of different resolutions into the diffusion model during inference can cause performance degradation due to distribution discrepancies. Therefore, we chose to use a tiling approach to adapt to images of arbitrary resolutions. To mitigate the potential block artifacts that may arise from piece-by-piece tiling, we employ the following two strategies. Firstly, we overlap the patches with surrounding ones to a certain extent, so that the pixels at the edge position are predicted by multiple patches, which makes the transition of the edge position smoother.In addition, we observe that the initial noise 𝑥 𝑇 of the diffusion model can be viewed as the boundary condition of the diffusion ordinary differential equation, which affects the stylistic characteristics of the sampled images [36] . Recognizing that a complete image should exhibit consistent stylistic features, we set the initial noise for all patches to a fixed distribution. This ensures consistent image style, further mitigates block artifacts, and enhances the performance of encoding and decoding images at arbitrary resolutions. Specifically, for a given patch size, we commence by randomly selecting a boundary condition (just a Gaussian noise). Subsequently, we maintain this noise as a uniform boundary condition and tile it across the entire image, resulting in a boundary condition 𝑥 𝑇 , that spans the entire image. Using 𝑥 𝑇 as the starting, we then employ our CGDM to initiate the diffusion process. Through this process, we ultimately reconstruct a high-quality image, X0, that exhibits a stable and consistent style. Sample-Adaptive Continuous Online Optimization. Similar to [35, 53] , the approach of using a syntax generator to extract global syntax information from images inherently brings the potential for online optimization in the encoding phase during inference. This process is analogous to the mode decision process in traditional hybrid coding frameworks, where the best mode is selected from a discrete set of candidates. However, employing iterative optimization allows for continuous selection of the best option from an infinite set, greatly enhancing the flexibility of this online optimization strategy.Specifically, during the inference process for each image, we iteratively optimize the encoder parameters of the syntax generator on randomly selected patches. This enables the generator to produce syntax vectors that more closely align with the image's semantics. For the iterative optimization process during inference, we set an optimization objective consistent with the fine-training process, which is detailed in the next section.Following the training strategy proposed by [54] , we use a coarseto-fine two-stage training strategy. Training at the coarse level aims to train the diffusion model to constrain noise, while training at the fine level further optimizes the diffusion model to further enhance the model performance by constraining the sampled clean images with fixed steps and corresponding ground truth ones.Our coarse training is analogous to existing conditional diffusion models, which aim to estimate noise. The difference is, that we introduce an additional bitrate control term to constrain the bit cost of syntax vectors. Consequently, the loss function for the coarse training is as follows:EQUATIONwhere 𝜇 𝜃 denotes our noise prediction network CGDM, 𝜖 𝑡 means the adding noise at step t in the forward process, 𝑅 means the We consider both perceptual (LPIPS, DISTS, FID) and distortion metrics (PSNR, VIF, MS-SSIM). The upper 2 rows(black frame) are the performance on Kodak datasets and the lower 2 rows (blue frame) are on CLIC professional dataset.bitrate and 𝜆 𝑐 is the hyper-parameter to trade-off between rate and distortion. During the fine training stage, we fixed the sampling strategy to the 9-step DDIM sampling and imposed constraints on the generated sampled images x0 to compensate for the unsatisfactory results from the coarse training. The loss function we used for the constraints is as follows:EQUATIONwhere L 𝑑 denotes the distortion loss and we use L1 loss during training, L 𝑝 denotes the perception loss and we utilize three kinds of perceptual objective functions DISTS [10] , Alex-based [31] and VGG-based [47] LPIPS [60] in the experiments. 𝜆 𝑑 , 𝜆 𝑝 and 𝜆 𝑓 are the hyper-parameters.  
experiments 4.1 implementation: 
 Network Implementation. We implement our diffusion model based on the architecture of [14] with fewer parameters. In addition, in order to further reduce the video memory consumption, we also removed the self-attention module. The detailed structure and hyperparameters of the networks are shown in the supplementary material. Training Details. We utilize the DIV2K [1] dataset as our training dataset, which comprises 800 natural images with an average resolution of 2K. To enable our model to adapt to images of various resolutions, we perform downsampling on the images to half their original resolution, serving as an augmentation of the training data. During the training process, we randomly crop 256 × 256 patches from each image.Our training process uses the Adam optimizer [27] and the learning rate is set to 1 × 10 -4 . We trained 6 models with different Inference Details. During inference, the patch size used is 256 × 256, and the overlap range is 8 pixels close to the edge. Our method applies the continuous mode decision on inference. For each image, based on the pre-trained network weights, we additionally employ the Adam optimizer with a learning rate 5 × 10 -5 to finetune the encoder for 250 iterations, and the optimization target is the same as L 𝑓 𝑖𝑛𝑒 . Evaluation Protocol. We evaluated our method on the Kodak image dataset [30] and the professional subset of the CLIC validation dataset [50] . The Kodak image dataset consists of 24 images, To demonstrate the superiority of our method in terms of distortion and perceptual quality, we utilized a set of diverse metrics. For distortion, we employ PSNR, VIF [46] , and MS-SSIM [55] . And for perceptual quality, we used VGG-based [47] LPIPS [60] , DISTS [10] , and FID [20] . The R-D curves and BD-rate [16] on different evaluation metrics are illustrated to compare different methods and settings.We compared our method with existing conventional transformbased methods BPG [5] , end-to-end learning-based image compression methods optimized for MSE like ELIC [18] , and image compression methods optimized for perceptual quality including HiFiC [38] , ILLM [42] , and CDC [56] . Fig. 4 presents the R-D curves of various metrics on the CLIC and Kodak datasets for our proposed method and comparison methods. Evidently, our approach demonstrates superior performance across different perceptual measures compared to other perceptual image compression methods, meanwhile achieving a more favorable distortion effect. Among the comparison methods, only CDC surpasses our approach in terms of the perception metrics. However, its performance in the distortion metrics was significantly inferior to other methods, resulting in an overall performance that remained below ours. This underscores the excellent balance our method achieves between fidelity and perceptual quality, highlighting its superiority.To provide a more intuitive comparison of the overall performance of our method with other benchmark methods across all evaluation metrics, we computed the BD-rate [16] for each indicator. Using our method as the anchor, Table 1 presents the average BD-rate achieved by each method across all distortion and perception metrics on both CLIC and Kodak datasets anchored on our method. It is evident from the table that the overall performance of all other methods under all evaluation indicators falls below that of our proposed approach.To further underscore the perceptual quality of our results, we present several illustrative cases comparing different image compression methods in Fig. 1 , 5 and 6. In Fig. 5 , it is clearly observed that traditional compression methods such as BPG [5] and MSEoptimized compression methods like ELIC [18] produce overly smooth images with significant loss of detail information. Among the perception-oriented optimization methods, ILLM [42] introduces numerous continuous and repetitive artifacts in the decoded images, and the details in HiFiC [38] are not as clear as those in our method though its bitrate is much higher than ours. Fig. 6 shows the results on high-resolution images, whose subjective performance is generally consistent with that on the Kodak. Evidently, the reconstructed images using our method exhibit richer visual details, and less artifact while utilizing fewer or comparable bits.We conduct extensive ablation studies for our proposed network architecture on the Kodak dataset. By replacing the syntax driven feature fusion module with direct element-wise addition, the model fuses information directly without syntax guided adaptive fusion(w/o SFF). By replacing the consistent boundary with random noise and inferring by pre-trained model parameters without online finetuning, the optimization during inference is moved (w/o Infer. Optim.). We do the above substitutions in turn and observe a performance drop as Table 2 shows, though the model sizes are kept almost the same. Hence, all of the components in our design contribute to performance improvement.  
conclusion: 
 In this work, a novel consistency guided diffusion model with neural syntax is proposed, introducing a diffusion model for perceptual image compression. The consistency guidance architecture, neural syntax driven mechanism and inference time optimization strategy lead the diffusion model to stably reconstruct high-quality images, making the final output excellent in terms of perception and fidelity.Experimental evaluation shows the superiority of our proposed methods. 
 
 Appendix:  


Decision: The paper proposed a novel image compression pipeline composed of an end-to-end autoencoder plus a consistency-guided diffusion model, which works as a post-processing step.  Although the idea of using diffusion as post-processing for an autoencoder-like image compression framework is not completely new, the paper advances the state-of-the-art by considering the ambiguity of the existing diffusion-based and using a syntax-driven feature fusion strategy.  During the review phase, the reviewers raised concerns with regard to the novelty of the work (since there are other works on diffusion as post-processing for image compression), the computational complexity of the model, the lack of a further discussion of related work, more comparison (in particular with HFD), and lack of subjective study.  During the rebuttal, the authors answered most of the concerns, provided a small-scale subjective study, and compared their work with HFD regarding BD-Rate, showing better performance.

Overall, although the reviewers do not fully agree with the ratings, the paper can bring an interesting discussion to the conference.

Reviews: review: paper_summary: paper_summary: This paper proposes a diffusion based post-processing network to enhance the perceptual quality for the compressed images, which are coded with end-to-end image codec.
This paper proposes consistency guided diffusion model (CGDM) to improve the quality of the decoded image, preserving the style-like information with the original image. 
Moreover, this paper also proposes a syntax driven feature fusion strategy to provide prior information of the original image, alleviating the ambiguity of the processed image. 
Based on the experimental results, performance improvements could be observed on various test datasets and metrics.
strength_weakness: strength_weakness: This paper considers the ambiguity of the existing diffusion based image post-processing model, which only make use of the information from distorted images. The CGDM and syntax driven feature fusion strategy are proposed to tackle this problem. Performance improvements could be observed.
limitations: limitations: 1. In table 1, there are various BD-rate numbers, evaluated under various metrics (various metrics for both distortion and perception). There are some very large numbers and the BD-rates of the distortion and perception are averaged directly. It could be more reasonable to provide the BD-rate results for every dataset under specific evaluation metric, for better understanding. 
2. What's the computational complexity of the proposed model, compared with the learned codec baselines in this paper.
3. How the BD-rate is calculated for some models in table 1, since there are only 3 performance points for some methods (HIFIC and CDC), according to figure 4. Generally, at least, 4 points are required to calculate the BD-rate.
4. Regarding ablation study, it is interesting to compare the performance of the detailed techniques in "Inference Time Optimization", especially the performance of "Sample-Adaptive Continuous Online Optimization".
paper_summary: paper_summary: This manuscript proposes a strategy to reduce distortion introduced in lossy image compression trough neural approaches (e.g., autoencoders) based on duffision models. Standard diffusion models suffers of two issues that may augment the degradation, hence, the main contribution of the manuscript lies in Consistency Guided Diffusione Models (CGDM) and Syntax driven feature fusion (SFF) to mitigate such problems.
strength_weakness: strength_weakness: * The paper is well-written and easy to read, while the proposed approach is well-motivated.

* Experiments section presents a quantitative and qualitative comparison which suggest the proposed method is promising. Such experiments are conducted on two popular state-of-art dataset, i.e. Kodak and CLIC.
limitations: limitations: * The manuscript focuses on image compression, which means it presents a unimodal characteristics.

* The manuscript does not consider the recent INR-based method for image compression which should be cited and also taking in consideration for enforcing the comparison section. Here, the most recent papers:

    * Y. Strümpler, J. Postels, R. Yang, L. V. Gool, and F. Tombari, “Implicit neural representations for image compression,” European Conference on Computer Vision. Springer, 2022.

    * E. Dupont, H. Loya, M. Alizadeh, A. Golinski, Y. W. Teh, and A. Doucet, “COIN++: Neural compression across modalities,” Transactions on Machine Learning Research, 2022.

    * L. Catania and D. Allegra, “Nif: A fast implicit image compression with bottleneck layers and modulated sinusoidal activations,” ACM International Conference on Multimedia, 2023

    * T. Ladune, P. Philippe, F. Henry, G. Clare, and T. Leguay, “Cool-chic: Coordinate-based low complexity hierarchical image codec,” International Conference on Computer Vision, 2023 



* The novelty is on the average, as the authors employ many SOTA methods for the proposed framework.

* The use of BD-Rate to prove the validity of the proposed approach should be better explained. Table 1 is not of immediate understanding.

* It is not clear if the source code will be made available after the publication. This is critical to give the opportunity to replicate the same experiments and allowing to properly compare with the proposed approach with new ones.

* Figure 5 and 6 are intended to show a qualitative comparison. However, both focus on a texturized area without strong edge and geometrical shape. I recommend to replace one of the Figure with something like the bottom row of the figure 4 of the supplementary material, in order to show the behavior in presence of very high frequencies.

------------
Overall, at the current state the manuscript presents some limitations but, in my opinion it is still acceptable and interesting for the ACMMM audience
paper_summary: paper_summary: The authors propose an image compression pipeline which features a novel consistency guided diffusion model postprocessing step. 
The consistency guided diffusion model 1) reduces distortion by using the initial autoencoder reconstruction as both unet-input and novel intermediate feature conditioning and 2) prevents reconstruction ambiguity and generative artifacts by injecting features from the original image via a novel syntax feature fusion strategy.
These proposed changes result in more faithful reconstructions with less artifacts compared to other autoencoder-based and generative compression codecs.
strength_weakness: strength_weakness: * The proposed syntax feature fusion module is interesting and novel. I especially like the idea to use features extracted from initial autoencoder as well as syntax information from the source image to guide the diffusion process.
* The authors perform many experiments and ablations to validate the performance of their method.
limitations: limitations: * Unconvincing results. While the proposed methods performs better in some quantitative metrics, performance in other metrics is comparable or worse. This would be ok with significantly better subjective perceptual quality reconstructions, yet the authors do not perform a user study to assess this, and the provided visual comparisons do not show significantly better improvements (at least in my opinion).
* Relatively weak related works section. Previous works HFD [1] and DIRAC [2] also have a similar diffusion post-processing strategy, and the differences to these works should be elaborated more in this section.
* Similarly, the method should also be compared with HFD [1] as reconstructions on the Kodak dataset are published by the authors (http://theis.io/hifidiff//).

[1] Hoogeboom, Emiel, et al. "High-fidelity image compression with score-based generative models." arXiv preprint arXiv:2305.18231 (2023).

[2] Ghouse, Noor Fathima Khanum Mohamed, et al. "Neural image compression with a diffusion-based decoder." (2022).