Enhanced Screen Content Image Compression: A Synergistic Approach for Structural Fidelity and Text Integrity Preservation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:24:46

Abstract: With the rapid development of video conferencing and online education applications, screen content image (SCI) compression has become increasingly crucial. Recently, deep learning techniques have made significant strides in compressing natural images, surpassing the performance of traditional standards like versatile video coding. However, directly applying these methods to SCIs is challenging due to the unique characteristics of SCIs. In this paper, we propose a synergistic approach to preserve structural fidelity and text integrity for SCIs. Firstly, external prior guidance is proposed to enhance structural fidelity and text integrity by providing global spatial attention. Then, a structural enhancement module is proposed to improve the preservation of structural information by enhanced spatial feature transform. Finally, the loss function is optimized for better compression efficiency in text regions by weighted mean square error. Experimental results show that the proposed method achieves 13.3\% BD-Rate saving compared to the baseline window attention convolutional neural networks (WACNN) on the JPEGAI, SIQAD, SCID, and MLSCID datasets on average.

Main paper: 
 related works: 
 This section first briefly reviews several popular learned image compression algorithms. Then, we describe the conditional compression, which is closely related to the method in our proposed algorithm. Finally, we talk about SCI compression, which is the research focus of our study.Learned image compression: In recent years, there has been remarkable progress in the field of learned image compression. Ball é et al. [2] first propose an end-to-end optimized model for image compression. Then, Ball é et al. [3] introduce a hyperprior architecture to effectively capture the spatial dependencies in latent features, thereby reducing spatial redundancy. With the success of auto-regressive priors in probabilistic generative models, Minnen et al. [24] introduce an auto-regressive module to improve coding efficiency. As the entropy model has great potential in optimizing image compression performance, numerous studies are working toward improving the module of entropy models [10, 14, 17, 18, 25] . Cheng et al. [10] employ discretized Gaussian mixture likelihoods, and Fu et al. [14] propose a Gaussian-Laplacian-logistic mixture model. In addition, there are some works to adjust network architecture. Liu et al. [22] propose a Non-Local Attention Module (NLAM) to capture both local and global correlations. Contrarily, Zou et al. [42] propose the WAM to further capture correlations among spatial neighboring features.Conditional compression: Conditional compression is typically employed in variable bitrate encoding, which aims to compress images at different bitrates using a single model [20] . Conditional compression is often achieved by altering the distribution of features. Cui et al. [11] introduce gain units to scale the latent representation in the channel dimension, thereby enabling adaptive continuous bitrate. Song et al. [30] utilize spatial feature transform (SFT) layers [37] to apply element-wise affine transform to intermediate features, enabling variable bitrate control and Region of Interest (ROI) encoding. Wang et al. [36] implement channel-wise scaling across all self-attention layers to achieve variable bitrate. It is worth noting that, as far as our knowledge, the previous works in this domain primarily aim at achieving variable bitrate, with no explicit exploration into the compression performance of the model. In our work, we alter the distribution of features with the external prior guidance to enhance learned SCI compression for better rate-distortion performance.Screen content image compression: The video coding standards, HEVC and VVC, have introduced techniques such as the palette mode, cross-component prediction, and transform skip to improve the compression efficiency of SCI. In existing palette coding approaches, separate palette coding is applied to the luma and chroma components. Then, Vishwanath et al. [33] consider the correlation between different components and propose crosscomponent palette prediction using luma for chroma. Ramin et al. [15] propose the Joint Cross-Component Linear Model (J-CCLM) to improve prediction efficiency by combining the conventional CCLM mode with an angular mode derived from the co-located luma block. Tang et al. [32] combine text detection with HEVC-SCC to propose a text semantic-aware encoding strategy, achieving high text semantic fidelity at low bitrate. Inspired by the transform skip technique in traditional coding, an end-to-end transform skip image compression framework is introduced by Wang et al. [35] , surpassing the SCI compression performance of the hyper-prior model in [3] . Heris et al. [19] introduce a segmentation network-based compression scheme to distinguish natural from synthetic content in SCI. Shen et al. [29] propose a pioneering entropy-efficient transfer learning module named Dec-Adapter to bridge natural image and SCI compression. However, the exploration of end-to-end SCI compression is limited, and its performance has not yet reached a level close to traditional methods. In our study, we will propose a structural fidelity and text integrity preservation SCI compression method to achieve superior performance compared to VVC. 
methods: 
 This section introduces our main idea of SCI compression for structural fidelity and text integrity preservation. Firstly, we describe the overall framework. Then, an EPG module is introduced. Finally, we discuss the SEM and the optimization of the loss function.Our proposed network architecture is shown in Fig. 1 , which is built based on the WACNN model [42] . The EPG is introduced to provide global spatial attention which is not considered in WACNN. The FEM is used to extract structural prior information of varying depths. The NESFT module is proposed to fuse the features of the FEM and the backbone convolutional layers by spatial affine transform. This is crucial for preserving the structural information of SCIs. The details of the network are described below. The main encoder 𝑔 𝑎 , built upon existing analysis transform modules, accepts the structural prior features that come from the prior branch 𝑔 𝑎,𝑝 and applies the spatial affine transform to intermedia features through the NESFT module. The main encoder 𝑔 𝑎 and the prior branch 𝑔 𝑎,𝑝 take the image 𝑥 and the external prior 𝑔 ′ as inputs to generate the latent feature 𝑦, as formulated below: where Φ 1 = 𝑔 𝑎,𝑝 (𝑥, 𝑔 ′ ).EQUATIONThe straight-through estimator (STE) strategy [5] is utilized to quantize the latent feature 𝑦, generating ŷ for subsequent entropy coding. The formula is as follows: ŷ = 𝑄 (𝑦).(EQUATIONThe hyper-encoder also performs structural prior guided spatial affine transform, as shown below:EQUATIONwhere Φ 2 = ℎ 𝑎,𝑝 (𝑦, 𝑔 ′ 1 ), with 𝑔 ′ 1 being the down-sampling of 𝑔 ′ to ensure dimensional compatibility with 𝑦.The entropy model is adopted the same as WACNN. The parameters (𝜇, 𝜎) of the Gaussian entropy model are obtained from the hyperprior ẑ to approximate the distribution of ŷ. The hyperprior ẑ retains spatial importance information. Thus, our framework does not explicitly transmit structural prior, with consideration for bit saving. It recovers an approximate structural prior 𝑤 from the hyper prior ẑ. The main decoder 𝑔 𝑠 takes the approximate structural prior 𝑤 and ŷ as inputs to reconstruct the original image, as shown in the following formula, where Φ 3 = 𝑔 𝑠,𝑝 ( ŷ, 𝑤).EQUATIONSCIs exhibit different characteristics from natural images, where simple regions may appear completely flat and complexity varies significantly between different regions. When applied to SCIs, the WAM struggles to effectively capture and process global structures, as it focuses more on details within the window. To address this challenge, we propose EPG to optimize image compression by utilizing global attention. The external priors are highly correlated with the characteristics of SCIs, ensuring the preservation of structural fidelity and text integrity. These external priors consist of basic structural prior and text position prior. Firstly, the structural information obtained by the Laplacian gradient operator is introduced as a basic structural prior. It can enhance the network's ability to preserve image edges and details, especially sharp edges and noticeable structural transition areas commonly found in SCIs. The Laplace gradient operator is defined as follows,EQUATION)We do not consider gradient direction information, as gradient magnitude is sufficient to reveal image structural information.Then, text position prior is introduced to refine the basic structural prior, aiming at preserving text integrity. Using the text detection method, we detect text regions and translate them into binary masks, with text regions denoted as 1 and non-text regions donated as 0. Employing this text mask, the structural prior of images is introduced to enhance their representation of text regions. The formula is shown as follows:EQUATIONwhere 𝑔 represents the basic structural prior, and 𝜖 is a predefined constant for fine-tuning the attention toward text regions. The revised priors enhance the contrast between text and non-text regions, aiding the network in distinguishing text regions from the rest.To generate the text region mask, we evaluate three representative text detectors, including EAST [40] , CRAFT [1] , and RDD [21] . By the evaluation results in section 4, CRAFT is chosen as our text detector.To preserve the structural information of the image, we design a SEM. Altering the distribution of features is widely utilized in conditional compression for variable bitrate coding. However, the SEM is proposed to achieve a high coding performance instead of achieving variable bitrate coding. This module integrates both FEM and NESFT modules, as shown in Fig. 2 . We craft the network architecture to endow the FEM with efficient feature extraction capabilities, enabling it to distinguish spatial structures intricately embedded within external priors. Leveraging a strategic combination of 1×1 convolution, we can enrich the network's nonlinear capabilities, significantly enhancing its capacity to handle complex external priors and affine transform relationships. We utilize NESFT to propagate the structural prior features extracted by the FEM into the backbone, instead of directly concatenating them with RGB features. This strategy ensures the efficient utilization of structural priors, augmenting feature representation in complex image regions and optimizing compression efficiency of flat regions. NESFT dynamically generates parameter pairs (𝛾, 𝛽) based on the output of FEM, subsequently employing them to execute spatial affine transform on the features. The feature transform formula is shown as follows:EQUATIONwhere F represents the intermediate features. As shown in Fig. 3 , the spatial feature transform (SFT) and conditional block (CB) proposed by Song et al. [30] fails to effectively extract spatial structural information from structural priors, resulting in noticeable stripe artifacts in reconstructed images. We visualize the channel with the highest entropy for WAM and our method in Fig. 4 . In the third column, it's obvious that our method can allocate fewer bits in simple regions and more bits in complex regions. However, WAM is constrained by the window-based mechanism, which allocates bits to flat regions. Moreover, the reconstructed images from our approach show clearer structure and fewer artifacts. To further enhance the compression efficiency of the network for SCIs, we make optimization to the loss function. The loss function for lossy image compression aims to achieve a balance between rate and distortion, which is defined as follows,EQUATIONwhere 𝑅 represents the bitrate, 𝐷 represents the distortion, and 𝜆 is a Lagrange multiplier that determines the trade-off between the bitrate and distortion. For text-priority compression demands, we introduce a refined loss function as follows,EQUATION)where Λ = (𝜆 𝑖 ) (𝑖=1:𝑁 ) is the Lagrange multiplier matrix. It is no longer a single constant but a set of Lagrange multipliers dynamically adjusted by a text region mask 𝑀 = (𝑀 𝑖 ) (𝑖=1:𝑁 ) . Each element 𝜆 𝑖 is determined by a corresponding 𝑀 𝑖 through a function mapping as follows, Λ = (𝑀/5 + 0.9) × 𝜆. (10) For the larger 𝑀 𝑖 , the mapping aims to assign greater 𝜆 𝑖 for the corresponding pixel 𝑥 𝑖 . Our designed loss function enables explicit spatial bit allocation in image compression based on 𝑀, giving priority to the compression quality of text regions. The estimation of rate 𝑅 requires an entropy model 𝑃 conditioned on 𝑀. The entropy model outputs the likelihood of latent representation ŷ. This allows for dynamic adjustments based on the local features of the image, achieving targeted compression for text regions instead of applying a single strategy to the entire image. The MS-SSIM used as the distortion item in image compression will cause poor text reconstructions [23] . Thus, weighted mean squared error based on the text region mask is chosen as our metric for distortion evaluation. The formula is shown as follows, L = -𝑙𝑜𝑔𝑃 ( ŷ | 𝑀) + 𝑁 ∑︁ 𝑖=1 𝜆 𝑖 (𝑥 𝑖 -𝑥 ′ 𝑖 ) 2 𝑁 (11) 
training details: 
 The training details and the hyperparameter settings are described in this subsection. Our hardware platform is Intel Xeon E5-2690 V3 @2.60GHz CPU with 64GB memory and two Titan X GPUs. The method is implemented on the CompressAI platform [4] . The 𝜆 belongs to {0.0018, 0.0035, 0.0067, 0.0130, 0.025}. For the training dataset, we utilize the JPEGAI dataset and 300,000 images randomly selected from the OpenImages dataset. Images are randomly cropped to a size of 256×256, with a batch size of 8. For entropy coding, we utilize the Range Asymmetric Numeral System [13] Table 1 : Comparison with existing compression methods on BD-Rate (%). The BD-Rate of BPG, VVC, cheng2020 [10] , ms2020-2-Decoder [19] , Dec-Adapter [29] , WACNN [42] and our method are provided for comparison. The average performance is calculated by the utilized datasets, and tabulated in the last column. A smaller valuer is more effective. We compute the peak signal-to-noise ratio (PSNR) and bits per pixel (BPP) for each image in the JPEGAI, SIQAD, and SCID datasets, and average the values to plot the rate-distortion curves as shown in Fig. 5 , Fig. 6 , and Fig. 7 . The coding performance is measured by BD-Rate and negative values represent performance gains. We retrain the WACNN with the same training strategy. The images in JPEGAI are cropped into sizes less than 1920×1080 for inference. For Dec-Adapter [29] and Zou [41] , we use the results from the Dec-Adapter [29] for comparison directly. For cheng2020 [10] , we retrain it with the same training strategy as our method. For ms2020-2-Decoder [19] , we evaluate the performance of the publicly available model. The proposed method is significantly better than the baseline method. The WACNN results are obtained by our retrained models. Our method achieves an improvement of nearly 0.3dB on PSNR compared to the baseline on the JPEGAI dataset. Additionally, we perform a comparison with the learned methods and other conventional codecs. The results are presented in Table 1 .We assess the performance of VVC under the all intra (AI) configuration. The version is VTM-10.0. It shows that our method achieves 13.3% BD-Rate saving on average. The snapshots of reconstructed images using different compression methods at similar bitrates are shown in Fig. 8 . Under approximately the same compression rate, the algorithm proposed in this paper preserves finer details and clearer structures over existing methods. It can be observed that our method restores more realistic character edges and thinner edges, demonstrating higher visual quality.We perform the ablation study on the effectiveness of different methods proposed in Section 3. Our results are evaluated on the SIQAD dataset. Effectiveness of SEM. As shown in Fig. 9 , it is evident that the external prior in our work is more abundant and complex than the previous prior. Hence, improvements of SFT & CB [30] are necessary to adapt to the difference. To show the effectiveness of SEM, we compare the SEM with SFT & CB. The results are tabulated in Table 2 . The SFT & CB fails to generate efficient structural features leading to the quality loss in the table and the appearance of artifacts in Fig 3 . In contrast, the SEM can handle complex external priors well, providing parameters for feature transform and leading to better performance. Effectiveness of structural prior. To evaluate the performance of EPG, we use uniform quality maps (W/O structural prior) [30] for comparison. The results are tabulated in Table 3 . It brings improvement in compression quality for SCIs due to the enhancement of latent features by generating structural prior. Effectiveness of text position prior. We compare our model with the W/O text position prior case. Table 6 shows the comparison results and Fig. 10 presents qualitative comparison results. As shown in Fig. 10 , the text position prior leads to better integrity preservation in character "e". These results indicate that the text region mask effectively preserves text integrity and improves text region compression performance.Comparison of different text detectors. The text region mask is decided by the text detector. Different text detectors are evaluated by the compression performance including EAST [40] , CRAFT [1] , and RRD [21] . As shown in Table 5 , CRAFT shows the best performance. This is because of the precise text region mask provided by CRAFT. The CRAFT obtains 1.2% BD-Rate saving compared with EAST.Effectiveness of loss metric. To verify the effectiveness of the proposed loss metric, we compare our WMSE loss function with MSE. Since ASIQE [16] is consistent with the visual quality of the image text context [32] , we measure objective quality with ASIQE in this experiment. The compression performance on the SIQAD dataset is tabulated in Table 6 . The model optimized with the proposed loss metric brings 21.3% BD-Rate (ASIQE) saving. 
conclusion: 
 In this work, we propose a learned compression framework for SCIs. Our motivation is from the different characteristics between SCIs and natural images. To help preserve the structural fidelity and text integrity, the EPG is proposed which aims to extract global spatial attention for better SCI compression. Then, the SEM is proposed, which includes FEM and NESFT. It shows more reasonable bit allocation and a better visual effect than the SFT & CB method. Finally, a text region mask-guided rate distortion loss function is proposed to 
: 
 entropy coding [28] . The representative standards are JPEG [34] , JPEG2000 [12] , intra frame coding of H.264/HEVC/VVC [7, 31, 38] . These standards are handcraft designed, which no longer meet the increasing requirements of image compression. Motivated by this, learned-based image compression models are proposed and show better rate-distortion performance than conventional methods on metrics of both peak signal-to-noise ratio (PSNR) and multi-scalestructural similarity index (MS-SSIM) for natural images [42] . However, little attention has been paid to screen content image (SCI) compression, where the SCIs are spread everywhere across the Internet.SCIs typically comprise text, graphics, and camera-captured natural images. Computer-generated text and graphics account for more than 90% of SCIs. In contrast to natural images, SCIs are characterized by their sharp edges, limited color palette, high contrast, noise-free, and significantly different region complexity [26] . Because of these distinctive attributes, conventional compression algorithms designed primarily for natural images are less effective when applied to SCIs. A significant loss of high-frequency information occurs during traditional compression, which seriously damages the quality of the reconstructed screen content due to edge and text blurring. As the human eye is sensitive to edge and text information, maintaining the high-quality compression of these details is crucial.Recently, numerous schemes have emerged to improve the coding efficiency of SCIs. In the latest generation of coding standards, such as VVC, AV1 [9] , and AVS3 [39] , numerous technologies have been designed specifically for SCI. For example, VVC adopts intrablock copy (IBC) to handle regions with regular patterns and repetitions in SCIs. The use of palette mode to depict a coding block is based on the simplicity of color schemes in SCIs, contrasting with the complex and diverse colors found in natural images. Furthermore, VVC employs transform skip, adaptive color transform, and block-based differential pulse code modulation to adapt to the characteristics of SCIs [27] . Through these optimizations, VVC achieves a notable 33.22% Bjøntegaard-Delta Rate (BD-Rate) [6] reduction when compressing SCIs [26] . Currently, deep learning-based image compression methods show excellent performance. However, it is essential to highlight that existing end-to-end image compression solutions are designed for natural images, with limited attention paid to SCI compression. Besides, SCIs show different characteristics compared to natural images. Therefore, algorithms for natural images do not perform optimally on SCIs. To resolve this problem, some researchers proposed learned SCI compression. Wang et al. [35] propose a transform skip-inspired end-to-end SCI compression scheme. However, this scheme does not exhibit performance superiority when compared to VVC. Heris et al. [19] propose a segmentation network-based compression scheme. However, the framework does not consider the SCI characteristics when designing the network architecture.In this paper, we propose a synergistic learned compression approach for SCI. This work aims to address the following three challenges: 1) resolving the global spatial structural information loss which is not considered in the window attention module (WAM), 2) resolving the improper bit allocation for SCI compression, 3) minimizing text region blurring in the reconstructed image. Experimental results show that the proposed method achieves 13.3% BD-Rate saving compared to the baseline window attention convolutional neural networks (WACNN) on the JPEGAI, SIQAD, SCID, and MLSCID datasets on average. The main contributions of our work are summarized as follows:( The rest of this paper is organized as follows. Section 2 provides a review of related works on end-to-end image compression, conditional compression, and SCI compression. Section 3 presents the proposed synergistic approach for structural fidelity and text integrity preservation. The experiment results are shown in Section 4, followed by the conclusion in Section 5. 
 
 Appendix:  


Decision: This paper proposed a novel image compression approach for screen content.  All reviewers appreciate the novelty of the work, but share the concerns around complexity and performance of the proposed approach.  Implementation details and more performance analysis would be helpful.

Reviews: review: paper_summary: paper_summary: 1. An External Priori guidance (EPG) module is proposed to provide global spatial attention by combining basic structure priors and text position priors to enhance structure fidelity and text integrity.
2. Structure enhancement module (SEM) is proposed, including feature extraction module (FEM) and nonlinear enhanced spatial feature transform (NESFT). FEM extracts prior structure information, and NESFT fuses it into the backbone network to improve the retention of structure information.
3. In the loss function, the weighted mean square error (WMSE) is used instead of MSE to give a higher weight to the text area, which improves the compression efficiency of the text area.
strength_weakness: strength_weakness: 1. This work focuses on the important but under-addressed area of image compression for screen content and proposes a novel approach to end-to-end structural fidelity and text integrity retention.
2. By introducing external prior guidance and structure enhancement module, the existing method effectively solves the problem of structural information and text detail loss when processing screen content images, and significantly improves the compression performance.
3. EPG module uses basic structure and text position prior to provide global spatial attention; The SEM module is designed with reasonable feature extraction and nonlinear enhancement to realize effective use of prior structure information.
limitations: limitations: 1. The paper lacks specific implementation details of some key modules, such as the specific network structure and parameter Settings of FEM and NESFT, which may affect the repeatability of the work.
2. The paper does not analyze the computational complexity and model size of the proposed method, which are crucial for practical applications.
paper_summary: paper_summary: This paper proposes an enhanced screen content image (SCI) compression approach, in which the structural fidelity and text integrity are preserved. Considering the unique characteristics of SCIs, this paper emphasizes optimizing the bit allocation for SCI compression and minimizing text region blurring problem. Experimental results show that the proposed method surpasses the performance of other baselines on BD-Rate saving.
strength_weakness: strength_weakness: 1. The external prior guidance (EPG) module is proposed to enhance structural fidelity and text integrity by providing global spatial attention. The Laplace gradient operator and text detectors are employed to formulate the external priors.
2. A structural enhancement module (SEM) is introduced to improve the preservation of structural information by enhanced spatial feature transform. In SEM, the feature extraction module (FEM) and nonlinear enhancement spatial feature transform (NESFT) is integrated to distinguish the structural prior features and propagate them into the backbone.
3. The weighted mean square error (WMSE) loss function is proposed for better compression efficiency in text regions.
limitations: limitations: The SEM incurs additional computational overhead, and the paper does not consider the computational complexity in the experiments.
The appearance of names at the top of the pages suggests there may be an issue with the template configuration.
paper_summary: paper_summary: The paper explores a global structural and textural prior based screen content image compression method. The proposed method consists of two part, an external prior guidance module and a structural enhancement module. Based on promising design of the network, the proposed method can get 13.3% BD-rate gain over WACNN in CVPR’22.

Novelty lies in several aspects. Firstly, the method proposed an external prior guidance module, which use laplace gradient operator to detect the region of text and use the text region as prior to fruther refine the basic structual prior and guide the distinguishment of text region and non-text region. Secondly, the paper proposed a structural enhancement module which consists of structural enhancement part and transformation part, with structural enhancement part further highlighting the textual part in image and using transformation part to extract these high frequency textual part to improve the total coding performance.
strength_weakness: strength_weakness: The paper is well-written and well-organized. How to efficiently predict and reconstruct the text part with strong structure and texture has always been an important part of screen content image compression. This paper proposes a module to distinguish between text and non-text regions and a module to extract structural and texture information, and improve the compression performance by emphasizing the compression of such information. In addition to this, it gives a clear theoretical foundation as well as a presentation of the relevant details thus supporting the validity of the method proposed in this paper.
limitations: limitations: First, there are some formatting problems with this article, such as “ACM reference format...” should be removed from the text. Secondly, I'm curious about the performance of this approach compared to enhanced compression model (ECM) with the screen content encoding tool fully enabled, as ECM makes a number of interesting enhancements to the screen content tools compared to VVC. It would be more beneficial to support the proposed method if some comparison of newer and better screen content compression methods could be given.