Frequency-Aware Transformer for Learned  Image Compression
Authors: Han Li
Shaohui Li
Wenrui Dai
Chenglin Li
Junni Zou
Hongkai Xiong
Venue: ICLR 2024 poster (2024)
Keywords: learned image compression
frequency-aware
transformer
entropy model
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Learned image compression (LIC) has gained traction as an effective solution for image storage and transmission in recent years. However, existing LIC methods are redundant in latent representation due to limitations in capturing anisotropic frequency components and preserving directional details. To overcome these challenges, we propose a novel frequency-aware transformer (FAT) block that for the first time achieves multiscale directional ananlysis for LIC. The FAT block comprises frequency-decomposition window attention (FDWA) modules to capture multiscale and directional frequency components of natural images. Additionally, we introduce frequency-modulation feed-forward network (FMFFN) to adaptively modulate different frequency components, improving rate-distortion performance. Furthermore, we present a transformer-based channel-wise autoregressive (T-CA) model that effectively exploits channel dependencies. Experiments show that our method achieves state-of-the-art rate-distortion performance compared to existing LIC methods, and evidently outperforms latest standardized codec VTM-12.1 by 14.5\%, 15.1\%, 13.0\% in BD-rate on the Kodak, Tecnick, and CLIC datasets.

Published as a conference paper at ICLR 2024
FREQUENCY-AWARE TRANSFORMER FOR LEARNED IMAGE COMPRESSION
Han Li', Shaohui Li Wenrui Dai' Chenglin Li', .unnii Zou', Ho.nkaai Xiong' Shanghai Jiao Tong University. TTsinghua Shenzhen International Graduate School, Tsinghua University {qingshi 9974, daiwenrut lel 985, zou junni, xionghengkai) ) jtu.edu 1i lishaohui 8sz. t tsinghua.edu. edu.
ABSTRACT
Learned image compression (LIC) has gained traction as effective solution for image storage and transmission in recent years. However, existing LIC methods redundant in latent representation due to limitations in capturing anisotropic frequency components and preserving directional details. To overcome these chal- lenges, we propose novel frequency-aware transformer (FAT) block that for the first time achieves multiscale directional ananlysis for LJC. The FAT block com- prises rrequenyyddecompesiie
window attention (FDWA) modules to capture multiscale and directional frequency components of natural images. Addition- ally, we introduce frequency-modue modulation feed-forward network (FMFFN) adap- tively modulate different frequency components, improving rate-diston distorttoiion perfor- mance. Furthermore, we present a transformer-basee based channel-wis autoregressive (T-CA) model that effectively exploits channel dependencies, Experiments show that our method achieves Mate-of-the-art rate-distortion performance compared existing LIC methods, and evidently outperforms latest standardized codec VTM-12.1 1 by 14.5%, 15.1%, 13.0% in BD-rate on the Kodak, Tecnick, and CLIC datasets. Code will be releaset at ht / / thub. com ngsh 9974/ ICLR2024-FTIC
INTRODUCTION
Learned image compression (LIC) models have emerged as promising solution to image storage and transmission and outperform traditional codecs in rate-distortion (R-D) metrics. Theoretically, LIC leverages the nonlinear transforms to alternatively enable multi-dimensional quantizer with adaptive quantization cells, and consequently, exceeds the traditional transform coding schemes with restricted constructions. Early LIC models usually adopt convolutional neural networks (CNNs) to achieve nonlinear analysis and synthesis transforms (Ballé et al. [0188] Minnen et al.) 2018). How- ever, the local receptive fields of CNNs limit the representative ability and render redundant latent representations. Recent works (Zou et al. 2022 Zhu et al. 2022 Liu et al., (023) employ attention modules transformers to capture the non-local spatial rel relationship for better R-D performance.
Despite the success of transformers, there lacks an interpretation on the frequency characteristics of natural images in LIC models, which plays crucial role in conventional image representation. For example, wavelet analysis allows the decomposition of an image into multiscale subbands, revealing details and structural information within different frequency ranges (Mallat 1989). Consequential multiscale geometric analysis (MGA) tools further extend wavelet transforms with improved di- rectional analysis realized by frequency partitioning (Candès & Donoho 2000), space- fnce-fre
tiling (Donoho & Huo (2002), and -unlti-directoonal filters (Do & Vetterli 2005). The elegant math-
ematical foundations and wide applications of MGA also inspire us in exploring deep learning-basee image representation with directional analysis.
In this paper, we propose a frequency-aware transformer for constructing the nonlinear trans- forms in LIC, which captures multiscale and directional frequency components of natural im-
"Corresponiing authors: Wenrui Dai; Shaohui Li.

---

Published as a conference paper at ICLR 2024
ages with isotropic and anisotropic window attention. Specifically, we first introduce frequency- decomposition window attention (FDWA), which comprises four types of attention modules that capture the low-frequency, sehhfrrequecy, vertical, and horizontal components. Subsequently, feed-forward Network (FMFFN) to modulate the frequency compo- nents adaptively. Furthermore, propose transformer-basee channel-wise autoregressive (T-CA) entropy model that exploits correlations across the directional components with causal masks.
develop a frequency-modulatee
To best knowledge, this paper the first achieve transformer- directional analysis in LIC. Different from handcrafted directional analysis tools in MGA, we achieve end-to-end optimization realize novel anisotropic window attention. Distinguished from existing transformer saaseoemee-bbaeed image compression methods (Lu et al., 2022; Zhu et al., [022] Liu et al.) 2023] Zafari et al.) (202)). we exploit the crucial directional information in nonlinear transforms to break the limitation of existing CNN-based and wransormer-basee LIC models. In summary, our contributions include:
We propose geguuecc-ddcompositte window attention (FDWA), which leverages di- verse window shapes to capture frequency components of natural images to achieve more efficient latent representation an end-to-end learned manner.
We develop a frequency-modulatiion modulation feed-forward network (FMFFN) that adaptively en- semble frequency components for improved R-D performance.
We present a transformer- based channel-wise autoregressive model (T-CA) for effectively modeling dependencies across frequency components.
Experiments show that our method achieves otate-of-the-art R-D performance, and outper- forms VTM-12. 12.1 by 14.5%, and 15.1%, 13.0% BD-rate on Kodak, Tecnick, and CLIC Professional Validation datasets respectively.
RELATED WORK
2.1 TRANSFORMER-BASE IMAGE COMPRESSION
Transformers Vaswani al. 2017 7) have achieved re-
Low-frequency
markable success in various computer vision tasks (Doso-
(LL-WA)
vitskiy et al. 2020 Carion et al. 2020 Li al. 2023) due to its powerful non- local model ing ability. Recently, re-
Multiscale
searchers have also incorporated transformer into learned image compression. Zhu et al. (022) first demonstrate that nonlinear transforms built on Swin-Transformer (Liu al., (22TT) can achieve superior compression efficiency compared to those built on CNNs. Liu et al. (2023) com- bine transformers and CNNs aggregate non-local and local information for more expressive nonlinear trans- forms. However, the standard window self-attention used in these works ignores the underlying different frequency components natural image, which hinders the extrac- tion of compact latent representations. To address this issue, our paper proposes a novel frequency-awarc trans- former (FAT) block.
frequeeyyy
(HH--WA) WI))
vertical (HL -WA)
Directional
Horizontal (LH+WWA)
Window
Feature
Spectrum
Figure 1: Illustration of the proposed eequeecy-decomppsitton widow atten- tion (FDWA) that realizes multiscale and directional decomposition. The first column shows diverse window shapes for capturing different frequency com- ponents, the second column visualizes the extracted features, and the third col- umn presents the Fourier spectrum of the features.
2.2 AUTOREGRESSIVE ENTROPY MODELING
Entropy modeling is crucial for learned image compres- sion models. A precise entropy model can eliminate the coding redundancy and minimize the size of compressed images. Most existing works are developed based joint autoregression and hyperprior model (Minnen et al. 2018), in which the autoregression module captures the dependencies within the latent representations and re- duces coding redundancy. The autoregression modules
2

---

Published as a conference paper at ICLR 2024
Go
rilpel
Black Merging
Block
IFFT
RAS
RSU
FMIFEN
W
FAT Illock,,
FAT Block x
Concisi
RBS
RBU
Laye: Norn
Concal
Blnck Purtitioung
Block
Block
Camebad 12
RBU
Convisl
POWA
A1
40
Norn
Hioal Spli
Cariisl
Cunvial
Canvisl
Channel Transforer-bssee
Entropy
architecture
(b) Frequency
Figure 2: Overview of the proposed Frequency-awee awarre Transformer-be measeerree- learned Image Compres- sion (FTIC) model. Multiple residual blocks with stride (RBS), residual block Upsampling (RBU), and frequency-awan transformer (FAT) blocks are employed in building the nonlinear transforms (i.e., analysis transform (-).) and synthesis transform (-)). ) For each two concatenated FAT blocks, the first employs the regular Sesquency-decompee window attention (FDWA) and the second one performs window operations.
could be classified as spatial autoregression Minnen et al. 2018) (SA), chamnel-wise autoregres- sion (Minnen & Singh 2020) (CA), and their combination (He et al. 2022). Recent works employ transformers to capture long-range dependency and improve the preciseness of entropy modeling. Qian et al. (2022) utilize global self-attention to capture long-range spatial dependency
distribu- tion estimation. Koyuncu et al. (2022) propose spatial-channel attention to fully exploit latent de- pendency and improve R-D performance. However, the heavy memory usage and dramatic compu- tational complexity render these methods impractical for real-world image compression, especially for high-resolution images. Liu et al. (2023) incorporate Swin-Trann Transformer into channel-wise au- toregressive entropy model to capture additional spatial dependency. However, the wwin-transformer increases the model size while the R-D improvement over the anchor model seems insignificant. Different from these works, the proposed T-CA concentrates improving channel-wise attention without obviously increasing additional parameters.
2.3 FREQUENCY DECOMPOSITION IN LEARNED IMAGE COMPRESSION
Traditional image codecs employ subband decomposition to decorrelate frequency components of images to achieve compact representations. Recent learned image compression models also benefit from the explicit frequency decomposition. [M et (2019 2020) introduce wavelet-like transform in LIC, but it is restricted by the lifting scheme, limiting the representation ability of network and constraining the latent space. [ao et al. (202T) propose a frequency decomposition model that manipulates the low- and high- high-frequency components in the input image separately. Zaffari et al. (2023) utilize HiLo attention (Pan et al.) (2022) to disentangle low- and high-frequenee components. However, the global self-attention in their approach poses computational challenges when dealing with large input images and directional decomposition can not be achieved. In this work, we propose FDWA with diverse window size capture the multiscale and directional frequency components by transformer simultaneously.
3
METHODS
3.1
OVERVIEW
Figure illustrates the architecture of the proposed Frequency-awar Transformer-basee learned Image Compression (FTIC) model. Given a raw image æ. the analysis transform (.(.) maps it to latent representation y. Then, quantization operator Q(-) discretizes y ÿ. タ is subsequently
3

---

Published as a conference paper at ICLR 2024
losslessly encoded by a range coder. Here, we assume that follows Guassian distribution of which the parameters (i.e., mean μ and scale σ) are predicted by the proposed transformer-bast channel-wise autoregressive (T-CA) entropy model, as presented in Figure 3] Following the settings in the original channel-wise autoregressive entropy model (Minnen & Singh 2020), we divide y into ny even slices {y), y2..... Yn. and feed these slices TT-A model. Therefore, the encoded slices can provide powerful contextual information when encoding subsequent slices. Formally, the forward process of T-CA model holds as below. The hyperprior & is obtained via a pair of hyper- encoder (-) and hyper-decoder h-)-))
= ha (y) 2 = Q()), Φ = (z)
Moreover, the T-CA entropy model outputs the estimated parameters of the Gaussian distributions:
ri M4 o; = ( 9es), S < flas
where μi and σ; are mean and scale values of Gaussian distributions, and r, the predicted latent residual that reduces quantization errors of ys A refined latent ÿ could be obtained by summing up the latent ý and latent residual r, i... 1 = D + r with r Concat(r,, T2,,.., Tn,). Consequently, the final & is reconstructed by feeding refined latent ÿ to the synthesis transform g=(-). i.e., t go (g).
To train our FTIC model, we formulate the problem Lagrangian multiplier-based R-D optimiza- tion, in which the loss function is defined as:
C C= + R(2)+ 2))
where a Lagrangian multiplier À controls the trade-off between rate and distortion. Different values are corresponding to different bitrates. D(x, .) denotes the distortion term between the raw image a and reconstructed image á. (#)) denote the bitrates of latents ÿ and 2. Please refer to Appendix [A] for the detailed model architecture.
3.2 FREQUENCY-AWARE TRANSFORMER BLOCK
We aim at building efficient frequency decomposition within the end-to-end optimized image com- pression framework. To this end, we propose a novel frequency-aware transformer (FAT) block, which achieves na've multiscale and directional decomposition. Specifically, the FAT block exploits frequency-decomposed window attention (FDWA) mechanism, which decomposes the input image into four components (i.e., low-frequency, high-frequeecy, vertical, and horizontal components pre- modulation feed-forward network (FMFFN) modulates the decomposed components and eliminate the potential redundancy across frequency components. The following subsections elaborates the implementations of FDWA and FMFFN.
sented in Figure D. Then a
3.2.1 FREQUEACYY-DECOMPOSS WINDOW ATTENTION
Recent studies (Park & Kim 2022; Pan et al. 2022) have demonstrated that a typical self-attention in fact a low-pass filter, and local window attention with smaller window size captures fine- grained high-frequency information. Therefore, leveraging windows of varying sizes enables the extraction of multiscale frequency components. However, the square windows employed in existing self-attention are inefficient for capturing directional frequency information due to their isotropic characteristic.
To eliminate the limitation on directional frequency decomposition, we propose frequency- decomposed window attention (FDWA) module, which performs self-attention with four diversely shaped windows in parallel. In our experiments, the sizes (height x width) of these four windows are 4s x 48, 8 x 8, 8x 4s, and x 8, with being the basic window size. These windows correspond to the low-frequency, figh-frequency. vertical, and horizontal components. From the perspective of separable two-dimensional frequency decomposition, such implementation equals to a permutation and combination of low(L)- and high(H) (freuuunerreycyy decomposition on each dimensions. Therefore, we also term the window attention (WA) achieved by these windows as LL-WA, HH-WA, HL-WA, and LH-WA.
Given the input hidden representation X E XC where H x W is the spatial resolution and C' denotes the number of channels, first linearly project it into K heads. Then we split the K
4

---

Published as a conference paper at ICLR 2024
heads evenly into four parallel groups with K/4 heads in each group (assuming K is multiple of 4), and each group performs a specific type of self-attention. Without loss of generality, we use the case of LH-WA
example.
Example: LH-WA. X is evenly partitioned into non-overlapping windows [X X X. where each window contains 4s tokens. Assuming the projected queries, keys, and values of the kth ( < k I K) head are dz-dimensional tensors, then output of the LH-WA for the kth head can be obtained follows.
X = X
X X=
Y Attention X W X'W, X'W)),
(4)
LH-WA&(X) = [ 1 ... YYY),
where X RCx4xx and M HxW projection matrices queries, keys and values for the kth head, respectively, with da = C/K.
,,.., M. W W W E RCxd, are the
4axs
The process for other three types of window attention can be derived easily, and their output for the kth head are denoted as LL-WA, output of these four parallel groups will be concatenated to form the overall output:
(HH-WAx(X).. HL- WAX) respectively. Finally the
FDWA(X) Concat(headı ... headk] W
LL-WAA (X)
k= K/4
HH-WA (X)
== K/444 K/2
(5)
with headg
HL-WAR (X)
k =
3K/4
LH- LH-WA,
1-3K/4+1,---
where Wo E RCC is the projection matrix that implements interaction between different fre- quency components.
3.2.2 FREQUENCY-MODULANE FEED-FORWARD NETWORK
The feed-forward network (FFN) in transformer is employed to refine the features produced by self- attention. As described above, FDWA produces features with diverse frequency components. How- ever, the contributions of these frequency components in compression high-bitrate models may require more high-frequency components to recover edges and fine ane-grained details, while low-bitrate models mainly rely low-frequency components to reconstruct the over- all structure.
not equal. For instance,
To alternatively decide the frequency components and further eliminate the redundancy across dif- ferent components, we develop a Frequency-Moduulaton FFN (FMFFN) that can adaptively mod- ulate frequency components. Specifically, we apply block-based fast Fourier transform (FFT) to transform the feature Xtta obtained by a standard FFN into frequency domain. Next, we intro- duce learnable filter matrix W to alternatively suppress or amplify all frequency components by element-wiss multiplication in the frequency domain, obtaining the frequency-modulate feature Im- Subsequently, Xim is inversely transformed using the inverse FFT (IFFT) and reshaped to obtain the refined feature X The overall process of FMFEN can be formulated as follows:
X Convix1 (GELU(Coovv (x)))
X F [BB((B(B))))) @ W
(6)
X X B 1 (Xm))
where X is the input hidden representation of the FMFEN, C denotes the element-wise multipli- cation. F(-) and F-) denote the FFT and in IFFT, respectively. B(-) and B B-)(-) denote block partitioning and block merging operation, respectively. The block size is set to 4s x 4s, which cor- responds to the maximum window size in our FDWA. By performing block partitioning, the size of the weight matrix W becomes independent the input image size. This characteristic allows our FMFEN to generalize input images of arbitrary sizes.

---

Published as a conference paper at ICLR 2024
of
(b
(0)
Parameters Net
FXT Binkk
Chessia 72
RBS i2
ERT Blokk
Condaa3
AD
FFN
(c) Parametenn
Nora
Göuniii
(Coniie
OCaenda
OCoen
GELU
GR2.1
G18.1
Masked-Sip Attention
+
Norm
Masked Slice Channel Artention
ha
OCan
OCann
Have
T-CA modd with hyppeggiin push Figure 3: Proposed Transformer-basee Channel-wise Autoregressive (T-CA) entropy model, the hyperprior path is also included. For briefness, we suppose each slice has 3 channels in (a). GCony denotes the group convolutions with kernel size of n..
Reshape ① SolMax
K Slice wise Mask Marris Mutoplication
3.3 TRANSFORMER BASED CHANNEL- -IIE AUTOREGRESSIVE (T-CA) ENTROPY MODEL
Previous channel-ver -wiee autoregressive models (Mnneen Singh 2020: He al. 2022) divide latent to several slices along the channel dimension, and then utilize separate CNNs to establish the dependencies of each slice on previously decoded slices. However, each channels vary dramatically across different input images, so that the fixed weights in CNNs can not fully exploit the channel-wise correlations. this paper, we propose a novel transformer- based channel-wist autoregressive (T-CA) entropy model as illustrated in Figure 3 By introducing channel-atee ctannel-attention in channel-we wise autoregression, our T-CA can effectively capture the inter- and intra-slices channel dependencies, leading to more precise distribution estimation.
coefficient distribution in
Our T-CA has L transformer layers with a similar architecture of vision transformer (ViT) (Doso- vitskiy al. 2020), while utilize channel attention instead of spatial attention. For the input quantized latent representation ÿ E R xx where M, H, W denote the number of channels, height and the width, we first evenly divide it into n, slices {Ū) 02, ...... Ôn. }. so that each slice has M, channels (i.e., 9. 07 ... A,A,) where i the index of slice and M, M/nn, Before feeding the slices to the transformer, we project each slice to (r PM.M channels independently, where r is a projection ratio larger than 1. This process can be simply implemented by inputting ÿ group convolution, of which the input and output channels are M and (r M), kernel size is 1x and the number of group n equal to ntg-
We modify the standard transformer layer to guarantee the causality of the entropy coding, First, we introduce the masked-slice channel attention by incorporating causal slice-wise mask, which ensures that the slices not encoded do not affect other slices. Besides, we also employ pseudo start slice ys similar to Mentzer et al. (2022). Second, we replace LayerNorm with GroupNorm and substitute all linear layers with 1 group convolution layers, where the number of groups Ny = n,. These group-wise operations not only facilitate improved modeling intra-slice channel dependencies, but also reduce the model complexity, leading to improved computational efficiency. Moreover, we utilize group convolutions to predict the Guassian parameters μ., a and the latent residual from the contextual information and hyperprior. Please refer Appendix-Ar A.3 for details.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We train the proposed FTIC models on the Flickr2W (Liu et al. 2020) and ImageNet- 1k (Deng et al.) 2009) dataset for 3.2M steps with batch size of The model is optimized using Adam optimizer

---

Published as a conference paper at ICLR 2024
Oun
MASCVPREZI
20u(CU9222)
Me/ACH19923)
Saler CUALSI
VVCVIN 12.31
SPG/8-4/41
3.2
0.4
0.6
0.8
1.0
02
24
0.6
Figure 4: R-D performance evaluated on the Kodak dataset. The compared methods include state- of-the-art LIC models and handcrafted image codecs. Left: PSNR; right: MS-SSIM.
with the learning rate initialized le-4. Specifically, we follow Zhu et al. (2022) to adopt the multi-stage training strategy. The details can be found in Appendix B We ues R- loss in Equation 3 to optimize the model. Two kinds of quality metrics, i.e., mean square error (MSE) and multiscale structural similarity (MS-SSIM), are used to measure the distortion D. The Lagrangian multiplier used for training MSE-optimized models (0.0025, 0.0035, 0. 0.0067, 0.0130, 0.0250, 0.0483), and those for ASS-SSIM-optimiin models are (2.40, 4.58, 33,3, 16.64, 31.73, 60.50).
For FAT blocks, the base window size is set as 4 in the nonlinear transforms (i.e., 9- and g. 9.).), and set as in the hyperprior transforms ( (.e., (-) and (-)). The channel number M of the latent 192. For T-CA entropy model,
is set 320, while the channel number N of hyper latent is the number of slices is set as 5, the number of transformer layers L is set as 12. and projection ratio is set as More hyper parameters of architecture can be found in Appendix-AA.. We use NVIDIA GeForce RTX 4090 and Intel Xeon Platinum 8260 to conduct the following experiments.
We evaluate our the proposed model on three benchmark datasets, i.e., Kodak image set (Kodak) 1993) with 24 images of 768 x 512 pixels, Tecnick testset (Asuni & Glachetti| (204]) with 100 images of 1200 1200 pixels, CLIC Professional Validation dataset (CLIC) (021) with images of at most 2K resolution. We use both PSNR and MS-SSIM to measure the distortion, while bits per pixel (BPP) used to evaluate bitrates.
4.2 RATE-DISTORTION PERFORMANCE
We compare our method with the state-of-ther thett--rrtttttt (SOTA) methods including the traditional image codecs BPG and VTM-12.1. and recent LIC models (Ballé et al. 2018, Cheng et al.) 2020 Xie
al. 2021 He et al., 2022 Zou al. 2022 Liu al. 2023 Fu et al. ala]a|__2222 2023). The R-D performance on Kodak dataset shown in Figure We use both PSNR and MS-SSIM
quality metric to our method. The results of Tecnick and CLIC datasets are shown in Figure TO in the Appendix, respectively. These additional results demonstrate that method consistently achieves excellent performance across all three datasets. Furthermore, we report the 2001) results to quantify the average bitrate savings with equal reconstruction
evaluate the performance
BD-rate (Bjontegaard (2001) quality, with VTM-12.1 as the anchor. Our method achives the state-of-the-art performance and outperforms VTM--2.1 by 14.5%, 15.1 %, 13.0% in BD-rate the Kodak, Tecnick, and CLIC dataset, respectively.
4.3 ABLATION STUDIES AND ANALYSIS
Effect of the architecture FAT block. We conduct ablation studies to investigate the effec- tiveness of the FAT block. For the baseline model, substitute proposed FDWA with the

---

Published as a conference paper at ICLR 2024
Table I: Comparison architectures of FAT Block, evaluated the Kodak dataset. The GFLOPs of running analysis transform (i.e, (a(-)) (-)) and the overall model parameters are listed for complexity comparison. The BD-rate is presented for performance comparison with VTM-12.1 the anchor.
Larger
Methods
FDWA
FMFEN
GFLOPs
#Params
BD-rate
Window
Bascline
77.3
70.29M
-9.3%
Variant
91.3
70.50M
-11.2%
Variant 2
82.2
70.36M
-13.3%
Variant 3 (FAT Block)
83.1
70.97M
-14.5%
VTM-12.1
0%
standard window-basee self-attention using a window size of 4x4, and replaced our FMFEN with conventional FFN. As shown Table adopting larger window size of 16x 16 can improve perfor- mance. However, this also introduces higher computational complexity. Furthermore, the adoption of FDWA results in significant improvement while maintaining comparable complexity. This indi- cates that the performance gains of introducing FDWA are derived from the ability to extract diverse frequency components rather than solely relying on large receptive field. Finally, by further intro- ducing FMFEN, we achieve the state-of-ther of-thh--rt BD-rate with only slightly increased computational complexity and model parameters.
Ablation study on T-CA. We then evaluate the ef- fect of our proposed T-CA entropy model. Here, we adopt the CNN- nonlinear transforms of Minnen et al. 20188) (-) and 9-(-) for con- venient comparison.
Table 2: Comparison of the proposed T-CA with CHARM.
Model
#Params
BD-rate
Specifically, we train all the models by loading the pretrained transforms weights of mbr20/ 88mman model from CompressAl library (Bégaint et al., 2020) and fine-tune them for IM batches. The BD-rate is evaluated on the Kodak dataset with BPG as the anchor. addition, we pro- vide the number of parameters of the entropy models show their complexity.
CHARM (n,=5)
18.3M
-14.2%
CHARM (n,=10)
34.8M
-15.9%
T-CA (n,=5)
30.4M
-19.2%
BPG
0%
Table 3: Ablations different parameters of T-CA. L is the number of transformer layers of T-CA. n, the number of slices.
We first compare our T-CA entropy model with CHARM (Minnen & Singh, 2020) to show the powerful ability of modeling channel dependency through channel attention. As shown in Table 2 our T-CA can outperform CHARM significantly with the same number of slices (i.e., 5). Furthermore, achieves superior R-D performance over CHARM with half the number of slices (5 vs 10) and reduced model complexity.
R2
#Params
BD-rate
4
5
14.5M
-17.6%
8
22.4M
-18.3%
12
30.4M
-19.2%
16
5
38.4M
-19.0%
12
4
37.8M
-19.0%
3 how different parameters impact the R-D performance and the complexity of our T-CA entropy model. The results show that, enlarging the number of transformer layers from 12 boost the performance, but using transformer layers more than 12 cannot bring additional benefits. In addition, we observe that increasing the number of slices n. does not lead to consistent performance improvement as demonstrated in Minnen & Singh (2020). This is due to that our T-CA model utilizes a shared transformer establish channel dependency among different slices, and the number of parameters does not increase linearly with n, CHARM [Minnen & Singh 2020). Conversely, increasing n, in T-CA results in a reduction number of parameters of the group convolution as well as the whole entropy model. Therefore, select n, as 5 and as 12 to balance the model size and performance.
12
5
30.4M
-19.2%
12
19.3M
-18.0%
12
10
15.6M
-18.4%
BPG
0%
Spectrum analysis of FDWA. In Figure we visualize the intensity of frequency component by applying Fast Fourier Transform (FFT) to the output feature maps from different attention modules in our FDWA. The visualisation clearly demonstrates that HH-WA captures more high frequencies,
8

---

Published as a conference paper at ICLR 2024
LL-WA
HIE-AA
LiE-A
3HL -AA
(a) analysis transform 8a
synthesis transform d+
Figure 5: Frequency intensity (16 16) from the output of FDWA the last FAT block for both (a) analysis transform 9a(-) and (b) synthesis transform (-(-). The model is trained with À set as 0.0483 and MSE metric. We show 6 output channels for each of LL-WA, HH-WA, HH-WA, and LH-WA. The magnitude values are averaged over 100 samples. Lighter colors indicate larger magnitudes, while pixels closer to the center represent lower frequencies.
(b) synthesis transform Figure 6: Visualization of the frequency domain learned filters of FMFFN at the deepest FAT block for both (a) analysis transform gu(-) and (b) synthesis transform g-(-). The four rows represent learned filters corresponding to four different channels, and the six columns from left to right repre- sent the models with increasing bitrates.
(a) analysis transform
while LL-WA mainly focuses on low frequencies. Additionally, HL-WA and LH-WA exhibit the ability to capture directional frequency. This observation strongly supports our primary goal of decomposing different frequency components within feature maps at single attention layer.
Visualization analysis of FMFEN. In Figure 6 we follow Rao et al. (2021) to visualize the fre- quency domain learned filters W in our FMFFN. For each row, from left to right, the six columns represent specific filter of models with increasing bitrates. We contains more high frequency component, this findings also supports our motivation of introducing FMFFN. In this way, model adaptively modulate different frequency components achieve better R-D performance.
that higher bitrates model
CONCLUSION
This paper proposes a novel approach for learned image compression (LIC) from the perspective of frequency decomposition. We address the challenge modeling diverse frequency information in LIC by the proposed decuuecccdcooooosttionn window attention (FDWA), which captures different orientation and spatial frequency components using various window sizes. We also introduce frequency-moddulation feed-forward network (FMFFN) module adaptively amplify suppress different frequency components for improved rate-distortion tradeoff. Furthermore, a transformer- based channel-wist autoregressive (T-CA) entropy model is developed effectively learn channel dependencies. Experimental results demonstrate that the proposed method achieves state-of the-art rate-distortion performance commonly used datasets.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported in part by the National Natural Science Foundation of China under Grant 62125109, Grant 62250055, Grant 61931023, Grant 61932022, Grant 62371288, Grant 62320106003, Grant 62301299, Grant T2122024, and Grant 62120106077
REFERENCES
Nicola Asuni and Andrea Giachetti. Testimages: large-scale archive for testing visual devices and basic image processing algorithms. In STAG, pp. 63-70, 2014.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with a scale hyperprior. In Proceedings of the International Conference on Learning Reprexentations, 2018.
Jean Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a py- torch library and evaluation platform for end-to-end compression research. arXiv preprint arXiv:201 I.03029, 2020.
Gisle Bjontegaard. Calculation of average psnr differences between rd-curves. In VCEG-M33, 2001.
Emmanuel Jean Candès and David Leigh Donoho. Curvelets: A surprisingly effective nonadaptive representation for objects with edges. Vanderbilt Univ. Press, 2000.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229, Springer, 2020.
Zhengxue Cheng, Heming Sun, Masaru Takeuchi, and Jiro Katto. Learned image compression with discretized gaussian mixture likelihoods and attention modules. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7939-7948, 2020.
CLIC. Workshop and challenge leamned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. pp. 248-255, 2009.
Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang. and Hongkai Xiong. Motion-aware contrastive video representation learning via foreground- background merging. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 1716-9726, 2022a.
Shuangrui Ding, Rui Qian, and Hongkai Xiong. Dual contrastive learning for spatio-temporal rep- resentation. In Proceedings of the 30th ACM international conference on multimedia, pp. 5649 5658, 2022b.
M.N. Do and M. Vetterli. The contourlet transform: an efficient directional multiresolution image representation. IEEE Transactions on Image Processing, (1122))091121000 2005.
David L. Donoho and Xiaoming Huo. Beamlets and multiscale image analysis. In Multiscale and Multiresolution Methods, pp. 149-196, Berlin, Heidelberg, 2002. Springer Berlin Heidelberg.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Haisheng Fu, Feng Liang, Jie Liang, Binglin Li, Guohe Zhang, and Jingning Han. Asymmet- ric learned image compression with multi-scale residual block, importance scaling, and post- quantization filtering. IEEE Transactions Circuits and Systems for Video Technology, 2023.
10

---

Published as a conference paper at ICLR 2024
Ge Gao, Pei You, Rong Pan, Shunyuan Han, Yuanyuan Zhang. Yuchao Dai, and Hojae Lee. Neural image compression via attentional multi-scale back projection and frequency decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14677-14686, 2021.
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. Elic: Efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Eastman Kodak. Kodak lossless true color image suite (photocd ped0992). 1993.
A Burakhan Koyuncu, Han Gao, and Eckehard Steinbach. Contextformer: A transformer with spatio-channel attention for context modeling in learned image compression. In European Con- ference Computer Vision, 2022.
Han Li, Bowen Shi, Wenrui Dai, Hongwei Zheng, Botao Wang. Yu Sun, Min Guo, Chenglin Li, Junni Zou, and Hongkai Xiong. Pose-oriented transformer with incertainty-guiderr refinement for 2d-to-3d human pose estimation. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 37, pp. 1296-1304, 2023.
Jiaheng Liu, Guo Lu, Zhihao Hu, and Dong Xu. A unified end-to-end framework for efficient deep image compression. arXiv preprint (xXiv:0002.03370 2020.
Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer- enn architectures. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14388-14397, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision pp. 10012-10022, 2021.
Ming Lu, Peiyao Guo, Huiqing Shi, Chuntong Cao, and Zhan Ma. tansformer-basee image com- pression. In 2022 Data Compression Conference (DCC), pp. 469 469. IEEE, 2022.
Haichuan Ma, Dong Liu, Ruiqin Xiong, and Feng Wu. iwave: Can-based wavelet-like transform for 2019.
image compression. IEEE Transactions Multimedia,
Haichuan Ma. Dong Liu, Ning Yan, Houqiang Li. and Feng Wu. End-to-end optimized versatile image compression with wavelet-like transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1247-122) 2020.
Stephane G Mallat, A theory for multiresolution signal decomposition: the wavelet representation. IEEE transactions on pattern analysis and machine intelligence, 11(7):674-692 1989.
Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson. Vct: video compression transformer. arXiv preprint Ni0.:22060003077 2022.
David Minnen and Saurabh Singh. Channel-wit autoregressive entropy models for learned image compression. In IEEE International Conference on Image Processing (ICIP), pp. 3339-3343.
IEEE, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018 8.
Zizheng Pan, Jianfei Cai, and Bohan Zhuang. Fast vision transformers with hilo attention. Advances in Neural Information Processing Systems, 11111444114555.. 2022.
Namuk Park and Songkuk Kim. How do vision transformers work? International Confer- Learning Representations, 2022. URL http:////ppeneeeeeeeeteeeeeeeeee
[DwN6xjiie HnK8
Badri Patro and Vijay Agneeswaran. Scattering vision transformer: Spectral mixing matters. Ad- vances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2024
Yichen Qian, Xiuyu Sun, Ming Lin, Zhiyu Tan, and Rong Jin. Entroformer: A transformer- entropy model for learned image compression. In Interaational Conference on Learning Repre- sentations, 2022.
Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou. Global filter networks for image classification. Advances in neural information processing systems, 34:980-993, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Yueqi Xie, Ka Leong Cheng, and Qifeng Chen. Enhanced invertible encoding for learned image compression. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 162- 170, 2021.
Ali Zafari, Atefeh Khoshkhahtinat, Piyush Mchta, Mohammad Saced Ebrahimi Saadabadi, Moham- mad Akyash, and Nasser M Nasrabadi. Frequency disentangled features in neural image com- pression. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 2819-2819. IEEE, 2023.
Yinhao Zhu, Yang Yang, and Taco Cohen. Transformer-bssed transform coding. In International Conference on Learning Representations, 2022. URU/ttpp://pppneeeiiee ffrum?
Id=IDwN6x (HnK8
Renjie Zou, Chunfeng Song, and Zhaoxiang Zhang. The devil is in the details: Window-based attention for image compression. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022.
12

---

Published as a conference paper at ICLR 2024
A MODEL ARCHITECTURE
A.1 DETAILS OF OVERALL FRAMEWORK
Figure 7 shows the overall framework of our FTIC and detailed architecture of the used RBS and RBU. The number of channels (C₁, C2 C3, .) (96, 144, 256, 320). In both (-(-) and ga(+). the numbers of attention heads are configured as (8, 8, 16, 16, 32, 32) from shallow to deep, whereas in the (-) and g-(-) all FAT blocks have 32 attention heads. The number of hyper latent channels is 192 for all layers, except for the output of hg. which has 640 channels. For the T-CA entropy model, the number of slices n, is set 5, the projection ratio is and the number of attention heads in the masked-slice channel attention is 16.
RBS 12
Aa
8s
Conudiad 13
sukppe
Com rt
Block
Block
Leak ReLU
x
Conv3x3
RBS 12
RBU 12
IGDN
Block x
Block
x
RBU 12
Residual Block Upsampling (RBU)
Block 2
x
Convis3 12
RBU 122
Leak ReLU
AA
Conv3x3
GDN
Transformer-ussed
Channel-wise Autoregression Entropy Model
architecture our learned image compression model
Stride (HBS)
Figure 7:
Left The overall framework of our FTIC. Right: The architectures of RBS and RBU, which are firstly adopted in Cheng et al. (2020)
A.2 DETAILS OF SHIFT- WINDOW OPERATION
In our FTIC, for each concatenated FAT blocks, the first one employs the regular frequency deuuueecddecompoeedd win- dow attention (FDWA) and the second one performs shift- window operations. In the shift-window operation, the window size are shifted with a bias of the half height and width size. The exact operations for each type of window attention in our FDWA are illustrated in Figure
Table 4: Detailed architectures of en- tropy parameters network.
Entropy Parameters Network
out: (3 x 320, 16, 16)
GConv (3 x 320, 3 x 320, 3, 5)
GELU0
A.3 DETAILS OF ENTROPY PARAMETERS NETWORK
GConv (3 * 320, 320, 3, 1, 1,
As shown in Figure we utilize entropy parameters net- work to predict the mean, scale, and latent residual for each slice. To obtain the input of entropy parameters network, we first reshape the decoded hyperprior ø from x to Mx2xXXXxx xx and reshape the output of nal transformer layer Yout from 4 to R M X4x Subsequently, concatenate these reshaped results, resulting in Yoomat reshaped from IR Mxx # to RR6M X XWW which serves as the input for the entropy parameters network. This procedure ensures the causality by avoiding in-
GELU0
GConv (6 x 320, x 320, 3, 1, 1,5)
input: (6 x 320, 16, 16)
13

---

Published as a conference paper at ICLR 2024
Next
Next
Block
Block
[A-WA
(c) ILL.
Next
Next
Block
Block
(h) LA
(d) LH-WA
Figure 9: The shift-window operations for each type of attention our FDWA.
formation interactions between different slices. The detailed architectures is presented Table where GCony denotes group convolution and the six parameters input channels, output channels, kernel size, stride, padding, and the number of group.
B TRAINING
Our learned image compression models are trained on Flickr2W and ImageNet-Ik 1k dataset. Previous works (He et al.] 2022 Liu eet al. 2023) encode each [y su] to the bitstream instead of and restore the coding-symbol as Ty - nd] μu, which can signifi- cantly benefit the single Gaussian entropy model. However, in autoregression model, this poses a train-test mismatch problem be- cause do not know the value of before passing through the model. To avoid this problem, in our methods, we adopt a mixed training strategy follows:
FAT Block
FAT Block
#B511
RSU
FAT Blocs
FAT Blok
In the first stage, we train our models with only a hyperpiror model the entropy model to obtain strong nonlinear transforms. The network architecture used in the first stage is presented in Figure g During this stage, we encode each restore the coding symbols as [ μ μ.
#B512
R2
g Ko
FAT Block
FAT Block
e1] the bitstream and
mau
w
In the second stage, load the transform weights (i.e., So and 9a) from the checkpoint of pretrained first stage model and fine-tune the transforms together with the random initialized T-CA. In this stage, due to the causality limitation, we directly encode each [v] to the bitstream and restore the coding-symbol [y]-
Res Vi
Somdet
FAT Block
FAT Block
Eanuta312
The first stage is trained for 2M steps with learning rate of 1e-4. Each batch contains patches with the size of 256 x 256 randomly cropped from the training images. The second stage is trained for 1M steps with the same learning rate. Finally, we train the model with learning rate of le-5 for 200K steps using larger crop size of 384 384. For all training, Adam optimizer used without weighted decay.
Figure 8: The framework of our FTIC with only hyper- prior model as entropy model
14

---

Published as a conference paper at ICLR 2024
Ours
M(e(ACMMM22) MinneniNiPS18) (VIM-12.3)
ZouICVPR221
MEIACCMMM23)
VVC (VTM
PSNA
PSNR
34
12
32
(a) Tecnick
(b) CLIC
Figure 10: R-D performance evaluated on (a) the Tecnick dataset and (b) the CLIC Professional Validation dataset.
Table 5:
Comparison coding complexity evaluated the Kodak dataset. The BD-rate is pre- sented for R-D performance comparison with VTM-12.1 as the anchor.
GFLOPs
Inference Latentcy (ms)
Model
#Params
BD-rate
Enc.
Dec.
Enc.
Dec.
Cheng et
2020
154
229
> >000
> 1000
26.60M
3.6%
Minnen & Singh 2020
101
100
56
43
55.13M
1.1%
Zhu al.
(2022)
116
116
110
99
32.71M
-3.3%
Zou al.
2022
285
276
101
99.58M
-4.3%
Liu et al.
(0233)
317
453
255
322
76.57M
-1.99
Ours
141
349
125
242
70.97M
-1.55
VTM-12.1
0%
C ADDITIONAL EXPERIMENTAL RESULTS
C. R-D PERFORMANCE TECNICK AND CLIC PROFESSIONAL VALIDATION DATESETS
We provide the additional rate-distortion results on Tecnick and CLIC Professional Validation date- sets in Figure I0 The proposed model achieves state-of-the-art on both datasets.
C.2 COMPARISON ON CODING COMPLEXITY
We compare the coding complexity of the proposed FTIC with existing state-of-the the-art LIC models in Table 5 The coding complexity is measured by inference latency during encoding and decoding process. The experiments are conducted on single NVIDIA GeForce RTX 4090 with GB mem- ory. Cheng et al. 2020)) exhibits high inference time due to the spatial auto-regressive entropy model. The experiments show that the proposed method could achieve the best BD-rate reduction with tolerable coding complexity.
C.3 COMPARISON ON TRAINING SPEED AND GPU MEMORY REQUIREMENT
We compare the training speed and GPU memory requirement of the proposed FTIC with existing state-of-the-an LIC models Table Compared with existing transformer-bar LIC methods, the proposed method exhibits comparable training speed and GPU memory requirement.
15

---

Published as a conference paper at ICLR 2024
Table Comparison on training speed and gpu memory requirement. For training, each batch con- tains 8 images with resolution of 256x256. For test, each batch contains one images with resolution of 512x 768.
Peak GPU Memory (GB)
Training Speed
Model
Training
Test
(steps/s)
Cheng et al. (020))
2.32
0.59
14.23
Minnen & Singh
2020)
3.48
0.50
8.26
Zhu et al.
(2022)
16.84
2.10
2.66
Zou et al.
(2022
10.87
0.68
4.29
Liu et al.
(0033)
14.14
1.71
2.13
Ours
12.68
1.09
3.17
C.4 VISUALIZATION OF CHANNEL ATTENTION WEIGHTS IN T-CA
638
5.14
608
48
64
662
16
48
64
600
Figure 11: Visualization results of the masked channel attention weights in our T-CA entropy model by inputting kodin01 in Kodak dataset. We display the average attention weights all the heads heads in the last transformer layers. Different slices are separated by red lines.
We present a visualization result of the channel attention weights in the last transformer layer of our T-CA entropy model in Figure I1 Each channel attention layers of T-CA consists 16 heads, with each head containing M M/nn n 80 x 80. The visualization result explicitly presents the inter-slice and intra-slice dependencies.
320/16 channels, resulting in an attention map size of
C.4.1 VISUAL QUALITY RESULTS
Figure 12 and Figure compares the visual quality of the proposed model optimized for MSE. Benefits from the extraction of multi-scale and directional frequency features, our method exhibits superior capability in reconstructing fine details and directional structures. In this way, our models achieve higher compression ratio and better reconstruction quality compared with STF (Zou et al. 2022) and VTM-12.1.
D LIMITATION AND FUTURE WORK
potential limitation of our FTIC lies the fact that our proposed Frequency-Decomppsiin Decomposition Win- dow Attention (FDWA) currently supports only two directions (horizontal and vertical). Expanding
16

---

Published as a conference paper at ICLR 2024
the range of supported directions, for example, by incorporating multi-scale and multi-directional
analysis like scattering transformation (Patro & Agneeswaran 2024) could potentially enhance the frequency decomposition of image representation. In addition, current approach focuses only image compression, our future work will extend to video compression by exploring the frequency decomposition of spatial-temporal representations (Ding et al. 2022a b).
17

---

Published as a conference paper at ICLR 2024
lifed
rijjnjinbviiaaiion
Ours
STF
VTM-12.1
Ground Truth
30.6 40.64dB/0.486bpp
29.64dB/0.549bpp
29.77dB/0.503bpp
Figure 12: The visualization of kodim08 in Kodak dataset by using FTIC model, STF ZZuu et al. 2022 and VTM-12. PSNR bitrate is listed below the subfigures.
18

---

Published as a conference paper at ICLR 2024
0
Ours
STF
VTM-12.1
Ground Truth
33.81dB/0.169bpp
33.02dB/0.172bpp
33.03dB/0.182bpp
Figure 13: The visualization of kodim in Kodak dataset by using our FTIC model, STF (Zou et al. 2022] and V..-12.1. PSNR bitrate is listed below the subfigures.
19


Reviews:
[{"review_id": "AHc9BqO2dv", "reviewer": "Reviewer_Xwct", "summary": "A learned image compression approach based on transformer architecture is presented. The main innovation is the introduction of a frequency-aware module that performs multi-scale, directional analysis of the transformer features, and then uses an FFT to weight the important frequency components. Results show mild to moderate improvements in the RD-curve and BD-metric on various datasets.", "questions": "Included in comment above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The intuition behind the method is sound. Writing and explanation is clear. Results show improvements over various transformer-based baselines.", "weaknesses": "This is a general comment rather than a specific weakness. It is not entirely clear how much improvement is brought about by the specific structure of the FDWA and FMFFN blocks, rather than the fact that some learnable layers have been included which would increase the overall capacity of the network. The authors have provided a few ablations in Figures 5. and 6. which do somewhat support their claim, but I feel this could be made stronger in future versions. For instance,\n\n1. What if additional windows with different aspect ratios were used? Would these provide further improvement? This could be achieved by either increasing $K$ (the number of heads), or by using the same number of heads but partitioning into more groups to accommodate the different aspect ratios?\n\n2. The impact of the FMFFN block seems to be rather small. What if instead of a block FFT, a learnable conv layer (and deconv for the IFFT) was used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yJhqk0MgCP", "reviewer": "Reviewer_kfZs", "summary": "This paper aims to reduce redundancies in the latent representation of Learned Image Compression methods.\nFor this purpose it proposes a new frequency-aware transformer block which utilizes two new components.\n1. FDWA: a window attention block to capture various frequency components\n2. FMFFN: a block which modulates the frequency components \n\nAdditionally this paper proposes a transformer-based channel-wise autoregressive entropy model (T-CA)\nIn combination these methods achieve SOTA performance on various datasets.", "questions": "How does the use of the T-CA entropy model affect inference times (compared to existing models such as CHARM)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method achieves SOTA performance without unreasonable performance cost.\n2. The paper is well written and provides nice visualizations of core ideas.", "weaknesses": "There are no explicit ablations of some design decisions of the FAT block. What effect do the relative window sizes of the FDWA module have? What's the impact of omitting some of the windows (eg. omitting vertical/horizontal windows)?\n\nThere are typos in the caption of Figure 3 (p.6) and Table 2 (p.8)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zyFAg6Y7e1", "reviewer": "Reviewer_fcgY", "summary": "To overcome the problem of existing LIC (Learned Image Compression) methods are redundant in latent representation, this paper suggests a nonlinear transformation and makes the following three improvements:\n\n1)\tThis paper proposes a frequency-decomposition window attention (FDWA), which leverages diverse window shapes to capture frequency components of natural images.\n\n2)\tThis paper develops a frequency-modulation feed-forward network (FMFFN) that adaptively ensembles frequency components for improved R-D performance.\n\n3)\tThis paper presents a transformer-based channel-wise autoregressive model (T-CA) for effectively modeling dependencies across frequency components.\n\nExperimental results show that this paper achieves state-of-the-art R-D performance on several datasets.", "questions": "1. Table 1 and Table 5 in the article involve flop calculations, may I ask the authors what methods or tools they used to calculate the complexity? As far as I know, many existing tools that count complexity do not calculate correctly the complexity of the internal operators of the transformer. Besides, Table 5 is kind of confusing. Can the authors explain what the (1), (2), (3) in the table mean here.\n\n2. Adding a transformer on the codec side would result in a longer training time and a larger memory requirement that is not hardware-friendly for the actual deployment of the compression model. Can the authors provide the training time as well as the corresponding average and peak memory consumption for both training and test? Besides, the three methods compared in Table 5 are all transformer-based methods, and there is no comparison of decoding time with, for example, Minnen[1] and GMM[2], where both the encoder and decoder use a CNN structure.\n\n[1] Minnen D, Singh S. Channel-wise autoregressive entropy models for learned image compression[C]//2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020: 3339-3343.\n\n[2] Cheng Z, Sun H, Takeuchi M, et al. Learned image compression with discretized gaussian mixture likelihoods and attention modules[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 7939-7948.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper is overall easy to understand and clearly written. One of the primary strengths of this paper is the claimed SOTA rate-distortion performance. \n2) The authors found that existing learned image compression methods lead to potential representation redundancy due to limitations in capturing the anisotropic frequency components of anisotropy and preserving directional details. Some attempts including FMFFN and FDWA are proposed to address this issue.", "weaknesses": "1) The idea of applying frequency processing to the learned compression framework is not new. For example, conv in the frequency domain [1] has been used in Balle’s early work [2]. Wavelet-based compression framework has also been proposed [3]. The authors should cite, compare, and identify their differences.\n\n[1] Rippel, Oren, Jasper Snoek, and Ryan P. Adams. \"Spectral representations for convolutional neural networks.\" Advances in neural information processing systems 28 (2015).\n\n[2] Ballé, Johannes, Valero Laparra, and Eero P. Simoncelli. \"End-to-end optimized image compression.\" arXiv preprint arXiv:1611.01704 (2016).\n\n[3] Ma, Haichuan, et al. \"iWave: CNN-based wavelet-like transform for image compression.\" IEEE Transactions on Multimedia 22.7 (2019): 1667-1679.\n\n2) The authors should compare with more existing works (e.g. [3,4]) to demonstrate the SOTA performance of the paper.\n\n[4] Liu J, Sun H, Katto J. Learned image compression with mixed transformer-cnn architectures[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 14388-14397.\n\n[5] Fu H, Liang F, Liang J, et al. Asymmetric Learned Image Compression with Multi-Scale Residual Block, Importance Scaling, and Post-Quantization Filtering[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OJFfVCq8OS", "reviewer": "Reviewer_ZpGq", "summary": "The paper studies the frequency problem of learned image compression and develops a frequency-aware method for this problem based on multiscale and directional analysis, called FAT. The method introduces two modules to capture frequency component. Based on FAT, the learned image compression achieves better rate-distortion performance. The method shows improvement over learned codec and conventional codec baselines by a healthy margin.", "questions": "1. Have you ever tried other decomposition ways such as (1) smaller size and bigger size for square window (I am wondering how much improvement can be obtained in this mechanism), or (2) more diverse shapes.\n \n2. What impact does the block size in FMFFN have? When you set the block size responding to the maximum window size in FDWA, does it mean that the FMFFN is used to refine the high-frequence of features?\n \n3. If the decomposition of window size works, how about directly using different sizes of convolution?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed FAT is a novel idea that captures multiscale and directional frequency components and outperforms SOTA.\n\n2. Nice visualization on multiscale and directional decomposition of frequency component. It is an interesting finding that structural information within different frequency ranges also plays a crucial role in learned image compression.", "weaknesses": "1. It is interesting to find that FDWA achieves a significant improvement compared to 4x4 blocks and 16x16 blocks. However, it is not clear that how does the multiscale decomposition and the directional decomposition affect. The authors are suggested to provide an ablation study on FDWA.\n \n2. The frequency-aware transformer is realized by two modules, FDWA and FMFFN. Have you tried these two mechanisms in the entropy model? Entropy model plays a crucial role in learned image compression. In my opinion, it is more important to capture frequency component in entropy model. It would be a great improvement if the frequency-aware mechanism works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)