Test-time Adaptation for Image Compression with Distribution Regularization

## Abstract

Abstract Current test- or compression-time adaptation image compression (TTA-IC) approaches, which leverage both latent and decoder refinements as a two-step adaptation scheme, have potentially enhanced the rate-distortion (R-D) performance of learned image compression models on cross-domain compression tasks, e.g., from natural to screen content images. However, compared with the emergence of various decoder refinement variants, the latent refinement, as an inseparable ingredient, is barely
tailored to cross-domain scenarios. To this end, we aim to develop an advanced latent refinement method by extending the effective hybrid latent refinement (HLR) method, which is designed for in-domain inference improvement but shows noticeable degradation of the rate cost in cross-domain tasks. Specifically, we first provide theoretical analyses, in a cue of marginalization approximation from in- to cross-domain scenarios, to uncover that the vanilla HLR suffers from an underlying mismatch between refined Gaussian conditional and hyperprior distributions, leading to deteriorated joint probability approximation of marginal distribution with increased rate consumption. To remedy this issue, we introduce a simple Bayesian approximation-endowed distribution regularization to encourage learning a better joint probability approximation in a plug-and-play manner. Extensive experiments on six in- and cross-domain datasets demonstrate that our proposed method not only improves the R-D performance compared with other latent refinement counterparts, but also can be flexibly integrated into existing TTA-IC methods with incremental benefits.

## 1 Introduction

With rapid developments in data streaming techniques, fruitful high-resolution images need to be transmitted online between edge devices. It is therefore imperative to develop more efficient, effective, and versatile image compression approaches for better storage and transmission. To this end, we have witnessed recent learning-based image compression (LIC) methods (Ballé et al., ; ; Cheng et al., ; Jiang et al., ; Kim et al., ) significantly outperform conventional codecs, such as VVC (Bross et al., ) and JPEG (Wallace, ), in terms of rate-distortion (R-D) performance. Such gains mainly derive from the unprecedented non-linear transform capacity of deep neural networks (DNN) and accurate probability representations for entropy coding, in an end-to-end R-D cost-guided learning framework.

Nevertheless, these DNN-based LIC approaches inevitably inherit the identically and independently distributed (i.i.d.) assumption between the source (training data) and target (testing data) domains, which may not always hold in versatile image compression scenarios, e.g., there is a significant distribution gap between natural and screen content images. Compared with in-domain compression (i.e., i.i.d. assumption holds), such domain shifts would deteriorate the R-D performance of DNN-based codecs in cross-domain compression. For example, most advanced LIC models leverage a hyperprior-based entropy framework (Ballé et al., ), where the hyperprior model extracts the side information $z$ of the latent variable $y$ to capture Gaussian conditional probability $p(y|z)$ for coding, and the side information is coding by the hyperprior probability $p(z)$ learned from entropy bottleneck. In cross-domain scenarios, the discrepancy in statistical property between source and target domains will cause inaccuracy or ineffectiveness of learned probability models (Ulhaq & Bajić, ), leading to suboptimal entropy coding with additional bit consumption. Moreover, it is difficult for a source-trained decoder to render high-fidelity target images due to domain shifts.

Figure: Figure 1: Comparison of various latent refinement methods in R-D performance under cross-domain tasks (testing on SIQAD screen content dataset). The Cheng20 (Cheng et al., ) (quality$=0$, pre-trained on natural images) model is used.
Refer to caption: x1.png

To this end, it is necessary to conduct a test- or compression-time adaptation for image compression (TTA-IC). One approach (Van Rozendaal et al., ; Lam et al., ; Zou et al., ) is to update the decoder of the entropy model, which involves the transmission of adapted parameters to the decoder side. Although recent parameter-efficient transfer learning (PETL)-based (Hu et al., ) extensions (Shen et al., ; Lv et al., ) reduce the huge bitrate overhead of parameter transmission to an acceptable level, the entropy optimization of updated parameters and extra bit consumption are still troublesome. Another promising approach (Djelouah & Schroers, ) is to directly refine the latent variables without altering any model parameters. Compared with the first approach, such a branch is still effective for many specialized neural decoders, where model parameters are hard-coded and non-modified (Dass et al., ).

Here, we focus on the second approach, i.e., latent refinement, due to its simple optimization, friendly bitrate, and natural immunity to catastrophic forgetting.
Some state-of-the-art (SOTA) TTA-IC methods (Campos et al., ; Shen et al., ) actually have introduced basic latent refinement (BLR) (Djelouah & Schroers, ) into their two-step adaptation framework including latent and decoder refinements, e.g., updating the latent variable $y$ by R-D objective. Yet, the Gaussian conditional probability $p(y|z)$ and hyperprior $p(z)$ are still inaccurate without refining the side information $z$, leading to suboptimal cross-domain R-D performance. A potential solution may be hybrid latent refinement (HLR) (Yang et al., ) that conducts a joint update of latent variable $y$ and side information $z$. Although the vanilla HLR, tailored to in-domain inference improvement, exhibits better R-D performance compared with BLR in in-domain TTA-IC tasks (Yang et al., ), it suffers from a significant downside in rate cost while enhancing the reconstruction quality in cross-domain scenarios, as depicted in Figure 1. This degradation in rate cost is, however, ignored by existing two-step TTA-IC approaches (Lv et al., ; Tsubota et al., ; Shen et al., ) that directly impose the vanilla HLR as their inseparable ingredient.

Motivated by the abovementioned analyses, this study aims to develop an advanced latent refinement method that can be adaptive to cross-domain TTA-IC with consistent R-D gains. Such a scheme can not only render better latent representations with zero model update and transmission but also enhance the R-D performance of existing SOTA TTA-IC approaches as an effective alternative to existing latent refinement. With these goals in mind, we propose to tailor the vanilla HLR method designed for in-domain inference improvement to cross-domain cases. To achieve this, a theoretical analysis, in a cue of marginalization approximation from in- to cross-domain scenarios, is provided to reveal the degradation reasons of the vanilla HLR in the cross-domain scenario. In a nutshell, we uncover that the underlying mismatch between refined Gaussian conditional and hyperprior distributions may trigger the deteriorated joint probability approximation of marginal distribution, leading to increased rate consumption. To remedy this issue, we introduce a novel distribution regularization to the existing R-D objective, which encourages learning a better joint probability approximation from a theoretical perspective. Moreover, we impose a Bayesian approximation of the proposed distribution regularization to circumvent any model modification in a plug-and-play manner. Experiments on in- and cross-domain tasks demonstrate that our proposed method surpasses other latent refinements approaches and contribute to SOTA TTA-IC models with increased gains.

## 2 Related works

To mitigate the domain gap issue, various studies have been conducted for test- or compression-time adaptation of image compression. One strategy entails fine-tuning the encoder during inference to refine the latent variables without introducing extra bit transmission. For instance, Djelouah & Schroers () addressed this challenge through the BLR, which employs gradient descent on the latent representation under the guidance of R-D cost. Guo et al. () proposed a two-step BLR extension that first optimizes the latent variable by R-D cost and further optimizes the side information by the rate cost. Beyond the BLR, Yang et al. () proposed the hybrid latent refinement (HLR), which not only conducts a joint latent refinement for latent variable and side information but also eliminates the discretization gap of latent representations. Although extensive experiments (Yang et al., ) demonstrated that
the HLR-based method can significantly enhance the R-D performance compared with BLR-based counterparts for in-domain inference improvements, its performance in cross-domain scenarios with strong domain shifts lacks comprehensive understanding.

Besides the latent refinement, decoder refinement is a widely used test-time adaptation method for image compression, where the model parameters on the decoder side are updated and transmitted. For example, Van Rozendaal et al. () performed comprehensive parameter updates in the decoder and entropy model using rate-distortion optimization. Lam et al. () solely updated the biases in convolutional layers within the post-processing network. Zou et al. () introduced multiplicative parameters that can overfit and update them for intraframe coding. Despite achievements, these approaches suffer from relatively more bits when compressing a single image due to the updates of many model parameters (Kim et al., ). To address this issue,
very recent approaches motivated by parameter-efficient transfer learning (PETL) (Hu et al., ) aimed to introduce the adaptor of the decoder for better reconstruction quality with lower bit consumption (Shen et al., ; Tsubota et al., ). For example, Lv et al. () proposed to make a marriage between latent refinement and low-rank adaptation of the decoder, thereby achieving SOTA performance on cross-domain tasks.

In a nutshell, recent SOTA TTA-IC approaches (Shen et al., ; Tsubota et al., ; Lv et al., ) mainly leverage a combination of latent refinement and PETL-based decoder refinement. However, compared with the emergence of various
PETL-based decoder refinement variants, the latent refinement is barely tailored to cross-domain scenarios, thereby exhibiting underexplored space.

## 3 Methodology

We first explain existing learned image compression from a perspective of marginalization approximation in the context of in-domain R-D cost-guided training. Then, we analyze the practical marginalization approximation when applying the learned image codec to cross-domain image compression. We further render a simple solution to achieve better R-D performance for cross-domain image compression based on the analysis of practical marginalization approximation.

### 3.1 In-domain Marginalization Approximation

First, we introduce the marginalization approximation used by existing entropy models (Ballé et al., ; Cheng et al., ; Zou et al., ). Specifically, the true (unknown) marginal distribution $p(y)$ of the latent variable $y$ is usually approximated by observed data $y$ conditioned
on its hyper latent variable $z$ (a.k.a., side information), which makes the conditional distribution $p(y|z)$ tractable, where hyperprior models are adopted to render such hyper latent variable $z$. In the context of hyperprior-based entropy models, the following result holds,

###### Lemma 1 ( Ballé et al. ( 2018 ) ) .

Let $p(y|z)$ and $p(z)$ be accessed, a joint probability over $y$ and $z$ can be constructed to approximate the true marginal probability over $y$,

$$ $p(y)=\int p(y,z)dz,\quad p(y,z)=p(y|z)\cdot p(z),\quad s.t.,y=g_{a}(x),z=h_{a} (y),$ (1) $$

where $x$, $g_{a}(\cdot)$, and $h_{a}(\cdot)$ denote a raw image, an analysis transform function of the entropy model, and an analysis transform function of the hyperprior model.

Lemma 1 implies that the compression of the raw image involves the compression of $z$ using the learned prior distribution $p(z)$ and further compression of $y$ using the learned conditional distribution $p(y|z)$. Practically, we implement $p(z)$ by a non-parametric, fully-factorized density model (a.k.a., entropy bottleneck) (Ballé et al., ), and implement $p(y|z)$ by a mean-scale Gaussian model, i.e.,

$$ $p(z)=p(z|\mathbf{\varphi}),p(y|z)=p(y|z,\theta_{h_{e}})=\mathcal{N}(\mu,\sigma ),\mu,\sigma=h_{e}(z;\theta_{h_{e}}),$ (2) $$

where $\varphi$ denotes the learnable parameters of the density model characterized by the univariate distribution, and $h_{e}(\cdot)$ denotes a synthesis transform function of the hyperprior model parameterized by learnable $\theta_{h_{e}}$. The learning of these parameterized probability models derives from training data, which means that the i.i.d. assumption between training and testing data is implicitly needed. We therefore call such marginalization approximation as the in-domain one.

The widely used R-D objective can also be formulated by maximizing the probability of joint distribution $p(y,z)$ and the posterior distribution $p(x|y,\theta_{g_{e}})$, i.e.,

$$ $\displaystyle\mathcal{L}_{rd}=-\log[p(y|z)\cdot p(z)]+\lambda(-\log p(x|y))= \underbrace{-\log p(y|z)-\log p(z)}_{Rate}+\underbrace{\lambda\|x-g_{e}(y)\|_{ 2}^{2}}_{Distortion},$ (3) $$

where $g_{e}(\cdot)$ is a synthesis transform function to reconstruct the image $x$. Minimizing Eq. (3) can contribute to an optimal joint probability $p(y,z)$, which approximates the marginal distribution $p(y)$, i.e., Lemma 1 holds. However, such marginalization approximation requires more information to be encoded (Townsend et al., ). Theoretically, we can quantify extra encoding information of marginalization approximation using the entropy $H(\cdot)$ of distribution as the rate consumption, i.e.,

###### Proposition 1 .

Let $y$ and $z$ be the latent and hyper latent variables, and these variables with the asterisk be their optimal representations. In the context of in-domain image compression, if an optimal joint probability approximation of true marginal distribution can be achieved by minimizing Eq. (3), the extra rate consumption of marginalization approximation is

$$ $\displaystyle\Delta H^{*}=H(y,z)-H(y^{*})=-\log p(z|y),$ (4) $$

###### Proof.

On the one hand, with joint probability and the Bayesian rule $p(y^{*})=\frac{p(y^{*}|z^{*})p(z^{*})}{p(z^{*}|y^{*})}$, we have

$$ $\displaystyle H(y,z)=[-\log p(y|z)-\log p(z)],\quad H(y^{*})=-\log p(y^{*}|z^{ *})-\log p(z^{*})+\log p(z^{*}|y^{*}).$ (5) $$

Then, we have

$$ $\displaystyle\Delta H^{*}$ $\displaystyle=$ $\displaystyle[-\log p(y|z)-(-\log p(y^{*}|z^{*}))]+[-\log p(z)-(-\log p(z^{*}) )]-\log p(z^{*}|y^{*}).$ (6) $$

On the other hand, with Lemma 1, we have

$$ $\displaystyle-\log p(y|z)-(-\log p(y^{*}|z^{*}))=0,\quad s.t.\quad p(y|z)=p(y^ {*}|z^{*}),$ (7) $\displaystyle-\log p(z)-(-\log p(z^{*}))=0,\quad s.t.\quad p(z)=p(z^{*})$ (8) $$

Thus,

$$ $\displaystyle\Delta H^{*}=-\log p(z^{*}|y^{*})=-\log p(z|y)$ (9) $$

For in-domain image compression, Eq. (7), Eq. (8), and Eq. (9) hold, as $p(y|z)$, $p(z)$, and $p(z|y)$ are close to optimal probability representations $p(y^{*}|z^{*})$, $p(z^{*})$, and $p(z^{*}|y^{*})$ due to the assumption of the optimal joint
probability approximation of true marginal distribution.
∎

Lemma 1 and Prop. 1 are mainly adaptive to in-domain image compression. By this cue, we will discuss practical marginalization approximation and its impact on extra rate consumption in the context of cross-domain image compression.

###### Lemma 1 ( Ballé et al. ( 2018 ) ) .

###### Proposition 1 .

###### Proof.

### 3.2 Generalize to cross-domain marginalization approximation

When we apply the learned image codec to cross-domain scenarios, e.g., assuming the source-domain image as $x_{s}$ and the target-domain image $x_{t}$, the i.i.d. assumption between source and target domains violates. We can derive that Lemma 1 may not hold due to the following insight:

###### Proposition 2 .

The practical joint probability $p(y_{t},z_{t})$ on cross-domain images will deteriorate to make it not equivalent to an optimal joint probability $p(y_{t}^{*},z_{t}^{*})$ (applicable to the cross-domain images) that also corresponds to an optimal marginal approximation distribution $p(y_{t}^{*})$, leading to deteriorated marginalization approximation, i.e.,

$$ $p(y_{t}^{*},z_{t}^{*})\not=p(y_{t},z_{t}),p(y_{t}^{*})\not\approx p(y_{t},z_{t }),p(y_{t},z_{t})=p(y_{t}|z_{t},\theta_{h_{e}}^{s})\cdot p(z_{t}|\varphi^{s}),$ (10) $$

###### Proof.

The distribution shifts, i.e., $p(x_{t})\neq p(x_{s})$, would result in the entropy bottleneck $p(z_{t}|\varphi^{s})$ parameterized by $\varphi^{s}$ (which replaces $\varphi$ in Eq. (2) to emphasize its correlation with source-domain images $x_{s}$) is quite poor at specializing and encoding $z_{t}$ for cross-domain images. Also, $p(y_{t}|z_{t},\theta_{h_{e}}^{s})$ is inaccurate to encode $y_{t}$ due to source image-correlated $\theta_{h_{e}}^{s}$ and unreliable $z_{t}$. Thus, the joint probability $p(y_{t},z_{t})$ significantly deteriorates compared with the optimal one $p(y_{t}^{*},z_{t}^{*})$ and further causes an unfavorable marginalization approximation $p(y_{t}^{*})\not\approx p(y_{t},z_{t})$.
∎

In light of Prop. 2, a cross-domain extension of Prop. 1 can be derived as follows,

###### Corollary 1 .

The extra rate consumption of cross-domain marginalization approximation $\Delta H$ will be larger than that of in-domain marginalization approximation, i.e.,,
$\Delta H>\Delta H^{*}$, due to deteriorated joint probability.

###### Proof.

With Prop. 2 as a condition, Eqs. (7) and (8) in Prop. 1 can be further represented as follows,

$$ $\displaystyle-\log p(y|z)-(-\log p(y^{*}|z^{*}))>0,\quad s.t.\quad p(y|z)\neq p (y^{*}|z^{*}),$ (11) $\displaystyle-\log p(z)-(-\log p(z^{*}))>0,\quad s.t.\quad p(z)\neq p(z^{*})$ (12) $$

Thus, we have

$$ $\displaystyle\Delta H=[-\log p(y|z)-(-\log p(y^{*}|z^{*}))]+[-\log p(z)-(-\log p (z^{*}))]-\log p(z^{*}|y^{*})>\Delta H^{*}$ (13) $$

Eq. (13) implies more rate consumption is potentially incurred in cross-domain scenarios.
∎

Besides, distortion error is increased as the posterior probability $p(x_{t}|y_{t},\theta_{g_{e}}^{s})$ correlated with source-domain images $x_{s}$ is suboptimal for cross-domain images $x_{t}$. To enhance the R-D performance on cross-domain images $x_{t}$, latent refinement is proposed to optimize latent variables while making model parameters unchanged. We summarize existing latent refinement methods as below.

Assume that the initial latent representations $y_{t}$ obtained by analysis transform as $y^{0}_{t}$ and the initial latent representations $z_{t}$ obtained by hyperprior transform as $z^{0}_{t}$.
The BLR scheme (refers to Figure 2) only updates the latent representation $y^{0}_{t}$ with $\mathrm{M}>1$ steps, for each step $m$:

$$ $y^{m+1}_{t}=y^{m}_{t}-\epsilon\cdot\frac{\partial\mathcal{L}_{blr}}{\partial y _{t}},\quad\mathcal{L}_{blr}=-\log p(y^{m}_{t}|z^{0}_{t},\theta_{h_{e}}^{s})+ \lambda(-\log p(x_{t}|y^{m}_{t},\theta_{g_{e}})),$ (14) $$

where the R-D cost excludes $-\log p(z)$ in Eq. (3) as the latent representation $z^{0}_{t}$ is unchanged. For BLR, the gain of R-D performance is limited, as it is difficult to obtain an updated $y_{t}^{m}$ that can simultaneously lead to minimal rate cost and minimal distortion error in Eq. (14) due to fixed $z_{t}^{0}$. Instead, the HLR in (Yang et al., ) not only conducts a joint amortization gap minimization but also eliminates the discretization gap for latent representations using the following optimization step,

$$ $\displaystyle y_{t}^{m+1}=y_{t}^{m}-\epsilon\cdot\frac{\partial\mathcal{L}_{ hlr}}{\partial y_{t}},z_{t}^{m+1}=z_{t}^{m}-\epsilon\cdot\frac{\partial \mathcal{L}_{hlr}}{\partial z_{t}},$ $\displaystyle\mathcal{L}_{hlr}=-\log p(y_{t}^{m}|z_{t}^{m},\theta_{h_{e}}^{s}) -\log p(z_{t}^{m}|\varphi^{s})+\lambda(-\log p(x_{t}|y_{t}^{m})).$ (15) $$

By simultaneously optimizing $y_{t}$ and $z_{t}$, Yang et al. () reported significant R-D gains compared with BLR for in-domain adaptive compression. However, when we impose the HLR scheme to cross-domain adaptive compression, such gains are marginal as shown in Figure 1, where the reconstruction quality of the HLR achieves significant gains compared with BLR, but the HLR consumes more bits.

Figure: (a) BLR
Refer to caption: x2.png

Discussion – we provide the analyses of degradation reasons of the vanilla HLR in the cross-domain scenario using Cor. 1 and Prop. 2. First, compared with in-domain adaptive compression, the biggest trouble in the cross-domain scenario is that the static entropy bottleneck correlated with the distribution property of source-domain images cannot well model the true distribution of cross-domain latent variable $z_{t}$ as in Prop. 2. Second, although updating $z_{t}^{0}$ to an appropriate $z_{t}^{m}$ that can render a minimal negative log-likelihood of $p(z_{t}^{m}|\varphi^{s})$ is possible, ensuring $z_{t}^{m}$ as an appropriate Gaussian condition of $y_{t}^{m}$ is questionable for a minimal negative log-likelihood of $p(y_{t}^{m}|z_{t}^{m},\theta_{h_{e}}^{s})$. Last but not least, by recalling Porp. 2 and Cor. 1, more bit consumption by the HLR implies that the finally updated joint probability $p(y_{t}^{\mathrm{M}},z_{t}^{\mathrm{M}})$ may be further away from the optimal (unknown) joint probability $p(y_{t}^{*},z_{t}^{*})$, and even the initial joint probability $p(y_{t}^{0},z_{t}^{0})$. This can be directly reflected by more rate consumption of marginal approximation by extending the result of Cor. 1 to specific update steps,

$$ $\Delta H^{\mathrm{M}}>\Delta H^{0}>\Delta H^{*}.$ (16) $$

To conclude, the underlying mismatch between $p({z_{t}^{m}|\varphi^{s}})$ and $p(y_{t}^{m}|z_{t}^{m},\theta_{h}^{s})$ (in the first two points) triggers deteriorated joint probability approximation of true marginal distribution (in the third point).

###### Proposition 2 .

###### Proof.

###### Corollary 1 .

###### Proof.

### 3.3 Distribution regularization via Bayesian approximation

Thus, we are interested in an advanced latent refinement method, which not only can significantly improve the reconstruction quality but also can enjoy a mild rate cost after latent refinement. Motivated by cross-domain joint probability approximation in Prop. 2 and corresponding impact for rate consumption in Cor. 1, we propose to introduce a simple yet efficient distribution regularization into the objective of vanilla HLR as follows,

$$ $\mathcal{L}_{DR}=-\log p(y_{t}^{m}|z_{t}^{m},\theta_{h_{e}}^{s})-\log p(z_{t}^ {m}|\varphi^{s})+\uwave{\beta(-\log p(z^{m}_{t}|y_{t}^{m}))}+\lambda(-\log p(x _{t}|y_{t}^{m})),$ (17) $$

where the third term is the distribution regularization has two advantages as follows.

(i) — If the balance coefficient $\beta$ is 1, minimizing the entropy of probability estimates ($i.e.,$ the first two terms in Eq. (17)) is equivalent to minimizing the distribution gap between estimated probability $p(y|z)$ or $p(z)$ and (unknown) optimal probability $p(y^{*}|z^{*})$ or $p(z^{*})$ (MacKay, ). This coincides with the objectives of the first two terms in Eq. (13) of Cor. 1. Thus, $\mathcal{L}_{DR}$ can be derived as follows,

$$ $\mathcal{L}_{DR}\propto\Delta H+\lambda(-\log p(x_{t}|y_{t}^{m})).$ (18) $$

By recalling Eq. (13) in Cor. 1 and Eq. (16), Eq. (18) implies that the minimization of extra rate consumption of marginalization approximation corresponds to encouraging the deteriorated joint probability approximation to approach the initial and even optimal ones. In other words, $\Delta H^{*}$ is the lower bound of the first term of Eq. (18), i.e., the better joint probability approximation, the closer to the lower bound. $\mathcal{L}_{DR}$ can potentially remedy the additional rate of consumption of vanilla HLR.

(ii) — For the optimization process in Eq. (15),
there is no explicit constraint to ensure that $z_{t}^{m}$ can well match its posterior distribution
under the condition of $y_{t}^{m}$, leading to deteriorated joint probability. As a practical implementation of the ideal posterior distribution, the proposed distribution regularization in Eq. (18) can eliminate such an issue at test or compression time.

To implement the $\mathcal{L}_{DR}$, it is necessary to model the estimated posterior distribution $p(z^{m}_{t}|y_{t}^{m})$ of $z_{t}^{m}$ given $y_{t}^{m}$. However, for existing hyperpiror models, the practical $\hat{z}_{t}^{m}$ conditioned by $y_{t}^{m}$, i.e., $\hat{z}_{t}^{m}=h_{a}(y_{t}^{m};\theta_{h_{a}})$, is a deterministic output without distribution property. Although it is feasible to approximate the $\hat{z}_{t}^{m}$ as the mean $\hat{\mu}_{t}^{m}$ of a Gaussian distribution (as the assumption of posterior distribution) while constructing a variance branch from scratch (Vahdat & Kautz, ), the optimization of the variance model may be difficult when model parameters are fixed during latent refinement phase. To address these issues, we utilize the dropout variational inference  (Gal & Ghahramani, ) as a Bayesian approximation to existing hyperprior models for modeling the posterior distribution. Specifically, without introducing any new network, the deterministic pre-trained model $\theta_{h_{a}}$ can be formulated as a probabilistic one $p(\theta_{h_{a}})$, by treating the weights as distributions. Thus, the estimated posterior estimation can be represented as follows,

$$ $p(z_{t}^{m}|y_{t}^{m})\approx\hat{p}(z_{t}^{m}|y_{t}^{m})=\mathcal{N}(\hat{\mu }_{t}^{m},(\hat{\sigma}_{t}^{m})^{2}),$ (19) $$

where the posterior distribution is estimated and assumed as a fully factorized Gaussian distribution. The estimated mean $\hat{\mu}_{t}^{m}$ and variance $(\hat{\sigma}_{t}^{m})^{2}$ can be represented as follows,

$$ $\hat{\mu}_{t}^{m}=\frac{1}{T}\sum_{i=1}^{T}h_{a}(y_{t}^{m};\theta_{h_{a}}^{i}) ,\hat{\sigma}_{t}^{m}=\frac{1}{T}\sum_{i=1}^{T}[h_{a}(y_{t}^{m};\theta_{h_{a}} ^{i})-\frac{1}{T}\sum_{i=1}^{T}h_{a}(y_{t}^{m};\theta_{h_{a}}^{i})]^{2},\theta _{h_{a}}^{i}\sim q_{\vartheta}(\theta_{h_{a}})$ (20) $$

For the practical dropout variational inference (DVI) (Gal & Ghahramani, ), the dropout strategy (Srivastava et al., ) is conducted to render approximate samples from the posterior distribution, which equals to use a Bernoulli variational distribution $q_{\vartheta}(\theta_{h_{a}})$, parameterized by $\vartheta$, to approximate the true model weight posterior $p(\theta_{h_{a}})$. By conducting $T$ times of Monte Carlo (MC) sampling from $q_{\vartheta}(\theta_{h_{a}})$, we can estimate the mean and the variance of the posterior distribution $p(z_{t}^{m}|y_{t}^{m})$. The dropout probability related to the Bernoulli variational distribution is set to 0.5. The computation consumption of the DVI is mild, as a single instance can parallelly sample $T$ masked weights by one inference in a batch of repeated instances. Moreover, it is flexible to degenerate to vanilla deterministic networks when the dropout probability is 1. Finally, we compute the negative log-likelihoods of the estimated posterior distribution given the current latent variable $z_{t}^{m}$.

Discussion – connection with Bit-back coding (BBC). Both BBC (Townsend et al., ; Ruan et al., ; Ho et al., ) and our method derive from the joint probability approximation of marginal distribution. However, BBC usually specializes in in-domain image compression to narrow the marginalization gap, i.e., transforming learned optimal joint probability to true marginal probability at compression time by minimizing the following objective:

$$ $\mathcal{L}_{BBC}=-\log p(y_{t}^{m}|z_{t}^{m},\theta_{h_{e}}^{s})-\log p(z_{t} ^{m}|\varphi^{s})-(-\log p(z^{m}_{t}|y_{t}^{m}))+\lambda(-\log p(x_{t}|y_{t}^{ m})).$ (21) $$

For cross-domain scenarios, such optimal joint probability does not hold due to mismatched encoding distribution as discussed in Eqs. (13) and (16).
Instead, we focus on refining deteriorated joint probability to potentially optimal (even initial) one by minimizing the extra rate consumption of marginal approximation, as discussed in Eq. (18).
More differences refer to Appendix.

## 4 Experiments

Datasets. By following previous literature (Lv et al., ; Tsubota et al., ; Shen et al., ), we collect six different datasets with four types of image styles to comprehensively evaluate the R-D performance of different approaches on cross-domain TTA-IC tasks, including natural image (Kodak(^1^11https://r0k.us/graphics/kodak/)), screen content image (SIQAD Yang et al. (), SCID (Ni et al., ), CCT (Min et al., )), pixel-style gaming image (Lv et al. ()’ self-collected), and painting image (DomainNet (Peng et al., )) datasets. The details of the used dataset can be found in the Appendix. Specifically, we consider the natural image dataset as in-domain evaluations, and others as cross-domain evaluations.

Figure: (a)
Refer to caption: x5.png

Implementation Details. We use CompressAI (Bégaint et al., ) to implement our proposed and baseline methods. Two widely-used backbone models are adopted, including the base hyperprior-based entropy model proposed by Ballé et al. (), namely Hyperprior, and the autoregressive context-based entropy model adopted by Cheng et al. (), namely AR-CM. Both are trained on natural images with various $\lambda$ settings. We use the pre-trained models provided by CompressAI. For TTA-IC, we use the same value of hyperparameter $\lambda=[0.0018,0.0035,0.0067,0.013,0.025,0.048]$ for latent refinement. The Adam optimizer is utilized to update the latent variables in a learning rate of 1$\times 10^{-3}$ with 2000 iterations. $T$ is empirically set to 20 for MC sampling. We discuss different implementations and hyperparameter settings (e.g., $\beta$) of dropout variational inference in sec. 4.3.

Table: Table 1: Comparison of our method with existing latent refinement approaches. The evaluation is measured in terms of average BD-rate savings (% $\downarrow$) using the respective base models (without test-time adaptation). Smaller values indicate superior performance.

Table: Table 2: Integration of our method with the SOTA TTA-IC approaches. The evaluation is measured in terms of average BD-rate savings (% $\downarrow$) using the respective base models (without test-time adaptation). Smaller values indicate superior performance. ^‡ Matrix decomposition-based adaptor. ^⋆ Entropy efficient adapter. Stages 1 and 2 denote the latent refinement and decoder adaptation phases.

Baselines.
For latent refinement methods, we adopt two common baselines: (i) BLR (Campos et al., ) conducts content adaptation by optimizing the latent variable $y$ using Eq. (14). (Shen et al., )
combine the BLR and decoder adaptation to achieve SOTA TTA-IC performance on screen content images. (ii) HLR (Yang et al., ) concurrently optimizes the latent variable $y$ and side information $z$ by jointly minimizing the amortization and discretization gaps, as formulated in Eq. (15). Tsubota et al. () and Lv et al. () impose the HLR and the PETL to fulfill SOTA TTA-IC performance on various target domains. For HLR, we follow Yang et al. () to use a temperature annealing schedule with defaulted hyperparameters, where $\tau_{0}=0.5$, $c_{0}=0.001$, and $t_{0}=700$.

Plug-and-play Validation. We also replace different latent refinement methods of existing SOTA TTA-IC approaches (Tsubota et al., ; Shen et al., ) with our proposed latent refinement counterpart in a plug-and-play manner. We follow their defaulted hyperparameter settings.

### 4.1 Comparison with Latent Refinement Methods

Rate-Distortion (R-D) Performance. We compare our proposed method with BLR and HLR using two different pre-trained backbone models, including Hyperprior and AR-CM. Note that the model without adaptation is also involved, namely No Adaptation.
The peak signal-to-noise ratio (PSNR) and the data rate in
bits per pixel (bpp) on different quality levels (refers to different $\lambda$ levels) are calculated to evaluate the R-D performance of different methods. Then, the R-D curves can be plotted. The results can be found in Figure 3. We can observe that all latent refinement methods achieve comparable performance, which may be reasonable as the Kodak dataset can be regarded as the in-domain data without domain shifts. The gains of HLR and our proposed method mainly derive from the minimization of the discretization gap compared with BLR. For out-of-domain images such as SIQAD, SCID, CCT, Pixel, and DomainNet, our proposed method outperforms other approaches with a clear margin regardless of different backbones. Especially, a better R-D performance on the low-bit conditions can be observed for AR-CM-based realizations on the Pixel dataset.

Bjøntegaard Delta bit-rate (BD-rate) Performance. We compute the BD-rate (Bjøntegaard, ), which is utilized to compare the
performance of two visual encoders at the same bit rate. A lower
BD rate demonstrates better performance. The baseline model without adaptation is the anchor model for BD-rate calculation. The results are shown in Table 1. Some observations can be summarized. First, the HLR and our proposed method outperform the BLR with a clear margin for the in-domain Kodak dataset, which coincides with Yang et al. ()’ results. However, on the cross-domain tasks, the HLR suffers from significant degradations compared with BLR, where HLR performs below the HLR on three out of five tasks, e.g., SCID, CCT, and DomainNet. This is reasonable as the static entropy bottleneck correlated with the distribution property of natural images cannot model the true distribution of latent variable $z$ well. Finally, our method can achieve obvious gains on all cross-domain tasks, when the proposed distribution regularization is plugged into vanilla HLR.

Figure: (a)
Refer to caption: x11.png

[Uncaptioned image]: x15.png

### 4.2 Comparison with SOTA TTA-IC – Plugging into Distribution Regularization

Moreover, we replace the existing
latent refinement methods used by SOTA TTA-IC approaches with our proposed method. The full R-D curves are in the Appendix. We present the BD-rate in Table 2.
It is obvious that our proposed method can improve the performance of these TTA-IC methods on all tasks. For example, when replacing vanilla HLR adopted by (Tsubota et al., ) with our proposed distribution regularization-based alternative, the BD-rate score improves from -2.94 to -5.46 on the DomainNet dataset for the latent refinement stage. Moreover, such gains are roughly maintained at a similar level when adopted by the decoder adaptation. Due to the benefit of hybrid latent refinement, these gains are significantly enlarged for Shen et al. (). Thus, our proposed method enjoys plug-and-play property, benefiting existing TTA-IC methods.

Figure: Figure 8: Correlation between adaptation performance and adaptation time (using a single NVIDIA GeForce 3090 GPU) on SIQAD.

### 4.3 In-depth Analyses of Our Proposed Method

We gain an in-depth understanding of the proposed distribution regularization from four views.

Why does the proposed distribution regularization improve the R-D performance? (i) From theoretical analyses in section 3.2, the domain shifts and the underlying mismatch between $p({z^{m}|\varphi^{s}})$ and $p(y_{t}^{m}|z_{t}^{m},\theta_{h_{e}}^{s})$ may trigger the deteriorated joint probability approximation $p(y_{t}^{M},z_{t}^{M})$ of true marginal distribution $p(y_{t}^{*})$ for vanilla HLR. By empirical experiments in Figure 4, we observe that it is indeed difficult to jointly optimize latent variable $y$ and side information $z$ for vanilla HLR as shown in Figs. 4(a) and 4(b), where $-\log p(z^{m})$ converges first and degrades with the iterations while $-\log p(y^{m})$ converges well. In contrast, due to the lack of optimization for slide information $z$, there is limited convergence of $-\log p(y^{m})$ for the BLR while $-\log p(z^{m})$ is unchanged. Thus, as shown in Figure
4(c), we can find an obvious degradation for vanilla HLR, i.e., $\Delta H^{0}<\Delta H^{\mathrm{M}}$, reflecting more rate consumption of marginal approximation. This implies that the finally updated joint probability $p(y^{\mathrm{M}},z^{\mathrm{M}})$ is further away from the optimal joint probability $p(y^{*},z^{*})$, and even the initial joint probability $p(y^{0},z^{0})$. (ii) By introducing the proposed distribution regularization, as shown in Figure 4(c), the degradation of joint probability approximation is alleviated well (blue curve) compared with vanilla HLR. Even, our method can achieve good convergence, i.e., $\Delta H^{0}>\Delta H^{\mathrm{M}}$, which is reasonable as we directly minimize $\Delta H$ as shown in Eq. (18).
In short, the proposed distribution regularization encourages the deteriorated joint probability approximation to approach the initial and even unknown optimal ones, leading to good convergence of rate consumption and a better R-D performance in Figure 4.
How about the effectiveness of Bayesian approximation for distribution regularization? To validate the effectiveness of Bayesian approximation for distribution regularization, we first ablate the number of dropout variational inference layers. Intuitively, more dropout layers facilitate a more accurate estimate of posterior distribution, due to more powerful distribution representations. As shown in Table 3, there are marginal gains with the improvement of dropout layers. Moreover, over-regularization, i.e., $\beta=10$, will result in obviously negative effects. Second, we try a deterministic version of our distribution regularization by removing the Bayesian approximation, i.e., assuming $\sigma=1$ without the dropout layer. As we can see, the performance of deterministic $\mathcal{L}_{DR}$ performs significantly below that of the Bayesian approximation-based one, which is reasonable as the over- or below-estimated variance of the posterior distribution is inaccurate compared with the practical dropout variational inference in Monte Carlo sampling. Finally, we observe negligible gains of the BBC on cross-domain image compression, which coincides with our discussions in section 3.3.
How about the adaptation efficiency compared with baseline methods? As illustrated in Table 8, we can observe that our proposed method has obvious performance gains in different adaptation steps. In contrast, the additional adaptation time taken by ours is mild in the same adaptation step, which is reasonable as a single instance can parallelly sample $T$ masked weights by one inference in a batch of repeated instances for Bayesian approximation. Moreover, using a more computationally efficient GPU can further reduce the adaptation time in the future.
What is the ability to generalize for larger distribution discrepancy?
We explore whether our proposed method can be scalable to more challenging medical images, where the image distribution of medical images is quite different from natural images. The qualitative and quantitative results in Figures. 5 and 6 show our method can improve R-D performance with better texture preservation.

## 5 Conclusion

We have approached an advanced latent refinement method by tailoring the vanilla HLR method designed for in-domain inference improvement to cross-domain cases. Specifically, we have provided a theoretical analysis to uncover the degradation reason of rate cost for the vanilla HLR, i.e., underlying mismatch between refined Gaussian conditional and hyperprior distributions may trigger the deteriorated joint probability approximation of marginal distribution, leading to increased rate consumption. Then, we introduce a Bayesian approximation-endowed distribution regularization to encourage learning better joint probability approximation in a plug-and-play manner. Extensive experiments demonstrate that our proposed method can achieve promising performance.

## Appendix A Details of adopted datasets

By following previous literature (Lv et al., ; Tsubota et al., ; Shen et al., ), we collect six different datasets with four types of image styles to comprehensively evaluate the R-D performance of different approaches on cross-domain TTA-IC tasks, including natural image (Kodak(^2^22https://r0k.us/graphics/kodak/)), screen content image (SIQAD Yang et al. (), SCID (Ni et al., ), CCT (Min et al., )), pixel-style gaming image (Lv et al. ()’ self-collected), and painting image (DomainNet (Peng et al., )) datasets. The details of the used dataset can be found in the Table 4. Specifically, we consider the natural image dataset as in-domain evaluations, and others as cross-domain evaluations.

Table: Table 4: The datasets for evaluation. The symbols $*$ and ${\dagger}$ denote in- and cross-domain datasets, respectively.

## Appendix B Comparison with SOTA TTA-IC Methods

We provide the full R-D curves in Figure 9 when we compared our proposed method with SOTA TTA-IC methods. Note that our proposed method is based on (Tsubota et al., ) with our proposed distribution regularization. We can observe that all latent refinement methods achieve comparable performance, which may be reasonable as the Kodak dataset can be regarded as the in-domain data without domain shifts. The gains of HLR and our proposed method mainly derive from the minimization of the discretization gap compared with BLR. For out-of-domain images such as SIQAD, SCID, CCT, Pixel, and DomainNet, our proposed method outperforms other approaches with a clear margin regardless of different backbones. Especially, a better R-D performance on the low-bit conditions can be observed for AR-CM-based realizations on the Pixel dataset.

Figure: (a)
Refer to caption: x17.png

## Appendix C Connection with bit-back coding

1) In-domain v.s. Cross-domain: Both BBC (Townsend et al., ; Ruan et al., ; Ho et al., ) and our proposed method derive from the joint probability approximation of marginal distribution. However, BBC usually specializes in in-domain image compression to narrow the marginalization gap, i.e., transforming learned optimal joint probability to true marginal probability at compression time.
Instead, in the context of cross-domain image compression, such optimal joint probability does not hold due to mismatched encoding distribution (e.g., entropy bottleneck), as discussed in Eqs. (13) and (16).

2) Minimizing posterior probability v.s. Maximizing posterior probability: BBC minimizes the second term of the first line of Eq. (4) as the true entropy of the marginal distribution, i.e.,

$$ $\mathcal{L}_{BBC}=-\log p(y_{t}^{m}|z_{t}^{m},\theta_{h_{e}}^{s})-\log p(z_{t} ^{m}|\varphi^{s})-(-\log p(z^{m}_{t}|y_{t}^{m}))+\lambda(-\log p(x_{t}|y_{t}^{ m})),$ (22) $$

where the third term corresponds to minimizing the posterior probability based on the in-domain optimal joint probability assumption of BBC. In contrast, our proposed method focuses on refining deteriorated joint probability to optimal one by minimizing the extra rate consumption of marginal approximation, as discussed in Eq. (18).

3) Performance and implementation differences: Our experiments in Table 3 observe negligible gains of BBC on cross-domain image compression. More importantly, in order to acquire distribution property, our proposed distribution regularization relies on more flexible variational Bayesian inference rather than introducing additional networks like BBC (Yang et al., ).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: * Please address the weakness above.
* The writing is a bit difficult to understand, especially in the discussion parts of both the methodology and experiment sections
soundness: 2
strengths: 1. The proposed method can be applied without altering any model parameters, making optimization easy and immune to catastrophic forgetting.
2. The paper tries providing detailed discussions on the proposed method, both in a principled and empirical way.
3. The experimental results consistently demonstrate the effectiveness of the proposed method in cross-domain image compression across different settings: (1) with only latent refinement, (2) integrated with state-of-the-art TTA-IC methods, which further include decoder adaptation, and (3) on medical images.
summary: This paper addresses a latent refinement techqniue in test-time adaptation methods for image compression.
Especially, the paper focuses on methods that use a hyper-prior model, where the image is encoded into a latent variable $y$ and a hyper-latent variable $z$ in a hierarchical way.
The paper analyzes the issue of recent latent refinement methods not generalizing well in cross-domain settings, arguing the cause to a mismatch between the hyper-prior $p(z)$ and the gaussian conditional $p(y|z)$ used in refinement methods.
Based on this conjecture, the paper proposes a distribution regularization term that uses dropout variational inference, claiming it can improve probability approximation. 
Experimental results show that the proposed method continuously improves latent refinement and demonstrates effectiveness even when integrated into state-of-the-art methods that jointly utilize latent and decoder refinement.
weaknesses: Most importantly, as i understand, the theoretical analysis and subsequent derivation of the proposed method, which are claimed as major contribution in this paper, contain many unclear and misleading aspects. 
1. (L173, 193) The definition of "optimal probability representation" is not clearly defined, throughout the paper.
2. The entropy function, i.e., $H(X)=-\mathbb{E}_{x\sim p(x)}[\log p(x)]$, is interpreted in the form of log probability, and it seems that the expectation term is not considered in the analysis and derivation.
3. In the Proposition 1, assuming an optimal joint probability approximation of the true marginal distribution $p(y^*)=\int p(y,z)dz$, the proposition simply interprets $H(Y,Z) - H(Y^*) = H(Z|Y) + H(Y) - H(Y^*) = H(Z|Y)$ when $H(Y) = H(Y^*)$. However, the assumption $p(y^*)=\int p(y,z)dz$ may not induce equations (7,8), which can be misleading.
4. In the proof of Proposition 2, the authors argue that the entropy bottleneck $p(z_t)$ and the posterior $p(y_t|z_t)$ are learned from the source dataset (L212, source image correlated) and may not work effectively. While this makes sense, it is not a theoretical analysis, in my point of view.
5. For Corollary 1, it is unclear whether $z^*$ and $y^*$ in equation 13 correspond to in-domain or cross-domain cases. In the cross-domain case with $y^*_t$ and $z_t^*$ as in Proposition 2, equation 13 is as follows: $\triangle H = H(Y, Z) - H(Y^*_t) > H(Z^*|Y^*)$, which holds according to equations (11, 12) only if $H(Y^*_t)=H(Y^*)$ and $H(Z^*_t|Y^*_t)=H(Z^*|Y^*)$. However, there is no guarantee for the condition.
6. Importantly, in Section 3.3, the paper proposes distribution regularization using equation 13, interpreting the first three terms in equation 17 as corresponding to $H(Y|Z), H(Z), H(Z^*|Y^*)$ (the three terms at the front) in equation 13. However, it is unclear how $H(Z^*|Y^*)=\mathbb{E}_{y^*,z^*\sim p(y^*,z^*)}[p(z^*|y^*)]$ leads to $-\log p(z^m_t|y^m_t)$ in the optimization objective. Note that $H(Z^*|Y^*)$ includes expectation from $p(y^*,z^*)$.

In conclusion, the theoretical analysis in this paper seems to have many over claimed or misclaimed elements.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. In Table 3, the results indicate that increasing the number of dropout layers can negatively impact image compression performance. Please add further discussion on how to effectively approximate the distribution regularization to avoid such performance degradation.
soundness: 3
strengths: 1. The authors offer a theoretical analysis highlighting limitations of existing methods, which informs their introduction of a distribution regularization technique. This technique effectively adapts the compression model to test data with domain shifts.
2. They provide detailed analysis results on the entropy curves of different probabilities about different methods. The results in Figure 4 demonstrate that the proposed method can lead to a good convergence of rate consumption and better R-D performance.
3. Experimental results demonstrate that the proposed method can be easily integrated with existing methods to enhance R-D performance across cross-domain image datasets.
summary: The authors propose a new latent refinement method to enhance the rate-distortion (R-D) performance for test-time adaptation image compression. They theoretically analyze that existing methods require higher rate consumption to estimate the marginal probability of the latent variables in cross-domain scenarios. To address this, the authors incorporate a new distribution regularization term into the R-D objective, promoting a better approximation of the latent representation. Experimental results across six cross-domain datasets demonstrate the effectiveness of the proposed approach, showing its superiority over existing methods. Please see my detailed comments below.
weaknesses: 1. The test datasets used in the experiments contain a limited number of images, which may affect the robustness of the results. To strengthen the findings, please consider providing additional results on larger-scale datasets.
2. It would be helpful to include a preliminary section explaining the image compression process before the theoretical analysis of existing methods. This section could clarify the steps involved in obtaining the latent variable y and hyper-latent variable z (side information), enhancing readers' understanding of the compression model and the subsequent theoretical analysis.
3. In addition to the adaptation cost of different methods, it is better to provide the compression cost of the proposed methods, which would offer a more comprehensive view of its performance. 
4. The authors do not provide a limitation analysis of the proposed methods. For instance, the long adaptation time may limit the proposed methods to be used in real-time applications.
5. More discussion on recent test-time adaptation works would be beneficial to place this study in context. Relevant works to consider include, but are not limited to, [A-E].

[A] Test-Time Training Can Close the Natural Distribution Shift Performance Gap in Deep Learning Based Compressed Sensing, ICML 2022.

[B] Efficient Test-Time Model Adaptation without Forgetting, ICML 2022.

[C] Tent: Fully Test-Time Adaptation by Entropy Minimization, ICLR 2021.

[D] Efficient Test-Time Adaptation for Super-Resolution with Second-Order Degradation and Reconstruction, NeurIPS 2023.

[E] Test Time Adaptation for Blind Image Quality Assessment, ICCV 2023.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: Will the authors make their code public to assist the community upon acceptance?
soundness: 3
strengths: 1. The paper effectively identifies the limitations of existing Hybrid Latent Refinement (HLR) methods in cross-domain image compression, particularly highlighting the mismatch between refined Gaussian conditionals and hyperprior distributions.

2. It provides a thorough theoretical analysis using marginalization approximation, establishing a solid foundation for the proposed improvements.

3. The introduction of a Bayesian approximation-based distribution regularization technique successfully addresses the identified mismatch, enhancing joint probability approximation. 

4. The method shows improvements in rate-distortion (R-D) performance compared to other latent refinement approaches, validating its effectiveness.
summary: Current Test-Time Adaptation Image Compression (TTA-IC) methods improve rate-distortion (R-D) performance on cross-domain tasks by refining both latent variables and decoders in a two-step process. However, latent refinement has not been specifically optimized for cross-domain scenarios. The authors conduct theoretical analyses to identify a mismatch between refined Gaussian conditionals and hyperprior distributions in the vanilla Hybrid Latent Refinement (HLR), leading to poorer joint probability approximations and higher rate consumption. To address this issue, they introduce a Bayesian approximation-based distribution regularization technique that enhances joint probability modeling in a plug-and-play manner. Extensive experiments across six in-domain and cross-domain datasets demonstrate that the proposed method outperforms existing latent refinement approaches in R-D performance.
weaknesses: 1. The paper does not include a comparison of GPU memory usage and GFLOPs, which are crucial metrics for evaluating the complexity of various methods. Including these metrics would illustrate the trade-off between performance gains and computational complexity more clearly. It would be beneficial to present memory usage across different datasets, especially since DVI is introduced during optimization, making it reasonable to assess the additional complexity incurred.

2. In [1], the posterior distribution of $\hat{z}$ is assumed to be a uniform distribution. It is unclear whether directly inferring $\hat{z}$ through a hyper analysis transform in out-of-distribution (OOD) scenarios is the better choice? Because this approach not only avoids incurring a larger $\Delta H$ but also naturally matches its posterior distribution during pre-training. 

3. The paper introduces a regularization term based on HLR in TTA-IC. While this contribution is valuable, its impact on the community appears moderate.

[1] Ballé, J., Minnen, D., Singh, S., Hwang, S. J., & Johnston, N. (2018). Variational image compression with a scale hyperprior. arXiv preprint arXiv:1802.01436.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: The compared methods are too old. More popular image reconstruction/compression datasets should be considered for comparisons.
soundness: 2
strengths: 1.	The authors propose their method from the perspective of latent refinement, which hasn’t been fully investigated. Compared with HLR, the method proposed by the author greatly improves the performance cross-domain compression tasks.
2.	The authors provide sufficient theoretical analysis on the method proposed in the article.
summary: This article investigates test-time adaption image compression method on cross-domain compression tasks from the perspective of latent refinement, by extending hybrid latent refinement and introducing a simple Bayesian approximation-endowed distribution regularization. The authors provide theoretical analysis and experimental results to demonstrate the performance of this method. My detailed comments are as follows.
weaknesses: 1.	The experimental results of this plug-and-play method are relatively weak, which cannot illustrate the generalization ability of this method on other TTA-IC approaches. It would be better to apply this method to more baseline approaches.
2.	It would be better to report the image compression results on top of popular image reconstruction benchmarks, such as Set5, Set14, BSD100, Urban100, etc.
3.	This paper is hard to follow and the writing could be further improved. It would be better to explicitly highlight the key contributions of this paper.
4.	The compared methods BLR and HLR are too old since both are published before 2021. It would be better to compare with more recent work, such as [A].
[A] Downscaled representation matters: Improving image rescaling with collaborative downscaled images, ICCV 2023.

---

confidence: 1
contribution: 2
initial_rating: 6
presentation: 3
questions: None
soundness: 3
strengths: This research topic of this work seems a bit a minority (image compression + test-time adaptation) and contains a lot of background knowledge. After hard attempts, I think I still can't give an accurate evaluation. It is recommended that AC find another highly relevant reviewer in order to make a more comprehensive judgment.
summary: The author propose a novel test-time adaptation framework for image compression through directly refine the latent variables without altering any model parameters.
weaknesses: None

Meta-review: This paper addresses test-time adaptation for learned image compression (TTA-IC) in the face of cross-domain inputs. It builds upon a latent-refinement approach originally designed for in-domain scenarios and identifies a key issue: a fundamental mismatch between the refined Gaussian conditionals and hyperprior distributions when adapting to out-of-distribution images. This mismatch leads to suboptimal joint probability approximations and increased bit rates. To remedy this, the authors introduce a Bayesian approximation-based distribution regularization, which better aligns latent distributions with their hyperpriors, resulting in improved rate-distortion (R-D) performance on cross-domain inputs.

Initial reviewer feedback was mixed. Major concerns included:

1. **Clarity and Accessibility:** The motivations and theoretical derivations were challenging to follow, especially for readers without substantial background in learned compression.
2. **Comparisons to General TTA Methods:** The paper lacked explicit connections and comparisons to broader test-time adaptation techniques.
3. **Computational Complexity:** Reviewers requested more details on computational overhead and complexity.
4. **Dataset Diversity:** The evaluation lacked experiments on a broader range of datasets.

During the rebuttal phase, the authors provided detailed clarifications, introduced additional experiments on more diverse datasets, and offered complexity analyses. These responses addressed most concerns, and by the end of the discussion, all reviewers agreed on a borderline acceptance.

The AC read paper, reviews, and discussions, and concurs with the reviewers. The additional clarifications and experiment results presented in the rebuttal adequately addresses the reviewers’ concerns. However, the AC also agrees with the reviewers that the paper can be benefited from a significant revision by improving its clarity and self-containedness; many of the mathematical derivations are either trivial or from the prior works, and can be greatly simplified  (e.g., Section 3.1.). Introducing more descriptions on background (e.g., what information the side-information is carrying and be used in decoding) can make the paper more easily accessable to non-experts of the domain. Overall, the AC believes that the paper provides a meaningful theoretical insight and demonstrates tangible performance improvements, hence recommends acceptance. The authors are encouraged to incorporate the clarifications and new results presented in the rebuttal into the camera-ready version for better clarity and accessibility.

Decision: Accept (Poster)