Accelerating Learned Image Compression Through Modeling Neural Training Dynamics
Authors: Yichi Zhang
Zhihao Duan
Yuning Huang
Fengqing Zhu
Venue: Accepted by TMLR ()

Abstract: As learned image compression (LIC) methods become increasingly computationally demanding, enhancing their training efficiency is crucial. This paper takes a step forward in accelerating the training of LIC methods by modeling the neural training dynamics. We first propose a Sensitivity-aware True and Dummy Embedding Training mechanism (STDET) that clusters LIC model parameters into few separate modes where parameters are expressed as affine transformations of reference parameters within the same mode. By further utilizing the stable intra-mode correlations throughout training and parameter sensitivities, we gradually embed non-reference parameters, reducing the number of trainable parameters. Additionally, we incorporate a Sampling-then-Moving Average (SMA) technique, interpolating sampled weights from stochastic gradient descent (SGD) training to obtain the moving average weights, ensuring smooth temporal behavior and minimizing training state variances. Overall, our method significantly reduces training space dimensions and the number of trainable parameters without sacrificing model performance, thus accelerating model convergence. We also provide a theoretical analysis on the Noisy quadratic model, showing that the proposed method achieves a lower training variance than standard SGD. Our approach offers valuable insights for further developing efficient training methods for LICs.

Published in Transactions on Machine Learning Research (04/2025)
Accelerating Learned Image Compression Through Modeling Neural Training Dynamics
Yichi Zhang Purdue University
zhan50968purdue.com edu
Zhihao Duan Purdue University
duan.008purduec.ommm edu
Yuning Huang Purdue University
huan 1 788 &purdue. edu
Fengqing Zhu Purduc University
zhuOSpardue. edu
Reviewed on OpenReview: https: // openreview. net/ forum? id= nannu450f5
Abstract
As learned image compression (LIC)) methods become increasingly computationally demand- ing, enhancing their training efficiency is crucial. This paper takes a step forward in accel- erating the training of LIC methods by modeling the neural training dynamics. We first propose a Sensitivity-aware True and Dummy Embedding Training mechanism (STDET) that clusters LIC model parameters into few separate modes where parameters are expressed as afline transformations of reference parameters within the same mode. By further utiliz- ing the stable intra- intra-mode correlations throughout training and parameter sensitivities, gradually embed non-reference parameters, reducing the number of trainable parameters. Additionally, we incorporate ing sampled weights from stochastic gradient descent. (SGD) training to obtain the moving average weights, ensuring smooth temporal behavior and minimizing training state vari- ances. Overall, our method significantly roduces training space dimensions and the number of trainable parameters without sacrificing model performance, thus accelerating model con- vergence. We also provide theoretical analysis the Noisy quadratic model, showing that the proposed method achieves lower training variance than standard SGD. Our approach offers valuable insights for further developing efficient training methods for LICs.
Average (SMA) technique, interpolat-
1 Introduction
With the widespread adoption of high-resolution cameras and the increasing prevalence of image-centric social media platforms and digital galleries, images have become a predominant form of media in daily life. The large file sizes of these high-resolution images place considerable demands on both transmission bandwidth and storage capacity. To address these challenges, lossy image compression has emerged critical technique for achieving efficient visual communication.
Recently, learned image compression (LIC) methods have garnered significant attention due to their remark-
able performance (He et al. 2022 Liu et al. 2023 Li et al. 2024a Zhang et al. 2024b 2025
Despite these advancements, LICs currently face a core challenge: high training complexity. Designing a new method requires substantial amount of computational resources, which severely hinders the emergence of new approaches. Modern LICs typically feature a vast number of parameters that, while enhancing performance, also introduce critical issues, such overwhelming computational burden and prolonged

---

Published in Transactions on Machine Learning Research (04/2025)
1.0
Mode 1
Epoch
2000
0.8
Change
20
40
60
80
100
0%
3.85%
31.644%
41.61%
61.70%
72.84%
81.70%
4000
iMoee
0.6
1% 2%
0.21%
9.22%
8.01%
7.38%
6.00%
5.81%
2% 5%
0.62%
19.049%
18.35%
10.92%
5.24%
4.02%
6000
3%
10%
1.04%
15.78%
12.87%
9.609
7.69%
4.11%
0.4
10%
20%
2.12%
11.10%
9.90%
5.21%
4.77%
2.44%
20%
50%
7.29%
10.89%
7.99%
4.333%
2.93%
1 1.65%
8000
0.2
50%
100%
84.87%
2.33%
1.27%
0.86%
0.53%
0.27%
2000
4000
6000
8000
Figure 1: ELIC He et al. 2022) model, X 0.0018. (a) Clustered correlation matrix of sampled 10k parameter trajectories trained on COC02017 dataset (Lin et al. 2014), decomposed to 10 modes. The diagonal block structure indicates high correlations of the parameters within each mode, which shows the accurate representation of the proposed method. (b) The table shows the percentage of affine coefficients [k.)/= relative to the total number of coefficients, grouped by relative change intervals, at different epochs during the training of the ELIC model. These relative changes are mcasured against the final cocfficient values at epoch 120. The rows correspond the relative change intervals (0% 1%, 1% 2%, 50% 100%), indicating how much the coefficients have relatively changed compared to their values at epoch 120. The columns represent specific epochs (0, 20, 40, 60, 80, 100). The percentages indicate the proportion of coefficients that fall within each relative change interval at the corresponding epoch. The table reveals that most coefficients either remain stable or undergo only minor changes as training progresses from epoch 20 to 120. Notably, the proportion of coefficients in the 0% to 1% interval increases significantly from 31.64% at epoch 20 to 81.70% al epoch 100, indicating a marked stabilization of the affine coefficients over time.
training times. For instance, training the FLIC models (Li et al. 2024a takes up to 520 hours (21 days) using a single 4090 GPU (Tab. while the design process of the FLIC models likely requires even more GPU resources, as it involves extensive experiments, including hyper-parameter tuning, structural adjustments, and other iterative processes. If the training speed of these models is not improved, the time required to develop new methods may become prohibilive. Therefore, developing stralegies to reduce training times is of paramount importance.
While there have been efforts to design more efficient LIC methods (c.g., (He et al. 2022 Zhang et al. 2024c) ).
improving the training efficiency of LIC methods has not received significant allention. A promising and effective direction to train LICs within low-dimensional subspaces, as has been examined in the context of
image classification tasks (Li et al.] 2018 Gressmann et al.) 2020 Li et al. 2022b Brokman et al. 2024). Instead of exploiting the full parameter space, which can be very large (in millions or even billions), subspace training constrains the training trajectory to a low-dimensional subspace. These approaches are based on the hypothesis that "learned wver-paaameterized models reside in a low intrinsic dimension" (Li et al.) 2018 Aghajanyan et al. 2021). ). In such subspaces, the degree of freedom for training is substantially reduced (to dozens or hundreds), leading to many favorable properties, such as fast convergence (Li et al. 2023b robust
performance (Li ct al. 2022b), and thcorctical insights (Li al. 2018 2022b
In this work, we present an innovative approach to improve training efficiency by modeling neural training dynamics in a low dimension space. Our proposed method draws inspiration from Dynamic Mode Decom- position (DMD) (Schmid 2010 2022 Brokman et al.) 2024 Mudrik et al.) 2024), a technique traditionally used in fluid dynamics to decompose complex systems into simpler modes. We exlend this concept. to LIC by proposing the Sensitivity aware True and Dummy Embedding Training (STDET) mechanism. The fun- damental principle of our approach is the recognition that LIC parameters are highly correlated and can be effectively represented by few distinct "Modes" (see Fig. D which capture their intrinsic dimensions. Within each mode, the parameters are aptly modeled through an affine transformation of the reference parameters,
2

---

Published in Transactions on Machine Learning Research (04/2025)
given their significant correlation. We also observe that after the initial head-stage training, the relative changes of the affine coefficients become negligible, and the coefficients tend to remain stable throughout the training, as illustrated in Fig. [ It is evident that the majority of the coefficients exhibit 0% to 1% relative changes, with the percentage in this interval continuing to increase. This stability permits a focused update of reference parameters and then allows for the "embedding" of non-reference parameters that have relatively invariant coefficients. Once embodded, these parameters are no longer trainable. Subsequently, updates of these embedded parameters are done solely through the fixed coefficients affine transformation of the reference parameters. Non-embedded and reference parameters are still updated via normal training, This mechanism significantly reduces the number of training parameters and the dimension of LIC models in practice along the training period, thereby expediting convergence, as demonstrated in Fig. 2 Ideally, the training parameters and dimensions could be reduced to the number of "Modes". Addilionally, Lo ensure smooth temporal behaviors and minimize training variances, which the core requirement of STDET, we introduce the well-known moving average tochniques the training phase. Our proposed Sampling-tiee then- Moving Average (SMA) method interpolates the periodically sampled parameter stales from SGD training to derive moving averaged parameters, thus enhancing stability and reducing the variance of final model states. various complex LIC models different
Our method is substantiated by comprehensive experiments domains and comparisons with other efficient training methods. The superiority of the proposed method is also validated through theoretical analysis on the noisy quadratic model.
Our contributions are summarized as follows:
We propose the Sensitivity-aware True and Dummy Embedding Training (STDET) mechanism, which approximates the SGD training of LICs in a low-dimensional space (Sec. 3.2
We introduce Sampling-then-Moon Average (SMA) to ensure the smooth temporal behavior re- quired by STDET, reducing the variances of final states and enhancing training stability (Sec. 3.3
We provide a theorctical analysis using the noisy quadratic model to demonstrate the low training variance the proposed method (Appendix Sec. A.6
Overall, our proposed method significantly accelerates the training of LICs while reducing the num- ber of trainable parameters and dimensions without compromising performance (Sec. 4.2 Sec. 4.3. Appendix Sec. A.1
EUC 56D10.3444)
TCM-S 56D10.34311
FLCC 3GD(0.3413)
ProposedEO.
Proposed(0.3422
0.45
Proposedio.
0.44
8 1000
R-0 100s
0.45
8-0 100s
0.39
0.36
Epoch
Eparh
Epach
(a) ELIC
(b) TCM-S
(c) FLIC
Figure 2:
Testing loss comparison of various methods. Please zoom in for more details. The proposed
method clearly converges much faster than standard SGD on varions LICs. Additionally, as shown in the upper right corner, our method achieves a similar final convergence compared to SGD. 0.0018, Testing R-D loss A- 2552. MSE + Bpp. Evaluated on the Kodak datasct.
3

---

Published in Transactions on Machine Learning Research (04/2025)
2 Related work
2.1 Efficient learned image compression
To address the issue of high compulational complexity in learned image compression, various approaches have been proposed: Minnen & Singh (2020) develop a channel-wise autoregressive model to capture channel relationships instead of spatial ones. He ct al. [2021] propose a parallclizable two-pass checkerboard model to accelerate spatial autoregressive models. [Tao et al. (2023) introduce dynamic transform routing to activate optimally-sized sub-CAEs within a slimmable supernet, conserving computational resources. Duan et al. (2023) incorporate a hierarchical VAE for probabilistic modeling, which generalizes autoregressive models. Guo-Hus Hua et al. 2023) propose an efficient singlo-model variable-bit cariable-bit-rate bit-rate codec with mask decay capable of running 30 FPS with 768x512 input images. Kamisli el. al. (2024) implement variable-bit-rate codec based on multi-objective optimization. Yang & Mandd (2023) adopt shallow or linear decoding transforms to
reduce decoding complexity. Ali et al. 2023 introduce correlation loss that forces the latents to be spatially
decorrelated, fitting them to an independent probability model and eliminating the need for autoregressive models. Minnen & Johnston (2023 conduct a auue-dditortiin-cmmittaaat study, leading to a family model architectures that optimize the trade-off between computational requirements and R-D performance, [Zhang ct al. (2024b) propose contextual clustering to replace computationally intensive self-attention mechanisms. Zhang et al. (2024c) test various transforms and context models to identify a series of efficient LIC models, However, these works primarily focus on test-time efficiency and often overlook the significant computational resources consumed during the training phase. Training complex LICs is resource-intensive and optimizing training efficiency is crucial.
2.2 Low dimension training/fine-tunigg of modern neural networks
Training or fine-tuning neural networks in low-dimensional spaces has garnered significant attention recently. Li et al. (2018) propose training neural networks in random subspaces of the parameter space identify the minimum dimension required for effective solutions. [Gressmann et al. (2020) enhance training performance in random bases by considering layerwise structures and redrawing the random bases iteratively. Li et al. (202bb) utilizes top eigenvectors of the covariance matrix to span the training space. Additionally, Li et al. (202cc apply subspace training to adversarial training, mitigating catastrophic and robust overfitting. Fur- ther advancements include Li et al. (2023b), which propase trainable weight averaging to optimize historical solutions in the reduced subspace, and Aghajaayann et al. (2021), which demonstrate that learned over- parameterized models reside in a low intrinsic dimension. Hu et al. (2022) introduce Low-Rank Adaptation based on the premise that weight changes during model adaptation have low "intrinsic rank". Similarly, Ding et al. (2023) present delta-tuning method optimizing only a small portion of model parameters, while Jia el al. (0022) propose Visual Prompt Tuning, introducing less than 1% of trainable parameters in the input space. Barbano et al. (202)) constrain deep image prior optimization to a sparse linear subspace of parameters, employing a synergy of dimensionality reduction techniques and second-order optimization methods. These works highlight the elegant performance of the low-dimensional hypothesis, which has not yet. been thoroughly explored the context of learned image compression.
3 Proposed method
3.1 Preliminaries learned image compression
Currently, LIC' predominantly utilize the non-linear transform coding framework (Ballé et al., 2020 where the transform function and entropy estimation function are parameterized by neural networks. This frame- work encodes data into a discrete representation for decorrelation and energy compression, and then an entropy model is used to estimale the probability distribution of the discrete representation for entropy coding, as illustrated in Fig. 3 The process begins with an autoencoder applying a learned nonlinear anal- ysis transform, Ja, to map the input image x to a lower-dimension latent variable y = (z). This latent variable y then quantized to & using a quantization function Q and encoded into a bitstream via a lossless codec (Duda] 2009), resulting in a much smaller file size. The decoder then reads the bitstream of ÿ, ap-
4

---

Published in Transactions on Machine Learning Research (04/2025)
plies the synthesis transform Is, and reconstructs the image as λ, with the goal of minimizing the distortion D(r, Rt.. Minimizing the entropy of & involves learning its probability distribution through entropy modeling, which is achieved using an entropy model P that includes both forward and backward adaptation methods. The forward adaptation employs a hyperprior estimator, which is another autoencoder with its own hyper analysis transform ha and hyper synthesis transform hg. This hyperprior effectively captures spatial depen- dencies in the latent representation y and generates separately encoded latent variable &, which sent the decoder. A factorized density model (Ballé et al. 2018) used to learn local histograms, estimating the probability mass ps(2)1yy) with model parameters V The backward adaptation estimates the entropy parameters of the current latent element fi based on previously coded elements fei, where i is the latent element index, exploring the redundancy between latent elements. This is achieved through a network that operates in an autoregressive manner over the spatial dimension, channel dimension, or a combination of both. The outputs from and the hyperprior are then used to parameterize the conditional distribution Py(0))) via an entropy parameters network 9e- The LIC framework can be formulated as:
9 (x;
f go ox),
(1)
2 = Quaa(V fa)),
Pa. (0) + (90( 0b), h2 (2)
with the Lagrange multiplier- based rate-distortion (R-D) loss function L for end-to-end training:
C(6,0, 1)) R(y) R(2) + ((x, a).
(2)
[l (0) Eloog (p=(3)+)) + D(,,
where D(x, .) measures the distortion (e.g., MSE) between the original image and the reconstructed image 2. R2) and R(ÿ) represent the rate of & and i, respectively.
Analysis
Hyper Analysis
X
Quantization
Quantization
B
h
Backward
Factorized
R())
R(2)
Entropy coding
Entropy coding
10101101
10101101
Entropy
g,
f
Synthesis
Hyper Synthesis
Figure typical pipeline learned image compression.
3.2
Sensitivity-aware True and Dummy Embedding Training Mechanism
3.2.1
Decomposition of non-linear dynamical systems:
As shown in Sec. and Fig. 3 LICs heavily rely on neural networks for transform functions and distribution parameters estimation, which are inherently nonlinear and complex systems. However, recent studics {Lusch] et al. 2018 Razzhigaev al.) 2024) demonstrate that, under certain conditions, neural networks exhibit intrinsic linear behavior. This linearity be effectively harnessed using techniques such Dynamic Mode

---

Published in Transactions on Machine Learning Research (04/2025)
Brokman et al., 2024 Mudrik al. 2024) and Koopman theory (Dogra & Redman 2020 Brunton et al. 2022), which provide linear approximations of the network's dynamics. By leveraging these methods, the analysis and efficiency of various tasks can be significantly improved. Approximating nonlinear LICs with linear operations could offer numerous benefits, inchuding laster convergence, better interpretability, and greater stability. Therefore, there is substantial potential in employing linear approximations for these complex LICs to achieve enhanced efficiency.
Decomposition (DMD) (Schmid 2010; 2022
We select variant of DMD as the core technique for the proposed method due to its simpler and more intuitive representation compared to Koopman theory methods. Let W (t) M U (t)m denote the set of LIC's trainable parameters, where i is the index of the non-reference parameters, m is the index of reference parameters, represents the training time, and N is the total number of parameters. We refer to W as the set of 'trajectories," with each (y(t)/wm(t (t) representing an individual 'trajectory.' DMD methods aim to represent the dynamics of complex models by decomposing them into several modes, where each mode is captured by a concise set of reference trajectories wm This can be expressed as: (I)()) 23 Σm (m) + di, where kg,m and d: are scalars. Specifically, we employ Correlation Mode Decomposition (CMD) (Brokman et al. 2024), which further simplifies this representation:
wx(t) ss (t) + di,
(3)
where k; and d are scalar affine coefficients associated with wy- CMD operates under the assumption that complex system can be represented by several modes, and within each mode, an individual parameter can be effectively represented by the affine transformation of the mode reference parameter. This assumption also holds true in the training of LICs, as shown in Fig, I CMD can be seen special, simplified case of DMDs, where the theoretical dimension corresponds to the number of reference trajectorics. With CMD, we achieve linear approximation of complex nonlinear LICs, which present desirable features.
Let's consider a discrete-tine setting in the training process, t E [L....,E], where E is the number of epochs. Let Nn be the number of trajectorics in mode m. Then each parameter Wm, w; R the parameter set W RNXX the mode m parameter set Wm E RNMXE. We use (1) represent at time 1, and e represent up to time t. We consider 1D vectors such u E R to be row vectors, and define the centering operation ū= == u E E u(t), outer product (u, v) = we'". euclidean norm ||]|| = Vuu? and (ū,6) Let K, D. E N such that K(t)= = ki, D(1) = d,, and let Km, Dm corr(u, v) = |لा|
Notation:
E R. be the parts
Wm I
of K, corresponding to the parameters mode m. Denote matrix form tm
E R2XE where
i = (1,1....,11) 11 E E and Km_ [KT, Dm] E RRR. x2, In this context, Eq. reads as:
Wm RJ Km
(4)
We then use Eq. ! and these notations to model the LICs trainable parameters. To clarify the term in the following section, we have two types of parameters: reference and non-reference. Non-reference parameters are further divided into embeddable and non-embeddable.
3.2.2 Proposed STDET:
The proposed STDET mechanism builds on the notation and concepts defined above. We perform CMD after an initial head-stage training to identify the reference parameters, extract modes, and select the best hyperparameters. We then estimate the sensitivity of the LIC parameters to determine which parameters are "embeddable". Next, the proposed STDET iteratively updates the alfine coefficients based on the updated neural state. Subsequently, stability of the coefficients and the parameter sensitivity. Our method reduces both the training space dimension and the number of trainable parameters as training moves forward, dynamically optimizing the overall training process.
embed the non-reference parameters every few epochs according to the
Step 1: Mode decomposition.
We select reference parameters after the initial head-stage training. The LIC is trained normally for a predefined number of epochs F to obtain the trajectories warr Wr up to epoch F. We then perform
6

---

Published in Transactions on Machine Learning Research (04/2025)
Correlation Mode Decomposition (CMD) [Bokmman et al., 2024) to get the reference parameters Wm.F, the mode of each Wi.F E WWr, and the affine coefficients Km (P) at epoch F based on Eq. 4 We select the hyperparamnterss (í.e., the number of modes used) based on the traversed instant CMD performance, as detailed in Sec. 4.4.4 for each model. This CMD determination and association is performed only once.
Once F is associated with mode m and reference parameter Fm,FF Wi remains in mode m with reference parameter Wm for the entire training process.
Step 2:
Parameter Sensitivity Estimation. Before starting to embed the non-reference parameters in LIC models, it is essential to determine which parameters are "embeddable" some parameters may be extremely sensitive to perturbations, leading to significant performance drops (Weng et al. 2020 Zhang et al. 2022). We do not embed these parameters. Accurately assessing parameter sensitivity is challenging due to the complexity involved in individually perturbing each parameter and measuring the prediction error. Given the lack of generally reliable estimation methods (Yvinee et al., 2023), we employ combination of accurate layer-wise assessment and rough parameter-wise cstimation to rank the scnsitivity of the LIC paramcters.
We begin by individually perturbing each layer and measuring the impact on the distorrit (R-D) loss (with n 256 in practice).
using randomly sampled portion of the training dataset X' = (z1,-.. ,.n) .nn Different strategies are employed for evaluating analysis transform Ia. synthesis transform 9s and hyper analysis transform ha, hyper synthesis transform li,, backward network 9b. entropy parameters network 9e based on their unique behaviors. For layers in 9ar9e: we compute the R-D loss before and after the perturbation. In contrast, for layers in has ht, Ib, ge; the perturbation does not affect the reconstruction performance (the D term), as these networks only involve calculating the R term, as shown in Fig. 3 Thus, we only need single forward pass to obtain Ú and then we individually perturb has hs 96: Ie layers based the existing ÿ. This approach allows us to avoid recurrently evaluating ú and focus solely on changes in R, thereby further reducing complexity. For the perturbation, Gaussian noise sampled from o") is added to all parameters when perturbing each layer, where a is a fraction of the maximum parameter value in the layer. After calculating the R-D loss (or R term) increase for all layers, those showing significant increases in R-D loss (or R term) upon perturbation are deemed more sensitive. Following Novak el al. (2018) and Yang et al. (2023), which demonstrate that approxinately 75% of parameters can be pruned without significantly affecting performance,
infer that only 25% of the parameters are sensitive. To identify these sensitive parameters, we employ straightforward and intuitive half-half strategy. We first identify the top 50% most sensitive layers, with 25% from Ja and Js: and 25% from ha, has lb, and ge-
Afterward, we estimate the parameter-wise sensitivity for the top 50% sensitive layers' parameters using the first-order estimation [Molchanov et al. 2019
0s
W O V (f.x) +
Ex |w,]
(5)
Ow, :E(L....NN)
where @ denotes elementwise multiplication. This function combines both the magnitude of the parameter Øf absolute value |w:] and the gradients of the R-D function I w.r.t. each parameter am,, which is effective in practice (Yvinec et al. 2023). By utilizing the estimated rough parameter-wise sensitivity, we identify the 50% relative sensitive parameters from these layers and consider these parameters "non-embeddable" (25% of total parameters), whereas the remaining 75% of paramoters are "cmbeddable". We calculate the sensitivily only once after the head-stage training.
Step& 3. Affine coefficients update. With the affine coefficients obtained in Step at epoch F. we now need to update them for each new epoch to observe their temporal dynamic behaviors. Let (t) be the coefficients Km evaluated at time L. Given the previous epoch Km 1), we can update Km following Brokman et al. (2024) using Eq. 6
kt)(t) = (Km ( 10 + W. (t)
ww (t) Wm wm)(t)
(6)
x (10,,- 1 - +
Wm
where Wm (t) and wn (t) are the t-th columns of Wn, and wm, respectively, and Wm.e comprises columns 1 through of T.n Eq. 0 is used for iterative updates after each new epoch.
7

---

Published in Transactions on Machine Learning Research (04/2025)
Step 4. True and Dummy Embedding. As we update the coefficients at each new epoch, it is observed that the affine coefficients k; and d, lend to stabilize after the head-stage training, shown in Fig. | We
can therefore fix K; and de based on their long-term changes evaluated every L epochs. (+ is defined as the Euclidaa distance between the current values of k; and d; and their L epochs carlier values:
(1 = / (l)(() - (t(1 1)) + IId(()) d,(1 - 1.
(7)
For h; and d, associated with "cmbeddable" paramcters, those with the least P% changes in terms of e are considered embedded coefficients, meaning h; and d, are fixed from this point onward. Then, the corresponding w, are the true embedded non-reference parameters. Note that we do not embed the reference parameters {wm j m=11- While the embedded parameters are no longer trainable, they still evolve as wm (t) changes, following wi = k (m (t) + d,. Consequently, the dimension of the training space and the number of trainable parameters gradually reduce as more and more parameters are embedded. This reduction in the degrees of freedom in the training dynamics facilitates the convergence of the training process. Ultimately, the dimension of the final state training space and the number of trainable parameters converge to the number of modes M in the theoretical case. In practice, we typically embed 50% of the parameters in our experiments.
Additionally, we reinitialize additional % parameters that have the least changes, excluding the truly em- bedded parameters. We assign these parameters the values from their embedding versions: w; t- (t) We refer to this process as dummy embedding. In the next epoch, the dummy embedded parameters are updated like the non-embedded parameters. Empirical results indicate that dummy embedding enhances performance. One explanation that this dummy embedding scheme simulates the Random Weight Pertur- bation (RWP) (Li et al., (2024c; Kanashiro Percira et al. ) which has been shown to improve performance and gencralization. Throughout the entire training process, we execute Step 1 and Step 2 once after the predefined epoch F. Subsequently, we repeatedly perform Step 3 and Step 4 at each new training epoch to embed parameters. The complete detailed STDET algorithm and CMD calculation are detailed in Appendix
Sec. A.7 in Algorithm [
3.3
Sampling-then-Morte Average
3.3.1
Moving average:
As described in Sec. 3.2 the proposed STDET relies heavily on stable correlation and smooth temporal behavior when embedding parameters, necessitating reduced training variance. However, due to the inherent complexity of training LIC models, maintaining such stability can be challenging. To regulate this, we incorporate the moving average technique directly into the training phase, which helps to smooth out noise
and randomness to ensure the consistency and stability of the training parameters (Martens) 2020 Chen
et al. 2021 Morales-Brotont et al. 2024).
The moving average technique is a straightforward and elficient method for consolidating multiple neural states into a single one, thereby enhancing performance, robustness, and generalization (Li et al., 2023c). It is particularly effective in models that exhibit a degree of similarity across states and does not intro- duce an unbearable additional computational overhead. Among the most commonly used techniques is the Exponential Moving Average (EMA) (Szegedy et al. 2016 Polyak Juditsky 1992).
EMA integrates early-stage states while assigning higher importance to more recent ones, which allows the model to adapt more quickly to changes during training. The EMA parameters are calculated as follows:
WEMA (0) = w()),, WEMA (t + 1) = - (1)-))EM)(() + oar(t + 1),
where a is a moving average factor that determines the weight given to recent versus older states.
Moving Average is argued to find flatter solutions in the loss asymmetric valleys than SGD, thus generalizing better to unsoen data, with a potential explanation that the loss function near minimum is often asym- metric, sharp in some directions, and flat in others. While SGD tends land near a sharp ascent, averaging iterates biased solutions towards flat region (He et al. 2019
8

---

Published in Transactions on Machine Learning Research (04/2025)
3.3.2 Proposed SMA:
While EMA is typically used during the testing phase, where its parameters do not influence training and only serve as the final model state, some methods (He et al. 2020; Song et al. 2023) integrate EMA into the training process. They achieve this by explicitly computing loss functions that encourage the alignment of SGD trajectories with EMA trajectories, thereby enhancing consistency and stability. In contrast, our approach follows more straightforward and simpler design. Rather than introducing additional alignment losses, we directly synchronize the EMA-updated sampled SGD parameters (SMA) to the SGD parameters. This enables smoother training dynamics while keeping low complexity.
A simple illustrative comparison of EMA (Szegedy et al. 2016 Polyak Juditsky 1992 explicit align-
ment (Song et al. 2023), and SMA provided in Fig. 16 Appendix Soc. A.4
Specifically, we sample states from the SGD trajectories al regular intervals and use the moving average rule to update the training parameters. Following EMA principles, the proposed Sampling-thert then-Moving Average (SMA) maintains set of SGD parameters and set of SMA parameters WsMA- The SGD parameters are obtained by applying the SGD optimization algorithm to batches of training examples. After every optimizer updates using SGD, we sample the current states and update the SMA parameters through interpolation based on EMA rules as in Eq. S Each time the SGD parameters are sampled and the SMA parameters are updated, we synchronize the SMA paramcters the SGD parameters to give a new starting point. The trajectory of the SMA parameters uSMA is thus characterized as an exponential moving average of the sampled SGD parameters states wy. After I optimizer steps, have:
USMA uSMM(t+1) 1)
(1-a)usMA(() (t) + (v))
= [t (1 a) atwx(t 11 +.... + (1 - a- + (1- wes'
(9)
where a is the moving average factor. The SMA parameters leverage recent states from SGD optimization while retaining some influence from earlier SGD parameters to effectively regulate temporal behavior and reduce variance. The additional moving average update and synchronization introduce negligible complexity. We further reduce the complexity by treating the sampling and updating as a single optimizer step, ensuring that the total number of iterations within each epoch remains unchanged. The SMA technique can be implemented simplified Lookahead optimizer (Zhang et al. 2019b Zhou et al. 2021a
By jointly applying STDET and SMA, the training trajectories of non-reference parameters w, are modeled as follows:
kowm (1) + dz if (t) is embedded wi(()= SMA update elsc
(10)
The reference parameters are updated using SMA.
4 Experimental results
4.1 Experimental settings
Training. We the COCO2017 dataset (Lin et. al., 2014) for training, which contains 118,287 images, each having around 420 pixels. We randomly crop 256 x 256 patches from these images.
All models are trained using the Lagrange multiplier-based rate-distortion loss as defined in Eq. 2
Following the settings CompressAT Bégaiin et al. 2020 and standard practice He et al. 2022 Liu et al. 2023 Li et al. 2024a), we set to {18, 35, 67, 130, 250, 483) x 10-4, For all models the * + SGD" series, we train each model using the Adam optimizer with 31 0.9 and B3 = 0.999. The 0.0018 models are trained for 120 epochs. For models with other À values, we fine-tune the model trained with X = 0.0018 for additional 80 epochs. For all models in the 5+ Proposed" scries, train cach X 0.0018 model using
Adam essentially SGD with first-order moments (t.e., mean) and second-order moments (t.e., variance) estimation.
9

---

Published in Transactions on Machine Learning Research (04/2025)
the Adam optimizer for 70 epochs. For models with other À values, we fine-tune the model trained with À = 0.0018 for additional 50 epochs. More details are provided in the Appendix Sec. A.8
3 and CLIC 2022 are used to evaluate the performance of the proposed method. We further demonstrate the robustness the proposed method and capacity for real-world application by conducting experiments on stereo images, remote sensing images, screen content images, and raw image compression, as shown in Appendix Sec. A.1
Testing. Three uidely-used benchmark datasets, including Kodak 2 Tecnick
4.2 Quantitative results
We compare our proposed method with standard SGD-trained models on prevalent complex LICs, including ELIC (He al. 2022), TCM-S (Liu et al. 2023). and FLIC (Li et al., 2024a to demonstrate its superior performance. We use SGD-trained models as the anchor to calculate BD-Rate (Bjontegaard 2001).
Tab. D presents the BD-Rate reduction of the proposed method compared to the SGD anchors across three datasets. Our proposed method consistently achieves comparable final results to all methods across these datasets, which include both normal and high-resolution images, demonstrating its effectiveness, For ex- ample, on the Kodak dataset, "ELIC + Proposed" achioves -0.71% against "ELIC + SGD". Fig. 4 further illustrates the R-D curves of all methods, showing that the R-D points of the SGD-trained model and the model trained by the proposed method almost overlap.
Table Computational Complexity and BD-Rate Compared SGD
BD-Rate (%)
Method
Training time |
Total trainable params
Final trainable parama
Kodak
Tocmick
CLJC2022
+ SGD Proposed
165b
18, 18,118M
36M1
0%
0%
0%
FLIC He
2022
101h
9,519M
20.66M
-0.71%
-0.70%
-0.61%
TOM-S
Liu et al. 2023
346h
23,493M
0%
Proposed
213h
12,142M
26.34M
-0.77%
-0.74%
-0.64%
4 SGD Proposed
520h
36,899M
70.96M
0%
0%
0%
2024a
320h
19,070M
41.39M
-0.65%
-0.51%
-0.63%
Training Conditions:1 x Nvidia 4090 GPU, 19-14900K CPU, 128GB RAM. Bold represents better performance.
38
39
37
38
37
36
35
PSNR (OB)
36
PSVR (dB)
35
34
PSWR
35
34
33
32
EUC
33
EUIC Proposed
ELIC + Proposed
32
EUC Proposed
SGD
30
TCM Proposed
32
TCM e Proposed
TCM Proposed
FUC SGD
29
FUC Proposed
31
FUC Proposed
FUC Proposed
0.2
0.4
0.6
0.6
0.1
0.2
0.3
0.4
0.5
0.6
0.2
0.4
bpp
(a) Kodak
(b) Tecnick 1200x1200
(c) CLIC 2022
Figure 4: R-D curves of various methods. Please zoom in for more details.
We also compare our approach with other efficient low-dimensional training methods, including P-SGD LLi et al. 2022b). P-BFGS [i et al. 2022b), TWA (Li et al. 2023b), and sparsity-based training methods, such
RigL Evci el al. 2020 and SRigL Lasby el. al.] 2024). P-SGD employs SGD in projected subspace
https://rOk.no http:///rOkkuagrrahhcc/kooaaa
http://conpeession.co
10

---

Published in Transactions on Machine Learning Research (04/2025)
derived from PCA. P-BFGS uses the quasi-Newton method BFGS within the same projected subspace. TWA utilizes Schmidt orthogonalization to build the subspace based on historical solutions. RigT. (Evci et al. 2020) updates sparse network topology via parameter magnitudes and occasional gradient computations, and SRigL (Lasby et al. 2024) is an improved variant emphasizing dynamic sparsity management.
For these compared low-dimensional methods, we follow the original paper and set the SGD pretrained epoch as 50, after which we train for another 20 epochs using these methods. For the sparsity- based training methods, we train all the models for 70 epochs with target sparsity of 50% following the original implementations. In total, the training epochs for all the compared methods and our proposed method are set to 70 to ensure a fair comparison.
As shown in Tab. 2 although these low-dimensional methods operate within significantly reduced training dimensions, they do not perform well in terms of performance. Both P-SGD and TWA result in loss divergence, ultimately causing the training to crash. P-BFGS also struggles to converge, leading to a substantially higher R-D loss compared to our proposed method, 0.3982 vs 0.3442. Several factors contribute to these outcomes: (1) These methods were originally designed for image classification tasks and typically use small, simple datasets such CIFAR10 and CIFAR100, whereas LIC training is inherently more complex; (2) The dimensionality reduction in these methods heavily depends on historical solutions (sampled epochs), limiting the dimensionality to around 50. Directly projecting extremely complex networks into such small dimensions appears to cause coneoonvergence. Additionally, we observe that P-BFGS does not save training time compared to other efficient training methods, as the quasi-Newton techniques involve time-consumingg estimation and iterative updates of the Hessian matrix.
Sparsity-based training methods, despite their efficiency in reducing inference FLOPs, do not accelcrate the convergence of LICs. This is primarily due to their growth-and-drop mechanism, which maintains constant sparsity ratio without reducing the training space dimensions. As a result, they fail to improve convergence speed. Both RigL and SRigL exhibit significantly higher losses compared to SGD and our proposed method. For instance, SRigL's 's R-D loss is 0.3601, while RigL's 's loss is 0.3624. Although sparsity-based methods achieve a notable 50% reduction in inference FLOOPs compared to all low-dimensional methods and SGD, their poor R-D performance renders this advantage ineffective. This outcome aligns with expectations, as these methods were originally designed for training sparse models in image classification tasks fundamentally different objective from our goal of accelerating LIC training.
The comparison with both low-dimensional training methods and sparsity- based approaches clearly highlights the superiority of our proposed method in accelerating LIC convergence without compromising performance
Table 2:
Comparison with Various Efficient Training Methods
Settings
Training epoch
Training space dimenssoon $ Total training time
Inferencee PLOPSs 1
R-- loss J
ELIC + SGD
120
35,424,506
38h
437.66
0.3444
ELIC P-SGD
al.
2022h
70
437.6C
div.
ELIC P-BFGS
20225
70
40
35h
437. 6G
0.3982
ELIC= + TWA
cl M.
21236
70
50
23h
137.6G
div.
ELIC- + RigL.
Evvll e5
2020
70
35,,424.506
23h
218.8G
0.3624
ELIC+ SRigt.
Lasby
2112
70
(35,424,508 505
24h
214.4G
0.3601
ELIC- Proposed
70
SU"
23h
437.6G
0.3442
Train Conditions:1 x Nvidia 4090 GPU, 19- CPU, 128GB RAM. "div." indicates that these
mothods result in loss divergence, eventually causing the training to crash. *: The training dimension
of our proposed method continues to decrease as training proceeds, theoretically converging to 50. = 0.0018. R-D loss = A- 255² MSE + Bpp. Bold indicates the best.
FLOPs are calculated using the ptilops library with an input image size of 512x512 pixels.
4.3 Complexity
We evaluate the complexity of SGD and our proposed method by measuring total training time, total trainable parameters for the whole training process, and trainable parameters the final epoch, as presented in Tab. ! Total training time is defined as the time required to train all models for each À value on a single 4090 GPU. Compared to models trained using SGD, our proposed method is significantly more efficient.
11

---

Published in Transactions on Machine Learning Research (04/2025)
reducing the training time to approximately 62% of that required by SGD. This efficiency is mainly attributed to the faster convergence of our method, which achieves the desired performance with fewer epochs while maintaining comparable training time per epoch. For example, training all FLIC models using SGD takes 520 hours 21 days on a single 4090 GPU), which is unbearable. In contrast, our proposed method drastically reduces the training time to 320 hours (13 days). Similarly, the training time for TCM-S is reduced from 346 hours (14 days) to 213 hours (8 days), highlighting the clear advantages of our approach.
The total trainable parameters calculated by multiplying the model's trainable parameters by the total number of training epochs across all À values. For example, in the case of "ELIC SGD", the total trainable parameters are computed as 35.42M x 520 epochs, resulting in 18, 418M. For "ELIC + Proposed", using À 0.0018 as an example, the total trainable parameters from epoch 1 to 20 are 35.42M x 20 epochs. Starting from epoch 21, we begin embedding 1% of the parameters in each epoch. Thus, for epochs 21 to 70, the total trainable parameters are calculated as 100% 35.42M 99% 35.42M + +... 51% x 35.42M. By performing similar calculations for all other values, we find that the total trainable parameters for "ELIC Proposed" are 9, 519M, approximately 51% of the total trainable parameters required by the SGD method. Similarly, the total trainable parameters for FLIC and TCM-S methods are reduced from 36, 899M (FLIC) and 23, 493M (TCM-S) to 19, 070M (FLIC) and 12, 142M (TCM-S), respectively, demonstrating a substantial reduction. We also report the final trainable parameters, representing the mumber of trainable parameters after the final epoch. For instance, FLIC's trainable parameters are reduced from 70.96M to 41.39M. By reducing the number of trainable parameters, we possess the potential to further accelerate training by utilizing sparse training methods (Zhou al. 2021b), which will be discussed in Appendix Sec. A.5
4.4 Ablation study
Hyperparameters are critical to the proposed method. To evaluate the influence of various factors and demonstrate the process of selecting the best hyperparameters, conducted comprchensive experiments across all three models. In the R-D plane (bpp-PSNR figures), better performance is indicated by points in the upper left region. Figures 5 6 and 7 depict the R-D curves for ELIC, TCM, and FLIC under different configurations of mode decomposition, embedding mechanisms, and SMA, respectively, using À = 0.0018, 0.0035, and 0.0067 as illustrative examples. The corresponding BD-Rate results are summarized in Tables and where the final configuration in each column is used as the anchor for BD-Rate calculations.
6i-1
PSNR (dB)
PSIRR (cIB)
P=2% 6 1=1
PSNR (d9)
P=2% 6
F=10
Pa5% $1m5
e-05 & 6/45
30
30
P-12% &
Increading
30
0-03 &
M-100
Decreasing
Se1k
Sensiiiity
5-5t
Bummy
Accurate
0-0. /-10
we SMA
M650, S=10k
bpp
hpp
hpp
(a) Mode decomposition
(b) Embeddig mechanism
(c) SMA
Figure 5: Ablation experiments on proposed methods. ELIC model.
4.4.1
What is the impact of the factors in the CMD calculation procedure?
In our method, three key factors influence the Step CMD calculation. The first key factor is the predefined epoch F, which determines when CMD performed and the embodding mechanism begins. As shown Fig. 2 the most significant loss changes occur within the first 20 epochs. To ensure embedding neither starts too early too late, we set. F 20. Starting the embedding process too early could compromise
12

---

Published in Transactions on Machine Learning Research (04/2025)
32
32
32
P-15 1-1g
Pel Giml
PSNR (dB)
Pa2k 6.
PSNA (dB)
Pa2% 51m2
PSWR
F-30
P-3256 6--10
/e5
A-11
Increading
K--10
Mio Dumny
em0.5 & /m3
am0.3 /=10
F-20
ceder
20
0.2
0.3
28
0.2
0.3
20
02 bpp
0.3
bpp
bpp
(a) Mode decomposition
(b) Embeddig mechanism
(c) SMA
Figure 6: Ablation experiments proposed methods. TCM-S model.
32
32
32
PSNR (dB)
PSWR (dB)
SAN (dg))
F-30
Fe10
P-12%
PSNN
e=05
30
30
Increading
30
em0.9 & //5
S-1k
Mtt SenstMitty
Decruasing
o-08 &
&
5-5t
Accurate
S=2tk
Rist order Dummy
we SMA
(e:50,5:00
P-1% & 1-1
e508 & 105
0.1
hpp
hpp
bpp
(a)
Mode decomposition
(b) Embeddigg mechanism
(c) SMA
Figure 7: Ablation experiments on proposed methods. FLIC model.
Table 3: Ablation study of hyperparameters on ELIC model. BD-Rate values show coding efficiency com-
pared to the bottom row baseline configurations (marked with 0%). Lower BD-Rate denotes better.
Mode Decomposition
Embedding
SMA
Setting
BD-Rate Rate
Setting
DD-Rate
Setting
BD-Rate
F' 10
11.54%
P 1% & L
-1.40%
0.5 & (
2.48%
F 30
0.24%
P 1.5% & L
5.20%
0.9 & [
0.48%
M 10
3.83%
P 2% & I. 1
10.72%
0.8 & 1 =
0.66%
M 100
0.63%
P 2% & L = 2
1.43%
0.8 & / 10
0.26%
S 1k
4.34%
P 5% & L L- 5
2.34%
a 0.5 & -
7.39%
S 5k
1.85%
P = 10% L 10
2.24%
a = 0.9 & = 10
1.75%
20k
0.66%
P: Increasing
0.15%
a w/o SMS
11.91%
F 20, M 50, S 10k
0%
w/o Sensitivity
12.34%
a 0.8 & -
0%
w/o Dummy
2.29%
Accurate
-1.11%
W First order
4.55%
P 1% & -
0%
13

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Ablation study of hyperparameters TCM-S model. BD-Rate values show coding efficiency
compared to the bottom row baseline configurations (marked with 0%). Lower BD-Rate denotes better.
Mode Decomposition
Embedding
SMA
Setting
BD-Rate
Setting
BD-Rate
Setting
BD-Rate
F 10
8.88%
P 1% & L == 2
-0.83%
a 0.5 & 1
2.15%
F 30
-0.40%
P = 1.5% & I. = 1
5.93%
a 0.9 & 1
1.03%
M 10
5.46%
P 2%
= 1
10.42%
a 0.8 & [
0.62%
M 100
-0.65%
P 2% & 2
1.62%
0.8 & 10
0.92%
5 1k
5.62%
P 5% & T.
2.22%
0.5 & 1
5.94%
S 5k
4.35%
P 10% & L 10 P: Increasing
2.29%
0.9 & - 10
1.46%
S 20k
-0.15%
0.45%
w/o SMA
9.72%
F = 20, M = 50, S = 10k
0%
w/o Sensitivity
10.42%
0.8 5
0%
w/o Dummy
2.40%
w/ Accurate
-1.38%
W First order
5.16%
P 1% & -
0%
Table 5: Ablation study of hyperparameters FLIC model. BD-Rate values show coding efficiency com- pared to the bottom row baseline configurations (marked with 0%). Lower BD-Rate denotes better.
Mode Decomposition
Embedding
SMA
Setting
BD-Rate
Setting
BD-Rate
Setting
BD-Rate
F = 10
10.53%
P = 1% & I I= 2
-0.92%
O 0.5 & 7 -
1.78%
F = 30
-0.48%
P = 1.5% L = 1
5.48%
a 0.9 & [
0.45%
M 10
7.67%
P 2% I.
10.36%
a 0.8 & |
0.13%
M 100
-155%
P 2% & 2
1.30%
0.8 & 10
0.83%
S 1k
7.43%
P 5% & L 5
2.06%
0.5 & [
6.17%
S 5k
4.86%
P 10% & -= 10
2.31%
0.9 & 1= 10
1.77%
S 20k
0.01%
P: Increasing
0.73%
w/o SMS
8.96%
F = 20, M 50, S = 10k
0%
w/o Sensitivity
10.73%
0.8 & -
0%
w/o Dummy
2.15%
w/ Accurate
-1.35%
W First order
5.11%
P 1% & --1
0%
14

---

Published in Transactions on Machine Learning Research (04/2025)
the accuracy of the computed reference parameters and sensitivity estimations due to ongoing significant changes, negatively impacting performance. Conversely, starting too late may delay convergence acceleration and reduce the number of embedded parameters, as fewer epochs remain for training. To evaluate the effects
of different F values, we tested F = 10, F = 20, and F = 30, with results reported in Fig. ((aa 6(a] 7(a) the
first column of Tab. 3 4 and 5 The findings indicate that F = leads to significantly worse convergence,
e.g.; with an 8.88% BD-Rate increase compared to the default F = 20 for the TCM-S model. Mcanwhile, F = 30 achieves only a marginal 0.40% BD-Rate reduction but requires more trainable parameters. Based these results, conclude that setting F 20 provides the balance between accuracy and efficiency.
The second critical factor is the number of modes M, which plays vital role in CMD computations. Too few modes fail to caplure the correlations between parameters accurately, adversely affecting performance. On the other hand, too many modes overcomplicate the decomposition process and increase memory require- ments, particularly during the calculation of correlations between parameters and reference parameters. We conducted experiments with M 10, M 50, and M 100, as shown in the figures and tables. The results suggest that the optimal M value may vary depending on the model. For instance, M 50 delivers the best results for the ELIC model, while M 100 performs best for the TCM-S and FLIC models. Further analysis of this hyperparameter is necessary.
Finally, to address the computational ional challenge of calculating the large N x N correlation matrix for identi- fying reference parameters, we uniformly sample S trajectories to determine the reference trajectories. We compare results using S = 1k, S 5k, S 10k, and S = 20k, as shown in the figures and tables. Similar to M, the results do not clearly indicate the optimal number of sampled trajectories. For example, S 10k yields the best results for the ELIC model, whereas S 20k is the best for the TCM-S model. $ = 10k and S = 20k shows negligible difference for FLIC model. Therefore, in CMD calculations, the predefined epoch F' can be set to 20 across all models, while the number of modes M and the number of sampled trajectories S should be model-specific. Further guidance on efficiently determining these parameters is provided in
Soc. 4.4.4
4.4.2 What is the impact of the factors of the embedding mechanism?
Determining the parameter sensitivity is a crucial process in the embedding mechanism. As mentioned earlier, embedding certain parameters may significantly reduce the performance. Therefore, we first add experiments that omit sensitivity estimation, and we observe a substantial decrease in the R-D performance, see Fig. 5(b) 6(b] 7b) the second column of Tab. 3 and 5 ww/o Sensitivity". This significant change supports our hesos that certain parameters are extremely sensitive and inappropriate embedding leads to degraded performance.
To further demonstrate the effectiveness of our proposed combined strategy- half accurate layer-wise as- sessment and half rough parameter-wise estimation we include two additional variants. The first variant, pure accurate layer-wise assessment ("w/ Accurate"), perturbs each LIC parameter individually, requiring millions of iterations to estimate sensitivity precisely. While this provides an upper bound for our combined strategy and offers an additional 1.2% BD-Rate improvement, it is computationally infeasible due to its high complexity. The second variant, first-order approximation ("w/ First order"), estimates parameter sensitivity using only a rough first-order approximation. However, as shown in the tables, this method leads to inaccurate sensitivity estimation and consequently suboptimal performance, with an average BD-Rate increase of around 5.5% across all three methods.
Our combined strategy strikes a balance between efficiency and effectiveness, achicving superior performance while avoiding the computational burden of exhaustive sensitivily analysis. These results highlight the advantages of our proposed strategy.
Other factors influencing the embedding mechanism include the embedding percentage, P, and the embed- ding period, L. The percentage P dictates the proportion of parameters to be embedded, while L determines the frequency of embedding over specified epochs. As demonstrated in figures and tables, experiments reveal that embodding 1% per epoch (P- 1% & L 1) and 1% every two epochs (P = 1% & I. = 2) yield the best results. P 1% & L = 2, however, results in too few parameters being embedded. Embodding 1.5%/2% in a single epoch (P- 1.5% & I. 1/P- 2% & L 1) leads to poorer performance due to excessive embedding
15

---

Published in Transactions on Machine Learning Research (04/2025)
in the training process, resulting in inaccurate modeling and decreased R-D performance. Comparing short-term small embedding percentages (P- 1% & T= and P 2% & I. = 2) with long-term large ones (P= 5% & L = 5 and P- = 10% & L = 10); we find that a shorter, smaller setup performs better. This likely stems from the disruptive effect of embedding a large number of parameters simultancously, which adversely impacts training. Further experiments involving either a linear increase or a decrease schedule for (We set P appropriately so that the embedded paramcters the final epoch account for 50% of the model's train- able parameters.) show that gradually increasing the embedding percentage yields performance similar to P = 1%, L = 1, but results in more total trainable parameters. Conversely, a gradual decrease leads to poor performance. Therefore, we choose the straightforward approach of P = 1%, I. = 1 due to its simplicity and effectiveness.
The proposed dummy embedding involves reinitializing the original parameters using the embedded version during the embedding process, akin to the RWP technique; which has proven effective in enhancing per- formance. To evaluate the efficacy of dummy embedding, we conducted experiments in its absence. The results, detailed "w/o Dummy"; indicate around 2% BD-Rate performance drop upon removal of dummy embedding, thus affirming its positive impact on performance.
4.4.3 What is the impact of the factors of the SMA?
We also conduct extensive experiments assess the two eritically important factors of SMA, as detailed in Fig. [[c] [(c] 7
the last column of Tab. 3 4 and 5 A setting of a 0.5, = results in poorer performance, possibly due to an over-reliance on past states. In contrast, the setting of a = 0.9.7 = 5 shows no significant difference compared with the utilized = 0.8,1= = 5. When we adjust 1, the values 3, 5, and 10 yield consistent performance. In our next. experiments with new combinations of a and 1, we find that a = 0.5, / = 3 performs the worst. a = 0.9, = 10 is slightly less effective than a = 0.8,/ = 5. We then choose the setting = 0.8, 1 based on these results. Finally, we also attempted "w/o SMA" As previously discussed, the SMA is crucial for ensuring that the STDET operates smoothly. It is evident that the absence of the SMA results in noticeable performance decline. We also individually examine the roles of the proposed STDET and SMA in Sec. A.3
remove the SMA,
From the comprehensive ablation study, we observe that only the number of modes M and the number of sampled trajectories S yield inconsistent results. For all other parameters, a unified configuration can be applied regardless of the model or A. The final unified settings are: predefined epoch = 20, embedding percentage P 1%, embedding period 1, moving average factor a 0.8, and optimizer step 1 5.
4.4.4 How to select the number of modes and the number of sampled parameters?
Based on the preliminary results in Sec. 4.4.1 our initial hypothesis suggests that larger models generally require a greater number of modes and sampled trajectorics. To validate this hypothesis, we conducted a comprehensive set of experiments across all three models, evaluating M E (10, 30, 50, 70, 100, 120) (number of modes) and S E (2k, 6k, 10k, 14k, 20k, 24k) (number of sampled trajectorics). The results are summarized in Fig. g For consistency, the anchor setting for each model is M 50 and S 10k.
From the results, we can draw the following conclusions:
Larger models require larger M and 8, but the performance saturates beyond certain thresholds.
For instance:
ELIC (35.42M parameters): Performance caps at M = 50 and 10k.
TCM-S (45.18M parameters): Performance reaches its max value at M = 70 and S = 14k.
FLIC (70.96M parameters): Performance saturates at M 100 and S = 20k.
Beyond these thresholds, further increases in M and provide negligible performance gains.
The highest performance for each number of modes is achieved when the number of sampled trajectories is roughly 200 times the number of modes.
This corresponds to the diagonal in the figure. For example:
16

---

Published in Transactions on Machine Learning Research (04/2025)
For M 10,30, 50, 70, 100, 120, the highest performance is achieved at S 2k, 6k, 10k, 14k, 20k, 24k, respectively for each model.
3. The number of modes has greater impact performance than the number of sampled trajectories. For example:
In the FLIC model, with only 10 modes, the performance is as low 7% BD-Rate. However, even with an inappropriate S = 2k, 120 modes can still achieve BD-Rate of -0.21%.
Additionally, the lower right region of the figure exhibits deeper colors compared to the upper left region, emphasizing the significant impact of the number of modes.
These findings confirm our initial hypothesis that larger models require a greater number of modes and sam- pled trajectories, and we only need to consider the diagonal combination. However, in real-world scenarios, it is often impractical to exhaustively search for the best hyperparameters and find their performance cap, even for few combinations. This raises critical question: How can determine the optimal mumber of modes and sampled trajectories without extensive training?
Fiol RD Boto
Ri Buli
14:00
4.1%%
5.400
0.857
5,52%
5.03%
0.115
2002
3.35%
0.34%
3.01%
-0.32%
7.0%%
2.59%
3.73%
10.90
201
2.055
2,05%
4.01%
1.7
6.90%
2.52%
0.12%
44.19%
41.6%%
-3.44%
383%
1:4m
0.00%
9.4%%
0.43%
1.05%
5.4%%
1.00%
0.00%
1.24%
0.41%
0.23%
7.0%%
2.13%
0.00%
3.79%
-1.5%
3.00%
6:0
3-42%
3.77%
0.08%
0.704
3.40%
1.20%
659
5.33%
1.0%
0.7%%
3.00%
+8.7%%
40.15%
2.03%
0.22%
43.33%
1..5%
4.38%
2000
A11%%
0.85%
1.82%
2.48
1.227
203
8.12%
1.3m
5.18%
18.11%
0.117
317
..33%
(a) ELIC
(b) TCM-8
(c) FLIC
Figure 8: Ablation experiments on modes M and sampled trajectories S. Deeper color represents better performance.
During our experiment, we observed an intriguing phenomenon: the CMD instant performance i.e., the performance the CMD-dccomposed model (Step 1) evaluated immediately on a randomly sampled dataset (the same set used Step 2) after decomposition under certain M,S without additional training-correlater correlated strongly with the final performance achieved after complete training. This observation led us hypothesize that the quality of CMD decomposition plays critical role in determining the ultimate model perfor- mance, analogous to how initialization methods in neural networks influence optimization and final accu- racy (Narkhede et al. 2022 Glorot & Bengio 2010 He et al. 2015).
To validate this hypothesis, we conducted systematic study measuring CMD instant performance across various settings of M (number modes) and S (number of sampled trajectories). The results, shown 200 x M, as suggested by earlier findings. These
in Fig. focus on diagonal combinations where experiments confirmed that the CMD instant performance improves with increasing M and S, eventually stabilizing at specific values. Notably, the saturation points of CMD instant performance closely align with the thresholds where the final performance stabilizes. For example, in the ELIC model, both CMD instant performance and final performance stabilize at M 50 and 5 = 10k. Similar trends are observed for the TCM-S and FLIC models.
This strong correlation suggests that CMD instant performance can serve as a reliable indicator for selecting the optimal values of M and S.
Building this insight, we propose efficient method for hyperparameter selection:
17

---

Published in Transactions on Machine Learning Research (04/2025)
1. During the CMD decomposition stage, evaluate CMD instant performance for set of diagonal combinations (M where 5 = 200 x M. This evaluation requires only single forward pass per combination, making it computationally lightweight.
2. Identify the combination where CMD instant performance stabilizes or reaches its peak. We define the performance cap as the point where subsequent combinations result in less than a 0.1% R-D loss improvement (approxinately corresponding to 0.2% BD-Rate difference). We then use the determined values of M and for subsequent training.
The detailed procedure is presented Algorithm I This method eliminates the need for exhaustive training to search for optimal hyperparameters, significantly reducing computational costs, as requires only a forward pass for each combination. Moreover, it is naturally extendable to new models. For instance, for model with more parameters (e.g., 100M parameters compared to FLIC's 70.96M), diagonal combinations with larger M and S (e.g., M = 70, 100, 120, 150 and S = 200 x M) can be explored. Given that FLIC reaches its performance cap at M = 100 and S = 20k, it is reasonable to expect that larger models will stabilizo at proportionally higher valuos of M and S.
This phenomenon parallels the influence of network initialization on final performance in neural networks. Initialization methods such as Xavier (Glorot & Bengio 2010), Kaiming (He et al., 2015), and orthogonal initialization (Saxe et al. 2013) have demonstrated that the quality of initialization significantly impacts optimization and final accuracy. Similarly, the quality of CMD decompasition serves strong initial- ization for subsequent training in our framework. By leveraging CMD instant performance, we strcamline hyperparameter selection and ensure efficient model training.
To summarize, the M (number of modes) and S (numberr of sampled trajectories) can be efficiently de- termined by traversing diagonal combinations and comparing the CMD instant performance. This process requires only a single forward pass per combination, making it a highly efficient and practical approach for hyperparameter selection.
Initer
Instant 20 Cute 1NC
2004
-0.5%
0.0%
0.00%
0.80
1.3%%
1.70
M55
3.98%
200
3,68%
200
3.78%
Modes
Modes
Kodes
(a) ELIC
(b) TCM-S
(c) FLIC
Figure 9: The instant CMD performance of different modes M and sampled trajectories S. Deeper color represents better performance.
5 Conclusion
In this paper, accelerate the training of LICs by modeling the neural training dynamics, considering the linear representations of LIC'' complex models. We extend the concept of Dynamic Mode Decomposition into our proposed Sensitivity-awarr Truc and Dummy Embedding Training (STDET). Our method progressively embeds non-reference parameters based on their stable correlation and sensitivity, thereby reducing the dimension the training space and the total number of trainable parameters. Additionally, we implement the Sampling-then-Moree Average (SMA) technique smooth the training trajectory and minimize training variance, enhancing stable corrclation. In this way, our approach results in faster convergence and fewer trainable parameters compared to the standard SGD training of LIC's across various image domains. Both
18

---

Published in Transactions on Machine Learning Research (04/2025)
analyses on the noisy quadratic model and extensive experimental validations underscore the superiority of our method over standard training schemes for complex, time-intensive LICs.
6 Acknowledgment
This work was supported by the National Cancer Institute under Grant RO1CA277839.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pp. 7319-7328, August 2021.
Muhammad Salman Ali, Yeongwoong Kim, Maryam Qamar, Sung-Chang Lim, Donghyun Kim, Chaoning Zhang, Sung-Ho Bae, and Hui Yong Kim. Towards efficient image compression without autoregressive models. Advances in Neural Information Processing Systems, 36:7392 7404, 2023.
Johannes Ballé, Philip Chou, David Minnen, Saurabh Singh, Nick Johnston, Eirikur Agustsson, Sung Jin Hwang, and George Toderici. Nonlinear transform coding. IEEE Journal of Selected Topics in Signal Processing, 15(2):339-38) 335,, 2020.
Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston. Variational image compression with scale hyperprior. International Conference Learning Representations, 2018.
Wei Bao, Wei Wang, Yuhua Xu, Yulan Guo, Siyu Hong, and Xiaohu Zhang. Instereo2k: a large real dataset for stereo matching in indoor scenes. Science China Information Sciences, 63:1-11, 2020.
Riccardo Barbano, Javier Antoran, Johannes Leuschner, José Mignel Hernández-Lobato, Bangti Jin, and Zeljko Kereta, Image reconstruction via deep image prior subspaces. Transactions on Machine Learning Research, 2024.
Joan Bégaint, Fabien Racapé, Simon Feltman, and Akshay Pushparaja. Compressai: a pytorch library and evaluation platform for end-lo-end compression research, 2020.
G. Bjontegaard. Calculation of average PSNR differences between rd-curves. ITU-T SG 16/Q6, 18th VCEG Meeting, April 2001.
Jonathan Brokman, Roy Betser, Rotem Turjeman, Tom Berkov, Ido Cohen, and Guy Gilboa. Enhancing neural training via correlated dynamics model. The Twelfth International Conference on Learning Representations, 2024.
Steven L. Brunton, Marko Budišíé, Eurika Kaiser, and J. Nathan Kutz. Modern koopman theory for dynamical systems. SIAM Review, 64(2):229 340, 2022.
Tianlong Chen, Zhenyu Zhang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Robust overfitting may be mitigated by properly learned smoothening, International Conference on Learning Representations, 2021.
Dongliang Cheng, Dilip K Prasad, and Michael S Brown. Illmminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution. Journal of the Optical Sociely of America A, (1(5):0049-1000 1058, 2014.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene under- standing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognétion, pp. 3213 3223, 2016.
19

---

Published in Transactions on Machine Learning Research (04/2025)
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficint fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220 235,2,0-233,,, 2023.
Akshunna S Dogra and William Rodman. Optimizing noural networks via koopman operator theory. Ad- vances in Neurul Information. Processing Systems, 33:2087 2097, 2020.
Zhihao Duan, Ming Lu, Jack Ma, Yuning Huang, Zhan Ma, and Fengqing Zhu. Qarv: Quantization-avvan resnet vae for lossy image compression. TEFE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Jarek Duda. Asymmetric numeral systems, 2009.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. Proceedings of the International Conference on Machine Learning, pp. 1943-2952 2020.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. Proceedings of the International Conference on Artificial Intelligence and Stalistics, pp. 249 256, 2010.
Frithjof Gressmann, Zach Eaton-Rosen, and Carlo Luschi. Improving neural network training in low dimen- sional random bases. Advances in Neural Information Processing Systems, 33:12140-12150, 2020.
Wang Guo-Hua, Jiahao Li, Bin Li, and Yan Lu. EVC: Towards real-time neural image compression with mask decay. The Eleventh International Conference on Learning Representations, 2023.
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14771-14780) 2021.
Dailan He, Ziming Yang, Weikun Peng, Rui Ma, Hongwei Qin, and Yan Wang. ELIC: Efficient learned image compression with unevenly grouped spaco-channel contextual adaptive coding. Proceedings of the IEEE/CVP Conference on Compuler Vision and Pallern Recognition, pp. 5718 5727, June 2022.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. Advances in Neural Information Processing Systems, 32, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human- level performance on imagenet classification. Proceedings of the IEEE/CV International Conference on Computer Vision, pp. 1026 1026-1034, 2015.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVE /CFF Conference on Computer Vision and Pattern Recognition, pp. 9729-9733 2020.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022.
ISO/IEC JTC 1/SC29/W WGL. JPEG AI Common Training & Test Conditions v8.0. CPM, 100th Meeting, Covilhā, Portugal, July 2023. Document number: N100600.
Pavel Ixmailov, Dmitri Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Aver- aging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018.
Menglin Jia, Luming Tang,
Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European Conference Computer Vision, pp. 709-727, 2022.
20

---

Published in Transactions on Machine Learning Research (04/2025)
Zhaoyang Jia, Bin Li, Jiahao Li, Wenxuan Xie, Linfeng Qi, Houqiang Li, and Yan Lu. Towards practical real-time neural video compression, 2025. URL https://arxiv... org/abs/2502 20762
Fatih Kamisli, Fabien Racapé, and Hyomin Choi. Variable-rate learned image compression with multi- objective optimization and quantiatioon-ocoostruiiooo offsets. Proceedings of the Data Compression Con- ference, 2024.
Lis Kanashiro Pereira, Yuki Taya, and Ichiro Kobayashi. Multi-layer random perturbation training for improving model generalization efficiently. Proceedings of the Fourth BlackborNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 303-310, November 2021.
Hakki Can Karaimer and Michael Brown. A software platform for manipulating the camera imaging pipeline. Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Octo- ber 11-14, 2016, Proceedings, Port I 14, pp. 429 444, 2016.
Pang Wei Koh and Percy Liang. Understanding black- box predictions via influence functions. Proceedings of the 94th International Conference on Machine Learning, 70:1885-189) 2017.
Mike Lasby, Anna Golubeva, Utku Evci, Mihai Nica, and Yani loannou. Dynamic sparse training with structured sparsity. Proceedings of the International Conference on Learning Representations, 2024.
Chunyuan Li, Hecrad Farkhoor, Rosanne Liu, and Jason Yosinski. Mcasuring the intrinsic dimension of objective landscapes. International Conference on Learning Regresenlations, 2018.
Han Li, Shaohui Li, Wenrui Dai, Chenglin Li, Junni Zou, and Hongkai Xiong. Frequency-aware transformer for learned image compression. The Twelfth International Conference on Learning Representations, 2024a.
Jiahao Li, Bin Li, and Yan Lu. Deep contextual video compression. Proceedinggs of the Advonces in Neural Information Processing Systems, 34:18114 18125, 2021.
Jiahao Li, Bin Li, and Yan Lu. Hybrid spatial-temporal entropy modelling for neural video compression.
Proceedings of the ACM International Conference on Multimedia, pp. 1503-1511, 2022a.
Jiahao Li, Bin Li, and Yan Lu. Neural video compression with diverse contexts. Proceedings of the IBEE/CVP Conference on Computer Vision and Pattern Recognition, pp. 226226, 2023a.
Jiahao Li, Bin Li, and Yan Lu. Neural video compression with feature modulation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognation, pp. 26099 26108, 2024b.
Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang, Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaccs. IEEE Transactions on Pattern Analysis and Machine Intelligence,
-4420, 2022b.
Tao Li, Yingwen Wu, Sizhe Chen, Kun Fang, and Xiaolin Huang. Subspace adversarial training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13409-140 13418,, 2022c.
Tao Li, Zhehao Huang, Qinghua Tao, Yingwen Wu, and Xiaolin Huang. Trainable weight averaging: Ef- ficient training by optimizing historical solutions. The Eleventh International Conference on Learning Represenlations, 2023b.
Tao Li, Qinghua Tao, Weihao Yan, Yingwen Wu, Zehao Lei, Kun Fang, Mingzhen He, and Xiaolin Huang. Revisiting random weight perturbation for efficiently improving generalization. Transactions on Machine Learning Research, 2024c.
Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey, 2023c.
Tsung-Y: Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. European Conference on Computer Vision, pp. 740-755, 2014.
21

---

Published in Transactions on Machine Learning Research (04/2025)
Jinming Liu, Heming Sun, and Jiro Katto. Learned image compression with mixed transformer-cnn archi- tectures. Proceedings of the IEEE/CVE Conference on Compuler Vision and Pallern Recognilion, pp- 14388 14397, June 2023.
Zhening Liu, Xinjie Zhang, Jiawei Shao, Zehong Lin, and Jun Zhang. Bidirrctiinall stereo image compression with cross-dimensional entropy model. Procecdings of the European Conference on Computer Vision, pp. 480-496, 2024.
Bethany Lusch, J Nathan Kutz, and Steven L Brunton. Deep learning for universal lincar embeddings of nonlinear dynamics. Nature commmaiccaiions, 9(1):4950, 2018.
James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research, (144)))1-76, 2020.
David Minnen and Nick Johnston. Advancing the rale-distortioo-commptaaiiio frontier for neural image compression. 2023 IEEE International Conference on Image Processing, pp. 2940-2944, 2023.
David Minnen and Saurabh Singh. Channel-wise autoregressive entropy models for learned image compres- sion. 2020 TEEE International Conference on Image Processing, pp. 3339 3343, 2020.
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. Proceedings of the IEEE/CVI /CFF Conference on Computer Vision and Pateen Recognition, pp. 11264 11272, 2019.
Daniel Morales-Brotee Thijs Vogels, and Hadrien Hendrikx. Exponential moving average of weights in doep learning: Dynamics and benefits. Transactions on Machine Learning Research, 2024.
Noga Mudrik, Yenho Chen, Eva Yezerets, Christopher J Rozell, and Adam S Charles. Decomposed linear dynamical systems (dlds) for learning the latent components of neural dynamics. Journal of Machine Learning Research, (5(59):1 44, 2024.
Daniel Müllner. Modern hierarchical, agglomerative clustering algorithms, 2011.
Seonghyeon Nam, Abhijith Punnappurath, Marcus A Brubaker, and Michael S Brown. Learning srgb-to-raw- rgb de-rendering with content-aware metadata. Proceedings of the IEEE/ //VF Conference Computer Vision and Pattern Recognition, pp. 17704-17713, 2022.
Meenal V Narkhede, Prashant P Bartakke, and Mukul S Sutaone. A review on weight initialization strategies for neural networks. Artificial intelligence review, 55(1):2911322 -222,, 2022.
Zhangkai Ni, Lin Ma, Huanqiang Zeng, Ying Fu, Lu Xing, and Kai-Kuang Ma. Seid: database for screen content images quality assessment. Proceedings of International Symposium on Intelligent Signal Processing and Communication Systems, pp. 774 779, 2017.
Roman Novak, Yasaman Bahri, Daniel A. Abolafia, Jeffrey Pennington, and Jascha Sohl-Dickstein. Sensi- tivity and generalization in neural networks: an empirical study. International Conference on Learning Representations, 2018.
Sandra Oltra and Oscar Valero. Banach's fixed point theorem for partial metric spaces, 2004.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM Journal on Control and Oplimization, 30(4):838 855, 1992.
Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer is secretly linear, 2024.
Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlincar dynamics of learning in deep linear neural networks, 2013.
Tom Schaul, Sixin Zhang, and Yann LeCun. No more pesky learning rates. Proceedings of the 30th Interna- tional Conference on Machine Learning, 28(3):343 351, 2013.
22

---

Published in Transactions on Machine Learning Research (04/2025)
Peter J Schmid. Dynamic mode decomposition of numerical and experimental data. Journal of Fluid Mechanics, 656:5- 28, 2010.
Peter J Schmid. Dynamic mode decomposition and its variants. Annual Review of Fluid Mechanics, 54(1): 225-254, 2022.
Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu. Temporal context. mining for learned video compression. IEEE Transactions Multimedia, 25:7311-7322 2022.
Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. Proceedings of the International Conference on Machine Learning, 202:32211 222222122222,,, 2023.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2016.
Lvfang Tao, Wei Gao, Ge Li, and Chenhao Zhang, Adanie: Towards practical neural image compression via dynamic transform routing. Proceedings of the IEEE/CW /CVF International Conference on Computer Vision, pp. 18879-16888, October 2023.
Yufei Wang, Yi Yu, Wenhan Yang, Lanqing Guo, Lap-Pui Chau, Alex C Kot, and Bihan Wen. Beyond learned vetadata-based raw image reconstruction. International Journal of Computer Vision, 132(12): 5514-5533, 2024.
Tsui-Wei Weng, Pu Zhao, Sijia Liu, Pun-Yu Chen, Xue Lin, and Luca Daniel. Towards certificated model robustness against weight perturbations. Proceedings of the AAAI Conference Artificial Intelligence, 34(04):6356-6333 2020.
Matthias Wödlinger, Jan Kotera, Manuel Keglevie, Jan Xu, and Robert Sablatnig. Eosic: attention for stereo image compression. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp.
Epipolar cross
2024.
Yuhuai Wu, Mongye Ren, Renjic Liao, and Roger Grosse. Understanding short-horizon bias in stochastic mela-opuimizztion. International Conference on Learning Representations, 2018.
Shao Xiang and Qiaokang Liang. Remote sensing image compression based high-frequency and low- frequency components. IEEE Transactions Geoscience and Remote Sensing, 2024.
Huan Yang, Yuming Fang, and Weisi Lin. Perceptual quality assessment of screen content images. IEEE Transactions on Image Processing, 24(11):4408 4421, 2015.
Wenchuan Yang, Haoran Yu, Baojiang Cui, Runqi Sui, and Tianyu Gu. Deep neural network pruming method based sensitive layers and reinforcement learning. Artificial Intelligence Review, 56(Suppl 2): 1897-1917, 2023.
Yibo Yang and Stephan Mandt. Computatiioalll--effceent neural image compression with shallow decoders. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 530-540, 2023.
Edouard Yvinec, Arnaud Dapoguy, and Kevin Bailly, Safer: Layer-level sensitivity assessment for efficient and robust neural network inference, 2023.
Chiyuan Zhang, Samy Bengio, and Yoram Singer. Are all layers created equal? Journal of Machine Learning Research, (677) 28, 2022.
Guodong Zhang, Lala Li, Zachary Nado, James Martens, Sushant Sachdeva, George E Dahl, Christopher J Shallue, and Roger Grosse. Which algorithmic choices matter at which batch sizes? insights from a noisy quadratic model. Advances in Neural Information Processing Systems, 2019a.
Jiangtao Zhang, Shunyu Liu, Jie Song, Tongtian Zhu, Zhengqi Xu, and Mingli Song. Lookaround optimizer: k steps around, step average. Thirty-sevente Conference on Neural Information Processing Systems, 2023.
23

---

Published in Transactions on Machine Learning Research (04/2025)
Michael Zhang, James Lucas, Jimmy Ba, and Geoftrey E Hinton. Lookahead optimizer: steps forward, 1 step back. Advances in Neural Information. Processing Systems, 32, 2019b.
Yichi Zhang, Zhihao Duan, Yuming Huang, and Fengqing Zhu. Theoretical bound-guid guided hierarchical vae for neural image codees. 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, 2024a.
Yichi Zhang, Zhihao Duan, Ming Lu, Dandan Ding, Fengqing Zhu, and Zhan Ma. Another way to the top: Exploit contextual clustering in learned image coding, Proceedings of the AAAI Conference on Artificial Intelligence, 38(8):9377-9388 2024b.
Yichi Zhang, Zhihao Duan, and Fengqing Zhu. On elficient neural network architectures for image compres- sion. 2024 IEEE International Conference on Image Processing, 2024c.
Yichi Zhang, Zhihao Duan, Yuning Huang, and Fengqing Zhu. Balanced rate-distortion optimization in learned image compression. arXiv preprint arXiv:2502.2010 2025.
Fangtao Zhou, Xiaofeng Huang, Peng Zhang, Meng Wang, Zhao Wang, Yang Zhou, and Haibing Yin. En- hanced screen content image compression: synergistic approach for structural fidelity and text integrity preservation. Proceedings of the ACM International Conference Mullimedia, pp. 7900-7908, 2024.
Pan Zhou, Hanshu Yan, Xiaotong Yuan, Jiashi Feng, and Shuicheng Yan. Towards understanding why lookahead generalizes better than sgd and beyond. Advances in Neural Information Processing Systems, 34:27290-27880 27304, 2021a.
Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, and Tong Zhang. Efficient neural network training via forward and backward propagation sparsification. Advances in Neural Information Processing Systems, 34:15216-15229, 2021b.
A
Appendix
This document provides more details about proposed method, discussion, and comparison.
A.1 Specific Domain Learned Image Compression Acceleration:
To demonstrate the robustness and real-world applicability of our method, we extend its evaluation to domain-specific scenarios, including stereo image compression, remote sensing image compression, screen content image compression, and raw image compression.
A.1.1 Stereo Image Compression
We first compare our proposed method with standard SGD-trained models stereo LICs, specifically ECSIC (Wödlinger et al. 2024) and BISIC (Liu et al. 2024), to showcase its superior performance. Standard SGD-trained models serve as the anchor for calculating BD-Rate [Bjontegaard 2001).
Training Details. Following established protocols (Wödlinger et al. 2024 Liu et al. 2024), we use the
Cityscapes (Cordts et al. 2016) and InStereo2K (Bao et al. 2020) datasets for training and evaluation.
Cityscapes: This dataset contains 5,000 outdoor stereo image pairs (2048 1024 resolution), with 2,975 pairs allocated for training and 1,525 for testing.
InStereo2K: This dataset consists of 2,060 indoor stereo image pairs (1080 x 860 resolution), split into 2,010 pairs for training and 50 for testing.
During training:
24

---

Published in Transactions on Machine Learning Research (04/2025)
For all models in the "SGD" series, the Adam optimizer (81 0.9, G2 =0.999) is used with 450 epochs Cityscapes and 650 epochs on InStereo2K.
For all models in the "Proposed" series, the Adam optimizer is also used, but training is limited 260 epochs on Cityscapes and 380 epochs on InStere02K.
A batch size of is used, with a learning rate initialized at x 10-1, adjusted via a Reducel RROn- Plateau scheduler (0.5 factor, 20 epochs patienco).
Hyperparameter Selection.
Prodefined epoch F 75 for Cityscapes and F 110 for InSterco2K.
Embodding percentage P = 0.27% for Cityscapes and P= 0.185% for InSteroo2K.
Embedding period: L moving average factor a 0.8, and optimizer step / 5.
These parameters are determined based on the main experiment settings. For instance, in the main LIC experiments, F is approximately 28.5% (70) of the total training epochs. Consequently, we set F 75 = 28.5% x 260 for Cityscapes. For the embedding percentage P, in the main experiment, 50% of the parameters are embedded over 50 epochs. Here, 50% of the parameters are embedded within 260 75 185 epochs. From Appondix 4.4 we know that the models benefit from smaller embodding percentages and more frequent embeddings. Therefore, we set P = 50% of sampled trajectories S are determined by traversing the same diagonal set described in Appendix 4.4.4. The resulting hyperparameters are M = 50, S 10k for ECSIC (25.74M) and M = 100, S = 20k for BISIC (78.21M). This dependency on model size underscores the adaptability our method across different architectures and different domains.
185 R 0.27% with L = 1. The number of modes M and the number
Results and Analysis. Fig. 10 presents the R-D curves of both methods, demonstrating that our ap- proach closely aligns with SGD-trained models. Tab. G highlights the BD-Rate reductions of our method compared to the SGD anchor, as well as the computational complexity in terms of training time, total train- able parameters, and final trainable parameters. Our method consistently achieves comparable or superior performance across datasets and methods. For example, on Cityscapes, "ECSIC + Proposed" achieves a BD-Rate reduction -187% compared to "ECSIC + SGD", even outperforming standard SGD some cases. Additionally, "ECSIC + Proposed" requires only 104 hours of training, which just 58. 38.75% of the training time for "ECSIC + SGD", highlighting significant efficiency improvements. The total train- able parameters are also significantly reduced. For "ECSIC SGD", the total trainable parameters are 25.7 75.74M x 450 epochs x 5 R-D points 57, 915M. In contrast, for "ECSIC Proposed": During the first 75 epochs, 100% of parameters x 25. 74M. From epochs 76 to 260, we embed 0.27% of parameters progressively reducing the trainable parameters from 100% to 50.32%. After calculations across all À values, the total trainable parameters for "ECSIC + Proposed" are reduced to 27,574M, approximately 47% of the total trainable parameters required by the SGD method. Similarly, the final trainable parameters after the last epoch are reduced from 25.74M to 12.99M. By reducing the number of trainable parameters and leveraging sparse training methods (Zhou et al. 2021b), our approach could further accelerate training, discussed in Appendix Sec. A.5
epoch,
Table 6: Computational Complexity and BD-Rate Compared to SGD for Stereo Image
BD-Rate (%)
Method
Training time
Total trainable params
Final trainable params
Citysrapes
InSteroo2K
ECSIC Wödlinger et al.
2024
+ SGD
177h
915M
25. 25.74M
0%
Propoeed
104h
27,574M
12.99M
-1.87%
-0.79%
SGD
207h
175,972M
78.21M
0%
0%
BISIC
LLu al.
Proposed
122h
83,702M
39.
-0.65%
-0.2%%
Training Conditions:1
Nvidia A40 GPU, AMD EPYC 7662 CPU, 1024GB RAM. Bold represents better
performance.
25

---

Published in Transactions on Machine Learning Research (04/2025)
41
40
42
39
40
PSAR (dB)
PSNR (dB)
38
36
35
36
ECSIC SGD
34
ECSIC + SGO
ECSIC Propesed
ECSIC Proposed
BISIC SGD
33
BISIC SGD
34
Proposed
Proposed
32
0.1
0.6
bpp
bpp
(a) Cityscapes
(b) InStereo2K
Figure 10: Rate-distortion curves in terms of PSNR on the Cityscapes and InStereo2K datasets.
A.1.2 Remote Sensing Image Compression
For remote sensing image compression, we evaluate our method on HL-RSComp.Net (Xiang & Liang 2024) using the following datasets:
GEset: This dataset consists of 8,064 RGB images, each with size of 640 x 640 pixels and a spatial resolution of approximately 0.35 m. Of these, 4,992 images are used for training, and 3,072 are used for testing.
GF1set: The GFlset dataset is derived from the Chinese GaoFen-1 (GF-1) satellite. It comprises
4,560 multispectral images with spatial resolution of 8 m and an image size of 640 x 640 pixels. Among these, 2,400 images are used for training, while the remaining images are reserved for testing. GF1 images include four spectral bands: Band 1 (0.45 0.52 pm), Band 2 (0.52-02 0.59 μm), Band 3 (0.63 0.69 μmm, and Band (0.77 0.89 pm).
GF7set: The GF7set is obtained from the Chinese GaoFen-7 (GF-7) satellite. It consists of 3,445 images for training and 2,297 images for testing. The images have a size of 640 x 640 pixels and a spatial resolution of 3.2 m. GF7 imagery also consists of four spectral bands: Band 1 (0.45-0.52 jum), Band (0.52 0.559 pmm) Band 3 (0.63-0.) 0.69 ym), and Band 4 (0.77-0.89 pm).
The panchromatic dataset is sourced from China's GF-6 satellite. It includes images with a spatial resolution of 2 m and a spectral range 0.45 0.90 pmm total of 2,100 images are used for training, and 1,600 are used for testing. Each image has a size 512 512 pixels. This dataset used evaluate the compression performance of different methods higher resolutions.
PANset:
Training Configuration. During training, the following seltings are applied:
For all models in the "SGD" series, the Adam optimizer (81 = 0.9, 32 = 0.999) is used, and the models are trained for 600 epochs.
For all models in the "Proposed" series, the Adamn optimizer is also used, but the training is limited to 350 epochs.
The batch size is set 8, and the learning rate is initialized at 10-4, A MultiStepLR scheduler with decay factor of 0.1 and step size of 200 epochs is employed.
Hyperparameter Selection. The hyperparameters for method defined follows:
26

---

Published in Transactions on Machine Learning Research (04/2025)
Predefined epoch F 100.
Embodding percentage P = 0.2%.
Embedding period I. = 1, moving average lactor 0.8, and optimizer step I 5.
These parameters are also determined based on the main experimental settings. Similar to the previous experiment, the number of modes M and sampled trajectories S are determined by traversing the diagonal set described Appendix 4.4.4. The resulting hyperparameters for HT-RSCommNNt (29.34M) are M and S 10k. This consistent dependence on model size highlights the adaptability of our proposed method across different archilectures and domains.
Results and Analysis. Fig. shows the R-D curves for both methods, demonstrating that our approach closcly matches the performance of SGD-trained models. Tab. 7 provides a detailed comparison of BD- Rate reductions achieved by our method relative to the SGD anchor, along with metrics for computations complexity, inchuding training time, total trainable parameters, and final trainable parameters. Our method consistently delivers comparable or superior performance across all datascts and methods. Additionally, it significantly accelerates the training of HL-RSComp.Vet, requiring only 60% of the training time while maintaining competitive performance.
36.5
40
15.5
33.5
PSWV (050
PSWA
26
32.5
34.0
32.0
33.5
36
Nu-R0Cmmatee: Propaed 150m)
Hu-tecanummi (ibee)
13.0
NLAASCmmtte:: Prepwed
$1.0
-U-bcarpter: Siore rol
1LASCemphe:: Fopoued
1L#fCcempiie Frencuet Botnt)
0.6
0.
12
24
18
04
3.5
(a) GEset
(b) GF1set
(c) CF7set
(d) PANset
Figure 11: Rate-distortion curves with different methods on all the datasets.
Table 7: Computational Complexity and BD-Rate Compared to SGD for Remote Sensing Image
Total trainable
Findd trsinable
BD-Rate (%)
Method
Training time
GEset
GF leet
GF7set
PANset
IIL-NSCompNet Xiang & Liang 2024
SGD
103h
105.6240 055624M
29:34M
0%
0%
0%
0%
Proposrd
62h
50,655M
14.67M
-0.10%
-0.59%
-0.77%
-0.74%
Training Conditions: performance.
Nvidia A40 GPU, AMD EPYC 7662 CPU, 1024GB RAM. Bold represents better
A.1.3 Screen content image compression
For screen content image compression, we compare our proposed method with standard SGD-trained models on SFTIP (Zhou et al. 2024).
Training Details. We follow the configurations specified in the paper (Zhou et al. 2024 for training. The models trained the JPEGAT dataset. (ISO/TEC JTC 1/SC29 WG1 2023), which includes 5,264 images for training and 350 images for validation. For evaluation, 2015), comprising 22 high-resolution, high-quality images, and the SCID dataset (Ni et al. 200 high-resolution, high-quality images.
the SIQAD dataset (Yang et al.
2017 containing
Training Configuration:
For all models in the "SGD" series, the models trained using the Adam optimizer for 200 epochs.
27

---

Published in Transactions on Machine Learning Research (04/2025)
For all models in the "Proposed" series, also use the Adam optimizer but train for only 120 epochs.
The batch size is set to 8, and the learning rate is initialized at 1 10-1, with ReduceL Plateau scheduler (0.3 factor, 4 epochs patience).
The hyperparameters are determined as follows:
Predefined epoch F 36.
Embodding percentage P 0.6%.
Embodding period == 1, moving average factor 0.8, and optimizer stop I 5.
These parameters are similarly determined using the same process as in previous experiments. The number of modes M and the number of sampled trajectorics S are determined by traversing the same diagonal set. The resulting hyperparameters are 100, 20k for SFTIP (95.49M).
Results and Analysis. Fig. 12 shows the R-D curves for both methods, demonstrating that our approach closely matches the performance of SGD-trained models. Tab. g provides detailed comparison of BD- Rate reductions achieved by our method relative to the SGD anchor, along with metrics for computational complexity, including training time, total trainable parameters, and final trainable parameters. Our method consistently delivers comparable or superior performance across all datasets and methods. Additionally, it significantly accelerates the training of SFTIP, requiring only 64% the training time while maintaining competitive performance.
38
33
38
37
37
PSMR (88)
PSNR (c8)
(o0)
31
35
29NR
30
35
34
SFTP SGD
SFTIP
33
SF SED
32
bpp
(a) JPEGAI
(b) SIQAD
(c) SCID
Figure 12: Rate-distortion curves with different compression methods on all the datasets.
Table 8: Computational Complexity and BD-Rate Compared to SGD for Screen Content Image
Total trainable
Final trainable
BD-Rate (%)
Method
Training time +
params
params 4
JPEGAI
SIQAD
SCID
SGD
50h
9504 47,307M
95.49M
0%
0%
0%
SFTIP Zhou et al.
2024
Proposed
32h
47.36M
-0.93%
-0.77%
-0.74%
Training Conditions:
Nvidia A40 GPU, AMD EPYC 7662 CPU, 1024GB RAM. Bold represents better
performance.
28

---

Published in Transactions on Machine Learning Research (04/2025)
A.1.4 RAW image compression
For RAW image compression, we compare our proposed method against standard SGD-trained models on R2LCM (ang el al. 2024).
Training Details. We follow the training configurations specified in (Wang al 2024 Nam et al. 2022) The models are trained on the NUS dataset (Cheng et al. 2014), which contains images captured by multiple cameras. Specifically, we use three cameras- Samsung NX2000, Olympus E-PL6, and Sony SLT- A57-com A57-comprising 202, 208, and 268 raw images, respectively. To preprocess the data, we first demosaic the raw Bayer images using standard bilinear interpolation to obtain 3-chamnel raw-RGB images. These images are then processed through a software ISP emmlator (Karaimer & Brown 2016) to generate sRGB images, mimicking real-world photo-finisinn applied by cameras. The dataset is split into training, validation, and test sets following the protocol (Nam et al., 2022
Training Configuration. To ensure fair comparison, both "SGD" and "Proposed" models are trained using the Adam optimizer with the following settings:
SGD Series: Trained for 500 epochs.
Proposed Scries: Trained for only 290 epochs, significantly reducing training time.
Batch Size: 8
Learning Rate: Initialized at 1 10 with ReduceLROnnPlteen scheduler (decay factor 0.1, patience of 60 epochs).
Hyperparameter Selection. The key hyperparameters are set as follows, using the same selection strategy as in previous experiments:
Predefined epoch: == 80
Embedding percentage: p 0.24%
Embedding period: I, moving average factor = 0.8, and optimizer step / l= 5.
The mumber of modes M and the mumber of sampled trajectorics S are determined by traversing the diagonal set, yielding M = 10 and S = 2k for R2LCM (2.35M).
Results and Analysis. Fig. presents the R-D curves for both methods, showing that our proposed approach achicves performance comparable to that of SGD-trained models. Tab. g provides a detailed BD- Rate comparison relative to the SGD baseline, alongside key computational metries such as training time, total trainable parameters, and final trainable parameters. Our method consistently delivers comparable superior performance across all datasets and model variants. Notably, it significantly accelerates R2LCM training, requiring only 61% of the training time while maintaining competitive performance. This highlights the effectiveness of our approach in improving training efficiency without compromising model quality on raw image compression task.
Table 9: Computational Complexity and BD-Rate Compared to SGD for RAW Image
Total trainable
Final trainable params
BD-Rate (%)
Method
Training time
params
Samsung
Olympus
Sony
+ SGD Proposed
49h
1175M 569M
2.35M
0%
0%
0%
R2LCM Wang et al. 2024
30h
1.16M
-1.16%
-0.93%
-1.10%
Training Conditions:
Nvidia A40 GPU, AMD EPYC 7662 CPU, 1021GB RAM. Bold represents better
performance.
29

---

Published in Transactions on Machine Learning Research (04/2025)
58
57
54
56
PSNR (88)
(d8)
PSNR (dl)
52
55
34
52
53
48
RZLCM SGD
R2LCCM 550
R2ICM SED
RZLCH Proposed
46
0.200
(a) Samsung
(b) Olympus
(c) Sony
Figure 13: Rate-distortion curves with different compression methods on all the datasets.
As has been demonstrated the exlensive experiments on stereo images, remote sensing images, screen content images, and raw images, our method consistently performs effectively across diverse domain-specifie image types while adhering to unified hyperparameter selection strategy. Notably, approach achieves these results with only 60% of the training time required by standard methods, without any degradation in performance- and in some cases, even showing slight improvements. These findings underscore the capability of our method to significantly accelerate the training process for various image codecs, making it a highly practical and promising solution for real-world applications.
A.2 Variance reduction
We performed additional experiments to evaluate the variance in model performance across five training runs, comparing the average value and variance of performance between our method and SGD using ELIC (He 2022 as a reference. Specifically, we trained the ELIC model with all As five times using both our method and SGD. The anchor is VVC-intra with a yuv444 config following standard practie (He et al. 2022 Liu et al. 2023 Li et al. 2024a Zhang al. 2024b). As shown in Tab. TO our method achieves both lower average BD-Rate and reduced variance compared to SGD, These results are consistent with our theoretical analysis, further underscoring the stability of the proposed method.
ct al.
Table 10:
Performance comparison across five runs.
Method
Avg BD-Rate (Variance)
ELIC + SGD
-5.83% (0.0184)
ELIC + Proposed
-6.53% (0.0023)
To gain deoper insights into the variance bohavior of the training process. we present the mean loss curve for À = 0.0018 along with its standard deviation"] range, as shown in Fig. 15 The solid line represents the mean loss computed over five independent runs at each training epoch, while the shaded region denotes the corresponding standard deviation, capturing the variability in training dynamics. As observed, the variance of standard SGD is significantly larger than that of the proposed method. This observation is consistent with the BD-Rate results in Tab. 10 and the theoretical analysis in Sec. A.6 further demonstrating the low-variance property of our proposed approach.
A.3 Discussion STDET and SMA
To better understand the distinct contributions of cach proposed technique, we conducted additional ex- periments to evaluate them independently. The experimental settings were kept identical to those used in
Standard deviation observable due the relatively small value of variance
30

---

Published in Transactions on Machine Learning Research (04/2025)
ELIC SGD
0.48
ELIC Proposed
0.44
R-D loss
0.40
0.36
25
50
75
100
Epoch
Figure 14:
Testing loss behavior across five independent runs for different methods. The evaluation is conducted on the Kodak dataset with À 0.0018. The testing R-D loss computed as 255 MSE Bpp.
the main experiments. We employed the ELIC model He el al. 2022 with À = 0.0018 as the illustrative baseline. The results are depicted in Fig.
First, we evaluated STDET without incorporating SMA (ELIC + STDET). When applied in isolation, STDET struggled to deliver a competitive performance. This limitation arises from the stochastic nature of the training process, which introduces variability and affects the consistency and accuracy of STDET's correlation modeling. Consequently, the inaccuracics in parameter embedding lead to noticcable perfor- mance drop. This highlights the need for complementary techniques to smooth and stabilize the training trajectories.
Next, we applied SMA independently to standard SGD-trained LIC model (ELIC + SMA). The results demonstrate that SMA achieves R-D performance comparable to both standard SGD and the final combined setup ("STDET + SMA"), with the addod benefit of smoother training trajectory. However, does not accelerate the training process. This confirms that SMA primarily acts smoothing and stabilizing technique when used alone and does not directly contribute to additional performance improvements much faster convergence, anticipated.
Overall, the combination of STDET and SMA offers the best balance between training efficiency and per- formance. STDET reduces the dimensionality of the training process, thereby accelerating training under stable correlation modeling, while SMA ensures stability throughout the process, enabling accurate corre- lation modeling, which required by STDET. Together, these techniques enable an efficient and accurate acceleration of the LIC training process.
A.4 Comparison of EMA, explicit alignment and SMA
To clarify the differences between EMA, explicit aligument, and SMA, Fig. 16 provides illustrative com- parison of the three mothods over four training iterations.
For EMA, In each training iteration, we first perform a standard SGD update on the model parameters WSGD. Then, we update the EMA parameters WEMA using the newly obtained WSGD and the previous WEMA state. Notably, during training, WEMA does influence WsGD-it updated passively and used only for testing after training ends. The trajectory of wSGD remains noisy, while WEMA follows smoother trajectory.
31

---

Published in Transactions on Machine Learning Research (04/2025)
ELIC 5GD(0.3444)
0.48
ELIC + STDET(0.3616)
ELIC + 3M4(0.3442)
ELIC Proposed(0.3442)
0.44
R-D loss
0..40
0.36
0
25
50
75
100
125
Epoch
Figure 15: Testing loss comparison of various techniques. The upper right corner shows the final convergence result. À = 0.0018, Testing R-D loss = A 255² MSE + Bpp. Evaluated on the Kodak dataset.
Gradient updale
EMA rule upditie
Test model
Synchronize weight
Test model
WaGD
Her
ter
ier
Train
fier
iter
iter
ier
Tmin
W
(a) EMA
(c) Proposed SMA 1)
Alignment
iter
Train
WAGD
iler
Trin
iter
Init
Inil
Wyus
(b) Explicit alignment
(c) Proposed SMA - 2)
Figure 16: Illustrative comparison of EMA (Szegedy et al. 2016 Polyak & Juditsky 1992), explicit align- ment (Song al. 2023), and SMA (I
& I 2) over four training iterations.
32

---

Published in Transactions on Machine Learning Research (04/2025)
For explicit alignment, the process begins similarly: WSGD is updated through standard SGD, and WEMA is computed following the FMA update rule. However, an additional loss function is introduced to align the SGD trajectory with the EMA trajectory explicitly. This loss is computed based on the discrepancy between the outputs of the current SGD model and the EMA model. By minimizing this loss, WSGD is encouraged to follow a smoother trajectory that aligns with WEMA- After sufficient training, WSGD and WEMA ideally converge to the same weight, leading to stabilized final model. However, this method introducos extra computational overhead due to the additional loss calculation and gradient updates.
For SMA, periodically synchronize the EMA-updated parameters at sampled intervals to the WsGD.
Taking as an example:
In the first iteration, we perform standard SGD update on WSGD- After the update, we sample this iteration's parameters and update WSMA using the EMA updale rule.
Once WSMA is updated, we synchronize it back to WSCD, and the synchronized WSGD is used as the starting point for the next iteration. This process seen as the second iteration.
In the third iteration, another normal SGD update is performed, followed by an update to uSMA in the fourth iteration, where synchronization occurs again.
This pattern contimues throughout training.
For the case of 1 the process remains the same, except that two consccutive SGD iterations occur before updating иşMA- The next synchronization step then serves iteration. This pattern repeats, alternating between two SGD iterations and an SMA update with synchronization.
In essence, our approach periodically samples parameters from the SGD trajectory, applies an EMA-style update to obtain WSMA, and synchronizes it back WSaD before continuing training. Since WSM follows EMA-style update, it naturally forms smoother trajectory, making the final training trajectory more stable.
In summary, both explicit alignment and SMA integrate EMA into the training phase to enhance stability. However, SMA achieves this in a more computationally efficient manner, as it does not have additional loss computations and gradient updates. Instead, it interpolates sampled SGD parameters and synchronizes them with WSGD, offering a simple yet effective way smooth the training process.
A.5 Limitations and future work.
Additional Memory Consumption:
The proposed method requires more memory due to additional non-learnable parameters. The notable additional parameters and their relative sizes are detailed below:
STDET: Vectors K and D, each with size equal to that of the model parameters.
STDET: vector storing the mode related to each weight, with the same length as K but containing positive integers representing mode numbers. This vector's size is approximately 0.25 times that of the model.
SMA: SMA model parameters WSMA, with a size equal to that of the model parameters.
Note that the GPU memory used to store these parameters insignificant during the training process compared to the sustained gradients (Izmailov et al. 2018), making the overall increase negligible.
Integration of Sparse Training: With the embedding of LICs' parameters, they cease to be learnable, meaning that in practice, we can avoid the time-consuming gradient computations during backpropagation for these parameters. However, due to the constraints of existing platforms like TensorFlow and Pororc,, this not straightforwardly achicvable. These platforms rely on the chain rule to compute gradients, and does not prevent the computation of these parameters within the chain rule; it only avoids the storage of their gradients. A potential solution could involve integrating
simply setting
33

---

Published in Transactions on Machine Learning Research (04/2025)
the proposed approach with sparse training techniques (Zhou et al 2021b thereby completely bypassing gradient computations for these parameters during each training iteration. As a result, with an increasing number of embedded parameters, the training time per epoch would decrease, further accelerating the overall training process. Nevertheless, implementing such an optimization within current frameworks is non-trivial and beyond the scope of this work, leaving it as a promising direction for future research.
Broader Application to Learned Video Compression: Our proposed method has been shown to effectively accelerate wide range of LICs across diverse domain images through both theoretical analysis and extensive experiments. A natural extension would be its application to learned video compression (LVC). Notably, the intra-frame coding component of LVC models and the contextual coders in the DCVC series (Li
et al.) 2021 Sheng et al.) 2022: Li et al.) 2022a 2023a 2024b Jia et al., 2025) ) are fundamentally LICs. Therefore, our method could be utilized to directly accelerate the training of these frame coders. However,
the motion estimation and temporal modeling components DCVCs introduce additional complexities that require further investigation. Exploring ways to extend our acceleration approach to these components remains open challenge for future work.
A.6 Noisy quadratic analysis
In this section, we analyze the proposed method on a noisy quadratic model. The quadratic noise function is a commonly adopted base model for analyzing the optimization process, where the stochasticity incorporates
noise introduced by mini-batch sampling (Schaul et al., 2013 Wu al. 2018 Zhang et al. 2019b Koh &
Liang 2017). We will show that the proposed method converges to smaller steady-state training variance than the normal SGD training.
Model Definition, The quadratic noise model is defined as:
1 ((0) 2 c) A(O c)- c),
(11)
where C ~ NN Σ), and Σ are diagonal matrices, and 6" = O (Wu et al. 2018 Zhang et al., 2019a 2023). Let a, and of be the i-th elements on the diagonal of and E, respectively. The expected loss of the iterates 0g can be written as:
((0,) = E(L(O,)) = Σ a (E + oz)),
(12)
where oG represents the i-th clement of the parameter 0. The steady-state risk of SGD and the proposed method are compared by unwrapping [[O:] and Vi0,] in Eq. 12 The expectation trajectories [[0,] contract to zero in both SGD and the proposed method. Thus, we focus on the variance:
Stcady-state risk. Let 0< 7 1/L be the learning rate satisfying L = max; a, In the noisy quadratic setup, the variance of the iterates obtained by SGD and the proposed method converges to the following matrices:
722 A
VSGD
(13)
I- (I AA)2
p pIE )a (I 11-7/4 2
VA Proposed az
V
VSDD
(14)
7
A 21 + 20 a) (I 7A))
ST, aE(0.1) and B|G]S1
where denotes the SMA weighting factor with varying trajectory states, p is the percentage of final embedded parameters, and [[A;]] is the expectation of squared affine coefficients k.
From Eq. [4 it is evident that for the steady-state variance of the proposed method, the first product term is always smaller than for E 10, and E[ < Therefore, the proposed method exhibits a smaller steady-state variance than the SGD optimizer the same learning rate, resulting in a lower expected loss (see Eq. 12 Additionally, due to the reduced variance in the training process, the correlation within the
34

---

Published in Transactions on Machine Learning Research (04/2025)
same modes is highly preserved, ensuring that the STDET avoids serious failure cases. The small steady- state variance ensures good generalizability in real-world scenarios the proposed method converges well.
(Wu et al. 2018 and demonstrates why
We use to represent normal SGD training parameters and ø to represent SMA-updated training parameters.
Proof. Here we present the proof of Eq. 14
Stochastic dynamics of SGD: From
Wu et al. 2018 we can obtain the dynamics of SGD with learning
rate 7 follows:
(I /A) 1, ([x = 7 + +7 A
(15)
(16)
Stochastic dynamics of Proposed method: We now compute the dynamics of the proposed method. The expectation and variance of the SMA have the following iterates based on the lookahead optimizers (Zhang
et al. 2019b 2023):
The expectation trajectory is represented as:
E[Ør-1] = (1 - a [o)] + aE(0
= =1 - a 7A)
(17)
a + a - > B[o,].
When combined with STDET, in the training dynamics, although the embedded parameters not trainable, they depend on the reference paramcters, which are also updated by SMA. Thus, all the paramcters follow the same expectation dynamies.
For the variance, we can write
V[dr+1] = = alo + 2a(11 - a)cov(or,0r.)
(18)
Also, it's casy to show:
v(O..] (I (d-7A)21 2 [b. 7 Σ (I 7 A
(19)
cov(oi, 0t.1) = (I - - V (o).
After substituting the SGD variance formula and some rearranging, we have:
Vi0++ = [1 a - [
+ a
(20)
Similarly, for the variance term, all the parameters follow the same SMA dynamics. Then, for the proposed method:
E[or+1] = [1 a + aalaa((Ia)))))) E(o,],
(21)
V(d++1] = =
[1 (1 a(I ( V
+
I 7
(22)
1=0
Now, we proceed to find the steady state of the expectation and variance: The given expectation term is:
E E [1 a -I->AA)] E(o.).
(23)
Define the mapping T as:
T(h.) [1 a(I [A]]- Φr-
(24)
35

---

Published in Transactions on Machine Learning Research (04/2025)
A mapping T is a contraction if there exists constant 0 S < 1 such that for all Φi and d/:
[T(()) - - s c - dill.
(25)
In our case:
[T(())) - T()))1 = |||| a a aa---A)) Φ: - 11
dil
(26)
= -
a a a[I-a\) (Ø. --)
Let M 1 =1-a+a(I--Aa We need to find the spectral radius p(M), which represents the largest absolute
value of the cigenvalues of M. Since (A) is contraction matrix (assuming the learning rate ry is chosen such that 0 y < I/L, where L max, a,), its eigenvalues X satisfy:
A < 1.
(27)
Therefore, the eigenvalues of (I- (I- are X and satisfy:
| 1.
(28)
Next, consider the matrix M:
M = ==1-a+a(1 a(-
(29)
The eigenvalues of M are given by:
((M) = 1 + ax!l
(30)
For the mapping to a contraction, the spectral radius p(M) must be less than
1 a + +a <1
(31)
Given < a < and X 1, the inequality |1 a at 1 will hold. By Banach's Fixed Point
Theorem Oltra & Valero 2004). since T is a contraction mapping, there exists a unique fixed point E such that T(E*)= E*.
To find the fixed point, set E[dr+i] = B[cbr] = E":
='= [1 a1 a a E
(32)
Simplifying:
[a-a(I-7A)} E" Ex== O.
(33)
Since O. The only solution is:
E* 0.
(34)
Using the contraction mapping approach, we have shown that the expectation term indeed converges to the fixed point Ex 0.
Now, we need to consider the effect of STDET. Let p represent the percentage final embedded parameters. Thus E = =F= + - Ftmbeed where E - and E- E E Ft + E + E[d,), where we empirically find that [[d] 0 in our cases. Thus, overall, the expectation trajectories contract to zero. Clearly, the expectation trajectories of SGD also contract to zero as it is also contract map.
Similarly, for the variance, under the same condition, we know that:
VSGD (1 7A) + +>2A2E
A
(35)
I (I 7
36

---

Published in Transactions on Machine Learning Research (04/2025)
For the proposed method,
V
[1 - +
11-++all a =+I[I=AA]]
+
SMA
1m0
a²
A
I- [11- a a 7
a (I (1-1A )
A
(36)
I- [11 a)I + a1I 7
I (I -7A
a² (I 7
A
a²(
(I 7A) 2) ) + 2a a) (1-rA)
(I -1---00 7A)
a (1 (A)
VSSDD
A 21 + 2a(1 a) (I-
Also, together with the STDET:
V Proposod
VVibbodd + (1 V) eemoad
= [][(k]) 1) VSMA + (1 - p)) V)
aP (1- + PIE Kb 1)a² (I (I - 7
(37)
A 221) 20 ai((1 7)))
SGDS
ST, if (v.(). and B|A
Thus, we have shown Eq. 14 and the proposed method will achieve a smaller loss for the same learning rate as the variance is reduced Eq.
A.7
Detailed algorithm
The full practical STDET algorithm is presented in Algorithm 1
A.8
Experiments details
A.8.1
Detailed Training Settings
We list detailed training information in Tab 11 including data augmentation, hyperparameters, and training devices.
A.8.2
Used implementations
We list the implementations the learning- based image codecs that we used for comparison in Tab. 12 list the compared efficient training method implementations in Tab. 13
We
37

---

Published in Transactions on Machine Learning Research (04/2025)
Table 11: Training Hyperparameters.
Training set
COCO 2017 train
# images
118,287 Around 640xx120 Crop, h-flip 256x256
Image size
Data augmentation
Train input size
Optimizer
Adam
Learning rate
1 1 10
LR schedule
LR schedule parameters
factor: 0.5, patience: 5
Batch size
16
Epochs (SGD)
120 =0.0018), 80 (others)
Epochs (Proposed)
70 (A =0.00 =0.0018), 50 (others)
Gradient clip
2.0
GPUs
1 x RTX 4090
STDET
Predefined epoch F
20 (A =0.0018), 10 (others)
Embeddable parameters
75%
Embedding period
True embedding percentage P Dummy embedding percentage
1%
min (1%, 25%), where is the current epoch
SMA
Moving average factors Optimizer step
0.8
5
Table 12: Learning-baat basod Codees Implementations.
Method
Implementation
ELIC
He al. 2022)
TCM-S
LLu et al. 2023)
httper//ggtaabbbccmm/m/ooo//111p
FLIC
(Li et al. 2024a
ECSIC Wödllinge et al. 2024)
httpe///ghhaaabbcc////ddddddeegr
BiSIC (Luu al. 2024
attper/gggtaauuoom//////uagggggg
HL-RSCompNe (Xiang & Liang 2024h
SFTIP (Zhou et al
2024
R2LCM Wang al.
2024
http:://tttaab..oo//////11122110
Table 13: Efficient Training Methods Implementations.
Method
Implementation
P-SGD (Li et al.
2022b
https://gittub..c//bbll///////
P-BFGS [Li et al. 2022b
http:://giitabb.ccm//aat///D
TWA Li et al. 2023b
https://gitaubb.cc//bb//////
RigL Evei et al. 2020
SRigL Lasby et al.
2024
38

---

Published in Transactions on Machine Learning Research (04/2025)
Algorithm STDET Algorithin
hyper-parameters.
F (Predefined epochs), L (Embedding period), P (Embedding percentage), M (Number of modes) range, $ = 200M (Number of sampled trajectories) range, Other inputs required for the training process and STDET (learning rate, etc.)
Procedure:
1. Run regular SGD epochs obtain weight trajectories We.
Initialize variables: prev R-D loss + 10, 000.
3. for each M,5 = 200M within the pre-defined range do
(a) Uniformly sample S trajectories from W avoid calculating large N x N correlation matrix.
(b) Calculate the sampled trajectory correlation matrix C:
C|i.j] porr("e.ps C RSS..
Cluster the sampled trajectories into M modes using the Farthest Point hierarchical clustoring method Müllner
2011
(d) Select Wm,F as the parameter trajectory with the highest average correlation to the other parameters within each mode.
(c) Calculate the remaining (not sampled) trajectory correlations:
C2li, m] corr(w,, ",,,, w),,)) C2 C C
(f) Assign each wi C Wr to mode by selecting the max of the i-th row of C2(t,
Compute (F. KT Di each mode using the pseudo-inverse of Eq.
Rm(F)= Wm, (10m, ,w,
C min I/Wm,F Rm (F)
Kmo IlEN,
[Wmm and i (1, I,........
where || [|en denotes the Frobenius norm, 10m
(h) Get the CMD instant model by Eq. and evaluate CMD instant performance the current M,S S on the sampled dataset, caleulate R-D losseurrent-
Check for saturation:
R-D losscurrent prev R-D loss
ARD
prev R-D loss
If AR-D < 0.001 then
Keep current K, D, and mode decomposition results. Break loop.
(j) Update: prev R-D R-D losScurrent-
4. end for
5. Find embeddable parameters calculating the parameter sensitivity the sampled dataset.
Initialize the set of embedded weights:
7. for > F do
(a) Perform regular SGD epoch.
(b) Iteratively update K and D using Eq. G
(c) Update weights (t) as follows:
ktw, d, if
(x(())
SGD update if g I
(d) if (t F)%L= == 0 then
Compute long- term change:
C(t) (K(t) K(t L))² (D(t) D(t L))²
i. Update I include the weights with the least P% changes terms of C, excluding reference weights and already embedded weights. iii Reinitialize additional min %, 25%) parameters that have the least C changes excluding I by: (t) I de.
(c) end if end for
39


Reviews:
[{"review_id": "LVZvkF7d7n", "reviewer": "Reviewer_Qp3R", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The proposed method provides significant speedup compared to other models in the literature at without sacrificing performance\n- They provide a theoretical foundation analyzing the noisy quadratic model\n- They provide ample amount of experiments to validate their claims\n\nWeaknesses:\n- It is not clear how much of an impact the choice of hyperparameters has on performance, so it is not obvious whether the proposed method would perform as well on other practical data\n- Regarding technical novelty, the key ideas behind their method are well known in the literature, so on the purely technical side the contribution is limited", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose Sensitivity-aware True and Dummy Embedding Training mechanism (STDET) for Learned Image Compression (LIC) which is combined with Sampling-then-Moving Average (SMA) to regularize training state variances.\nThey separate parameters into nodes and then successively embeds lower sensitivity parameters.\nBy adaptively reducing the number of parameters during training, they achieve faster convergence without sacrificing performance.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Regarding the hyperparamers, is there a (structured) way to find good hyperparameters for new data distributions, i.e. is there a better approach than brute force (grid search).? If the authors have intuition about this it would be good to add.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "biatdhLZx9", "reviewer": "Reviewer_EKLo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**  \n(+) This work addresses a crucial challenge in accelerating the training of learned image compression (LIC) models, which are typically computationally expensive.  \n\n(+) The proposed STDET mechanism effectively reduces the number of trainable parameters by leveraging parameter correlations, leading to improved efficiency.  \n\n(+) The SMA technique provides a well-motivated strategy to stabilize training and minimize variance, ensuring smoother optimization.  \n\n(+) Theoretical analysis and extensive experiments on multiple LIC models demonstrate the effectiveness of the approach in reducing training time without sacrificing performance.  \n\n**Weaknesses:**  \n(-) The extent to which the proposed STDET mechanism generalizes across different LIC architectures remains unclear.  \n\n(-) While the method reduces training costs, the core idea of parameter clustering and embedding follows existing low-dimensional training strategies, limiting technical novelty.  \n\n(-) The ablation studies could be expanded to provide a more detailed analysis of the impact of different hyperparameter choices on the effectiveness of STDET and SMA.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work accelerates learned image compression (LIC) training by modeling neural training dynamics. The authors propose **STDET**, which clusters parameters into modes and progressively embeds non-reference parameters based on stability, reducing trainable parameters. They further introduce SMA, a moving average technique to smooth training and minimize variance. Theoretical analysis and experiments show significant training speedup without performance loss.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The proposed STDET mechanism clusters parameters into modes and embeds non-reference parameters dynamically, but its effectiveness across different LIC architectures remains unclear. The authors should provide additional empirical evidence on a broader set of models to demonstrate generalizability.  \n\n- The paper would benefit from more comprehensive ablation studies to analyze the impact of key components such as the embedding threshold, number of modes, and sensitivity estimation strategy. This would strengthen claims regarding the effectiveness of STDET and SMA.  \n\n- While comparisons with standard SGD and a few subspace methods are provided, additional baselines such as sparsity-based training techniques or recent efficient optimization methods for deep learning could provide further insight into the advantages and limitations of the proposed approach.  \n\n- Since the method relies on progressively reducing the number of trainable parameters, an analysis of potential instability or performance degradation in later training stages would be useful. Providing learning curves with variance analysis could help assess this.  \n\n- The paper highlights reduced training time, but a breakdown of computational savings in terms of FLOPs, memory usage, or energy consumption would add more depth to the efficiency discussion.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "js5MktKowC", "reviewer": "Reviewer_FNpp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The extensive experimental results show that the proposed method is quite effective in reducing training time without sacrificing performance.\n2) The theoretical analysis for the proposed STDET is well-studied and promising.\n\n\nWeaknesses:\n1) The technique contribution of SMA is not very clear. While the authors state that \"EMA is designed for the testing phase, I do not agree with this statement. A lot of research have applied EMA in this training phase, e.g. [1]. I do not see any technique contribution for the proposed SMA because it is just an EMA on the training phase. I suggest the authors clarify this difference between the proposed SMA and the EMA algorithm.\n2) The descriptions in Figure (1) need to be clarified. Why is epoch 70 chosen as the reference stage? Are 70 epochs enough for model convergence? In addition, I recommend adding one more column about epoch 0 to the table in Figure 1 so that the readers can know the parameter change during the whole optimization.\n3) Some notations need to be clarified. In Table 3, the meaning of the %value should be clarified in the table, rather than requiring the readers to check it in the main text.\n \n\n[1] Song, Yang, et al. \"Consistency Models.\" International Conference on Machine Learning. PMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an efficient training framework for Learned Image Compression (LIC) methods by reducing training costs without sacrificing performance. Specifically, the **Sensitivity-aware True and Dummy Embedding Training (STDET)** mechanism clusters parameters and gradually embeds less sensitive ones, lowering the number of trainable parameters. To stabilize this process, a **Sampling-then-Moving Average (SMA)** technique is introduced, ensuring smooth weight updates and reducing training variance.  Overall, the proposed approach accelerates LIC training with fewer parameters and reduced variance, offering insights for scalable and efficient LIC optimization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see weaknesses.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AJL3SQxVUi", "reviewer": "Reviewer_jsnA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n1. Introduce a novel STDET mechanism to approximate the stochastic gradient descent training of learned image codecs within a low-dimensional space.\n2. Provide a theoretical analysis using the noisy quadratic model, demonstrating that the proposed method achieves lower training variance compared to standard SGD.\n3. Achieve faster convergence speed without compromising compression performance.\n4. Conduct extensive studies on various types of image compression models.\n\nWeaknesses:\nThe paper could benefit from a discussion on future work, particularly regarding the application of the proposed method to other neural compression domains, such as video codecs. Exploring the adaptability and effectiveness of this approach across different types of neural compression could provide valuable insights and broaden its potential impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method for accelerating the training of existing learned image compression models. It proposes a sensitivity-aware true and dummy embedding training mechanism to achieve a linear approximation of complex nonlinear learned image compressors (LICs). Additionally, it integrates a sampling-then-moving average principle to update both reference and non-reference parameters. Experimental results indicate that the proposed method not only reduces training time but also slightly enhances compression performance across various advanced baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Is it possible to apply the proposed method to the latest neural video compression techniques, such as the DCVC codec series [1]? Please discuss this in the appendix.\n\n[1] https://github.com/microsoft/DCVC", "broader_impact_concerns": "I think there is not ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the paper brings useful insights to the LIC domain by accelerating training without compromising performance. While the core techniques (e.g., parameter embedding, smoothing) build on existing ideas, their combination and adaptation are novel in this context. Theoretical reasoning is sound, and experiments show consistent efficiency gains. Some concerns were raised around generalizability and marginal performance improvements, but these don't outweigh the practical value of the contribution.
Decision: Accept (poster)