On Quantizing Neural Representation for Variable-Rate Video Coding
Authors: Junqi Shi
Zhujia Chen
Hanfei Li
Qi Zhao
Ming Lu
Tong Chen
Zhan Ma
Venue: ICLR 2025 Spotlight (2025)
Keywords: Variable Rate
Video Coding
Quantization
Neural Representation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: This work introduces NeuroQuant, a novel post-training quantization (PTQ) approach tailored to non-generalized Implicit Neural Representations for variable-rate Video Coding (INR-VC). Unlike existing methods that require extensive weight retraining for each target bitrate, we hypothesize that variable-rate coding can be achieved by adjusting quantization parameters (QPs) of pre-trained weights. Our study reveals that traditional quantization methods, which assume inter-layer independence, are ineffective for non-generalized INR-VC models due to significant dependencies across layers. To address this, we redefine variable-rate INR-VC as a mixed-precision quantization problem and establish a theoretical framework for sensitivity criteria aimed at simplified, fine-grained rate control. Additionally, we propose network-wise calibration and channel-wise quantization strategies to minimize quantization-induced errors, arriving at a unified formula for representation-oriented PTQ calibration. Our experimental evaluations demonstrate that NeuroQuant significantly outperforms existing techniques in varying bitwidth quantization and compression efficiency, accelerating encoding by up to eight times and enabling quantization down to INT2 with minimal reconstruction loss. This work introduces variable-rate INR-VC for the first time and lays a theoretical foundation for future research in rate-distortion optimization, advancing the field of video coding technology. The materials
will be available at https://github.com/Eric-qi/NeuroQuant.

Published as a conference paper at ICLR 2025
ON QUANTIZING NEURAL REPRESENTATION FOR VARIABLE-RATE VIDEO CODING
Junqi Shi, Zhujia Chen, Hanfei Li, Qi Zhao, Ming Tong Chen, Zhan Ma School of Electronic Science and Engineering, Nanjing University
(junqi shi, zhuj jachen, hanfei 11 qi 2 @ -nju edu..ddu.com cn, [minglu, chentong, ma zaazaaaneee en ju..euu.cnn edu.cn
ABSTRACT
This work introduces NeuroQuant, novel post-training quantization (PTQ) ap- proach tailored to non-generalized Implicit Neural Representations for variable- rate Video Coding (INR-VC). Unlike existing methods that require extensive weight retraining for each target bitrate, we hypothesize that variable-rate cod- ing can be achieved by adjusting quantization parameters (QPs) of pre-trained weights. Our study reveals that traditional quantization methods, which assume inter-layer independence, are ineffective for non nonnggenrallized INR-VC VC models due to significant dependencies across layers. To address this, we redefine variable- rate INR-VC as mixed-precision quantization problem and establish a theoret- ical framework for sensitivity criteria aimed at simplified, fine-grained rate con- trol. Additionally, we propose Nework-wwss. calibration and channel-wise quan- tization strategies to minimize quantization-indun formula for representation-orienti tions demonstrate that NeuroQuant significantly outperforms existing techniques in varying bitwidth quantization and compression efficiency, accelerating encod- ing by up to eight times and enabling quantization down to INT2 with minimal reconstruction loss. This work introduces variable-rate INR-VC for the first time and lays a theoretical foundation for future research in rate-distortion optimiza- tion, advancing the field of video coding technology. The materials will be avail- able https: : / g thub. com/ /rr///rriiggieeee ccmm//ricggg//euurquuaaaa /Neurouuant..
errors, arriving at a unified PTQ calibration. Our experimental evalua-
INTRODUCTION
Implicit Neural Representations (INRs) (Sitzmann et al., 2020; Chen et al., 2021a) have recently introduced a new approach video coding. They focus on learning a mapping from coordinates, like frame indices, to pixel values, such as colors. This represents significant departure from the widely used variational autoencoder (VAE)-based frameworks (Lu et al., 2019; Li et al., 2021a; Lu et al., 2024), which rely on generalized models trained on large datasets to create compact representations for various input signals. Instead, INR-based video coding (INR-VC) encodes each video as a unique neural network through end-to-end training, removing the need for extensive datasets. By using specific, non-generalized network weights for each video, INR-VC VC provides a tailored video coding method that has shown promising results (Chen et al., 2023; Kwan et al., 2024a).
INR-VC typically focuses on two main objectives: 1) Representation, where a neural network models the target video with minimized distortion, and 2) Compression, where the network's weights are compressed to lower the bitrate. Many prominent methods adopt a consistent precision (quantization bitwidth) for all weights before lossless entropy coding, meaning the video bitrate depends solely on the number of learnable weights. Consequently, independent weight training is needed for each target bitrate, making the process very time-consuming. For example, encoding 1080p video with 600 frames at a specific bitrate can take up to 10 hours.
To address this inefficiency, we consider how bitrate is managed in pretrained INR- VC model. which is proportional to the sum of the bitwidth of each weight. Inspired by generalized codecs (Sullivan
"Corresoonding Author

---

Published as a conference paper at ICLR 2025
Param,
Train
GPU
GPU
GPU
Train
GPU
max
Fixed QP
Variable QP
Weights
Weights
Decode
Decode
Figure I: Left: Typical INR-VCs assume consistent bitwidth and require separate weight training with varying quantities for each target rate. Right: The proposed NeuroQuant achieves variable rate by modifying the corresponding QPs, significantly reducing training costs.
et al., 2012; Li et al., 2023) that adjust quantization parameters (QPs) (Wang & Kwong, 2008) to control bitrate, we pose the hypothesis: Can variable-rate INR-VC be achieved by modifying the QP of post-training weights, thus climinating the need for repeated model training for each target rate? In the context of weight quantization, this can be approached by: 1) allocating quantization bitwidth to match the target bitrate, and 2) calibrating QPs to preserve reconstruction fidelity.
However, directly adopting a consistent quantization bitwidth cannot support fine-grained rate con- trol, e.g., only seven options from INT2 to INT8 are available. Additionally, existing mixed- precision quantization methods (Nagel et al., 2021; Chen et al., 2021b), primarily designed for general-purpose neural networks, encounter two key problems when applied to non-generalized INR-VCs. First, mixed-preciste algorithms (Dong et al., 2019; 2020; Chen et al., 2021b) typically assume inter-layer independence with tolerable approximation errors. This assumption breaks down in non-generalized INR-VCs, where layers exhibit significant dependencies. Second, popular layer- wise calibration methods' (Nagel et al., 2020; Li al., 2021b) also rely on inter-layer independence and aims at generalizing the network, making them unsuitable for INR-VC VC. Therefore, dedicated quantization methodology tailored for variable-rate INR-VC VC- is necessary.
In this work, we explore, for the first time, the post- post-training quantization (PTQ) of weights in non- generalized -CC. Building on both empirical and theoretical insights, we propose NeuroQuant, a state-of-the-art PTQ approach for INR-VC that enables variable-rate coding without complex re- training. Our contributions tackle key challenges through the following research questions:
1. How to realize variable bitrate (Sec. 3.1): We redefine variable-rate coding mixed-precision quantization problem. By theoretically demonstrating that the assumption of inter-layer indepen- dence (Dong et al., 2020; Guan et al., 2024) does not apply to non non-generalliee models, we highlight the necessity of incorporating weight perturbation directionality and off-diagonal Hessian informa- tion for sensitivity assessment in quantizing INR-VC. Additionally, we introduce the Hessian-Vector product to simplify computations by eliminating the need for explicit Hessian calculations.
2. How to ensure reconstruction quality (Sec. 3.2): We enhance reconstruction quality by calibrat- ing the QPs on the corresponding video-specific weights. Through second-order analysis, we derive unified formula for MSE-oriented calibration across varying granularities. By considering sig- nificant cross-layer dependencies and the diverse distribution of weights, we conduct network-wise calibration and channel-wise guantization to minimize reconstruction loss.
3. How NeuroQuan performs (Sec.4) 4): We benchmark proposed NeuroQuant across various archi- tectures against existing quantization techniques, achieving state-of-the-an results. For variable-rate coding, NeuroQuant outperforms competitors while reducing encoding time by 80%. Moreover, NeuroQuant is able quantize weights down to INT2 without notable performance degradation.
4. How to advance VCR-VC (Sec. 3.3): We revisit INR-VC through the lens of variational infer- ence, proposing that the success of NeuroQuant stems from resolving the mismatch between the representation and compression. We also suggest that rate-distortion (R-D) optimization is also applicable to INR-VE VC and has the potential to achieve improved performance.
'To avoid ambiguity, we use the term calibration to describe the process of optimizing QPs, though some literature refers to this reconstruction. In this paper, reconstruction refers to the decoded videa from INR-VC system. And for simplicity, layer calibration also stands for block calibration.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
Basic Notations. We follow popular notations used in neural network. Vectors are denoted by low- ercase bold letters, while matrices (or tensors) are denoted by uppercase bold letters. For instance, W refers to the weight tensor, and w is its flattened version. The superseript of w(() indicates the layer index. For a convolutional or a fully-connected layer, we mark input and output vectors by æ and z. Given a feedforward neural network with layers, the forward process is expressed as
x (1+1) h)=(9)) h( (0),, 1 I n.
where h(.) denotes the activation function. For simplicity, we omit the additive bias, merging it into the activation. In the following, the notation || represents the Frobenius norm. Suppose æ is sampled from the dataset X, then the overall task loss is expressed as xx (e=xx(((xxx)))
INR-based Video Coding. INR-VC RC operates on the principle that a target video can be encoded into learned weights through end-to-end training. For each frame V, in an RGB video sequence V [v E I xx INR-VC assumes the existence of an implicit continuous mapping F [0, 1 Rdout in the real-world system such that V2 F o t. According to the Universal Approximation Theorem (Hanin, 2019; Park et al., 2021), the unknown F can be approximated by a neural network 'D of finite length LD. The estimated V. is then expressed as:
Vi D (t) = Uz h UL ...... o h o U1 E(t).
where D consists of cascaded upsampling layers U, and E(-) is an embedding of the timestamp t. Typically, index-based INR-VCS (Chen et al., 2021a) employ fixed Positional Encoding function or a learnable grid (Lee et al., 2023) as E.). while content-based INR-VCs (Chen et al., 2023; Zhao et al., 2023) utilize learnable encoder. The encoding of INR-VC involves training the learnable weights w and subsequently compressing w into a bitstream using quantization and entropy coding techniques. While existing INR-VC works primarily focus on minimizing distortion during the training stage, video coding is fundamentally a R-D trade-off.
Post- TesinTrain Quantization. PTQ offers a push-button solution to quantize pretrained models without weights training. It contrasts with Quantization-Awart Training (QAT), which involves both weight optimization and quantization during training. leading huge training cost. PTQ is generally a two-step process: 1) initializing QPs (e.g., steps) with allocated bitwidth and weight distribution statistics; 2) calibrating QPs reduce guantization-induee
loss. PTQ typically employs uniform affine transformation to map continuous E IR to fixed-point integers w. Traditional methods aim to minimize quantization error || w|l. However, an increasing number of explorations (Stock et al., 2020; Nagel et al., 2020: Hubara et al., 2021) suggest that this approach can yield sub-optimal results, as the parameter space error does not equivalently reflect task loss. To analyze quantization- induced loss degradation, AdaRound (Nagel et al., 2020) interprets quantization error as weight perturbation, i.e., ú u + Aw. The loss degradation can be approximated using Taylor series:
1 B[L(w + Aw, x) L(w, x)] R Awr g (w) + + AwT (Hw) Aw,
where ((w)) V and H = E represent expected gradient and the second- order Hessian matrix, respectively. For well-converged weights, gradients tend to be close O. AdaRound further assumes inter-layer independence, leading to a diagonal Hessian matrix opti- mization. BRECQ (Li et al., 2021b) extends AdaRound's layer-wise calibration block granularity based on inter-block independence. However, these methods can significantly degrade the perfor- mance of non-generalized INR-VCs, which exhibit significant dependencies among layers.
Quantization. Mixed-precision quantization facilitates fine-grained rate control in INR-VCs. with bit allocation being crucial due the varying levels of redundancy across layers and their different contributions to overall performance. However, determining optimal bitwidth assignments presents significant challenge because of the extensive search space. For a network with N layers and M candidate bitwidths per layer, exhaustive combinatorial searches exhibit expo- nential time complexity of O To address it, various strategies have been explored, including search-based based reinforcement learning Wang et al. (2019); Lou et al. (2019), neural architecture search (Wu et al., 2016), and Hessian based criteria (Dong et al., 2019; 2020). Despite these efforts, they often prove impractical for INR- VCs, as the search costs may surpass those of retraining a model. Furthermore, many existing criteria lack a robust theoretical basis for their optimality, rendering them less reliable in INR-VC systems.
3

---

Published as a conference paper at ICLR 2025
HNeRV
10
6.677
6015
000
a 10
0.24
10
02
43 0..
0.0
2
Layers
bny
(a) Layer-wise sensitivity Ω
(h) loss landscape
(c) 6-th landscpae
Figure 2: Examples of quantizing layers sequence. (a) Different layers exhibits varying sensitiv-
ities. (b) Lower S2 means flatter loss landscape. (c) Higher Ω is otherwise, and the loss landscape
shows pronounced directivity, indicating the necessity of considering the direction of Aw.
3 METHODOLOGY
We introduce the proposed NeuroQuant for lig--performance variable-rate INR-VC as follows:
Problem 1 (NeuroQuant). Given learned hideo-specifp weights, the objective of NeuroQuant is to achieve different R-D trade-offs by quantizing post-training weights with variable QPs. This can be formulated as rate-constrained optimization process:
arg min E(C(((w)) Q(e)) L(w,e)
(4)
s.t. Param(w
b() + " Param(el)) be R
(5)
where R represents the target bitrate, e denotes the embedding, Paraml-) indicates the number of parameters, and b denotes the bitwidth.
The rate-constrained term in Eq. is defined as muxeddprecisiog bit assignment problem, accounting for fine-grained rate control and varying layer sensitivity; 2) Sec. 3.2: The objective in Eq. interpreted as QP calibration problem, focusing calibration and quantization granularity of non-generalized INR-VC; 3) Sec. 3.3: We revisit the entire problem from the perspective of variational inference provide a broader theoretical grounding.
We decouple this problem into three sub-problems: 1)Sec. 3.1:
3.1 How TO REALIZE VARIABLE BITRATE
quantization is to allocate higher pre- cision (e.g., greater bitwidth) to sensitive layers while reducing precision insensitive ones. Sen- sitivity be intuitively understood through the flatness of the loss landscape (Li al., 2018), as illustrated in Fig. 2. A flatter landscape, indicating lower sensitivity, corresponds to smaller loss changes with weight perturbations, whereas sharper landscape indicates otherwise. Sensitivity es- sentially captures the curvature of the loss function, often described using second-ords -rder information,
Sensitivity Criterion. The concept of
particularly the Hessian matrix H (w). H (w) defines how perturbations in weights affect task loss. For instance, HAWQ (Dong et al., 2019) uses the top Hessian eigenvalue as a sensitivity criterion. while HAWQ-V2 (Dong et al., 2020) demonstrates that the trace offers a better measure. However, these criteria rely on two key assumptions: 1) Layer Independence: Layers are mutually inde- pendent, allowing (w) be treated as diagonal. 2) Isotropy: The loss function is directionally uniform under weight perturbations Aw, meaning only (w)) is considered, ignoring Aw.
H(w)
While these assumptions may hold for general general-purpose purpose networks, they break down in the context of gonngeneralized INR-VC, where significant inter-layer dependencies (Fig. 3(c)) and anisotropic behavior (Fig. 2(c)) exist. The following toy examples demonstrate why relying solely on diagonal information from H is suboptimal.
4

---

Published as a conference paper at ICLR 2025
Example 1 (Inter-Layer Dependencies). Consider three functions, F1 = -xx² ty 2 F2 and F3 = 4r² + 2y² + 5xy. Their corresponding Hessians are given as:
=4x²+y²
+2y2
H .
H H (F 8
0 1
H
8
1 5
(6)
2
All three functions share the same top eigenvalue (8), yer F2 and F33 are clearly more sensitive than Fi. Although F2 and F3 have the same trace (12), F3 exhibits greater sensitivity due to the presence of off-diagonal terms (i.e., 5xy).
This demonstrates that inter-layer dependencies are overlooked when relying solely on diagonal information (e.g., eigenvalues traces). Off-diagonal terms are essential to accurately capture sensitivity, highlighting the need to consider the full Hessian matrix. The story does not end there.
Example 2 (Weight Perturbation Directions). Assuming a perturbation [ax, Ayl applied H(Fs) from above, the increase in loss is approximately proportional to
Fa( ( + Ar, y Ay) Fa(x,1) v) R |Ar, Ayj H (A, Ay]" 8 + 4&y² 10ArAy. (7) Now, consider two cases; 1) Lower perturbation: [Ar, Ay] = [0.1, 0.1]; 2) Higher perturbation:
[Ax, Ay] [0.2, 0.2).. The increases in task loss are 0.22 and 0.08, respectively. Surprisingly, the higher perturbation results in a smaller task loss.
This counterintuitive behavior is also observed in practice, where quantizing layers with higher H sensitivity lower bitwidth does not necessarily lead to significant performance degradation. We argue that allocating higher bitwidth to layers primarily reduces | A.l However, this does not always guarantee a lower task loss, as L anisotropy under Au in INR-VC. The key insight that task loss also depends on the direction of Aw, not just its magnitude ||Aw|||-
In conclusion, the sensitivity criterion of INR-V VC must account for both the full Hessian matrix H and the direction of weight perturbations Aw. This leads the following theorem:
Theorem 1. Assuming the INR-VC weights are twice differentiable and have converged to a local minima such that the first and second order optimality conditions are satisfied (i.e., the gradients are zero and the Hessian is positive semi-defintte,, ). the optimal sensitivity criteria for mixed-ppecissee INR-VC VC is given by weighted Hessian information f2 Aw" H (w) Aw.
The criteria Ω2 formed by Hessian-Vecte product, can essentially be interpreted as a linear transfor- mation on (w)), accounting for Hiw) along the weight perturbation directions. Existing Hessian- based criteria can be viewed as a degraded version of the proposed Ω that neglects the off-diagonal terms. For instance, Eq. would degrade 8Ax² + 4Ay?, and thus, loss is independent of inter- variable dependencies and perturbation direction.
Approximating Hessian Vector Product. and store its quadratic complexity relative to the number of weights. Instead of forming ((w) explicitly, we focus on the sensitivity criterion & Aw" H()) Aw. Let's construct a function of the form G = gAw, where g is the gradient of with respect to w. The gradient of G can be expressed
The Hessian matrix challenging explicitly compute
G dgAw Ög Aw dAw OC Au + dAw = ((") Au + dAw
(8)
dw Dw
dw Dw
dw
Ow
In a converged model, g approaches Moreover, quantization error can be modeled as ran- dom vector, with its component sampled independently form a Uniform distribution: Aw ~ U--0.5, 0.5) (Ballé et al., 2017). Thus, the second term in Eq. can be ignored. This approxi- mation is also akin to straight-through estimator (STE) (Liu al., 2022), where die dw leads to a&w = 0. Consequently, we arrive at the final formulation for Ω: dh
Sw Sw
Ω = E[Aw' "V... V..g.. where g = gaw = E(V i. CAw), g R!.
(9)
In Eq. 9, Aw is treated as perturbation around w, allowing us to compute g centered at w. For each potential bitwidth configuration, we only need to compute Aw and the gradient of G in linear time. Notably, different from using L directly, such criteria-based methods do not require super- vised labels or forward inference over the entire full datasets for each potential bitwidth candidate, enabling efficient mixed-precissin search using techniques like integer programming, genetic algo- rithms (Guo et al., 2020), or iterative approaches. So far, we have realized bit allocation for a target bitrate. The next step involves calibrating QPs to minimize the reconstruction distortion.
5

---

Published as a conference paper at ICLR 2025
1.5
Exemplified Hessian
0.31
1.0
3.2
2:14
6.11
1.:02
Calibration Granularity
frequeney
5
Min/Max Value
0.5
0.30
2.44
0.05
0.05
0.21
4
0.0
0.06
-0.5
0.04
0.00
0.30
2
0..04
-1.0
0.02
-0.4
0.0 value
0.4
20 40 60 80
Channel's index
Layer-wise
Block-wise
Network-wise
(a) Layer statistics
(b) Channel statistics
(c) Hessian statistics
Figure 3: Statistic of the weight distribution among (a) layers and (b) channels. Other layers/chan-
nels exhibit similar distributions. (c) "on-generalized representation shows obvious layer/block dependencies, implying that layer/block calibration not suitable for INR-VC. Exemplified statistic information is based on HNeRV (1M) in Beauty sequence.
3.2 How TO ENSURE RECONSTRUCTION QUALITY
We follow the principle of PTQ to calibrate QPs (including steps and rounding variables in Eq. 16) without optimizing the underlying weights. PTQ allows to preserve reconstruction quality by only calibrating QPs, instead of engaging in complex weight training as QAT.
Unified Calibration Objective. We begin by investigating the unified calibration objective. The quantization-indee inducedd task loss degradation of well-converged model can be approximated using the second-order Taylor expansion:
Ei((w + Aw, 2) - L(w, x)] = _''" H(w) Aw.
(10)
From this estimation, we aim to calibrate QPs by minimizing the proxy loss, defined as: min Aw". ((w) Aw. Denote the neural network output as z("), Using the chain rule, we can compute the Hessian matrix H follows:
S à
m Oc 0z (n)
172
OC
0
(n) z
m
3z(m))
(n)
OC
0z (n)
(z)00 Ow; oz Oz(")) Ow, (11)
ow,dw; ow
Oz (n)
Ow.
Oz (n)
Ow,Ow,
k.1=1
Since the post-training model converges local minimum, we assume the Hessian positive- semi-definite (PSD). With V 1 L close to O, the first term in Eq. 11 is negligible (Martens, 2010), yielding Gaussian-Newton form G(w):
H (()) (2()) H Jz(n),
(12)
where Jz(n) is the Jacobian matrix the network output (n) with respect to the weights w.
Considering C given by commonly used mean squared error (MSE), have:
m
H (= (m)) = 2I. s.l. L
Σ
2 (n)
-x
(13)
By substituting the Eq. 12 and Eq. 13 into the Eq. 10, we get the following minimization objective:
min Awo (w) Aw ss min
Aw) ss min E]]|A=(")11
(14)
Remark 1. This results in unified form of MSE-oriented calibration. Other distortion metrics may have analogous form, but this is not guaranteed, as not all metrics necessarily correspond to analytical Hessian form. This can be addressed through approximated Hessian (Appendix C).

---

Published as a conference paper at ICLR 2025
Calibration Granularity. Based the above objetive, we further investigate the optimal granular- ity for calibration. Following Li et al. (2021b); Wei et al. (2022), we can define:
1. Layer-wise Calibration: This assumes layers are mutually independent and calibrates each layers individually, resulting in a layer-diagonal Hessian matrix (blue diagonal in Fig. 3(c)), where A in Eq. 14 reflects to the output error of individual layers.
2. Block-wise Calibration: A block is defined as a collection of several layers (e.g., residual blocks). This calibration considers intra-block dependencies while assuming inter- indepen- dence. Calibration is performed block by block, leading to a block-diagonal Hessian matrix (orange diagonal in Fig. 3())), with A= representing the output error of the block whole.
3. Network-wise Calibration: This granularity calibrates entire quantized network by considering the global Hessian (full matrix in Fig. 3(c)), where A= reflects the entire network's output error.
In generalized networks, layer/block-wise calibration is widely accepted, as dependencies are pri- marily found within layers/blocks, allowing inter-layer/block dependencies be neglected (Li et al., 2021b). However, their network -wise calibration may lead to poor performance due to high general- ization error. In the context of INR-VC. the situation differs: inter-layer/block dependencies cannot be ignored (Fig. 3(c)), and INR-VC typically prioritizes non-generalized representation over gener- alization. Therefore, we advocate for network- wise calibration as our preferred approach, as better captures the dependencies across the network layers.
Quantization Granularity. Next, we consider the granularity for quantization steps. Weight distri- butions vary significantly across different layers of given INR-VC weights (Fig. 3(a)), and even among channels within a specific layer (Fig. 3(b)). This variability suggests that weights should be modeled on a channel-wisc basis, where all weights in a channel share the same quantization step.
Calibration. Once the granularity of both calibration and quantization are determined, we next calibrate QPs to minimize distortion. While solving Eq. 14 circumvents the complexity issues asso- ciated with the Hessian, remains an discrete optimization problem. Inspired by Nagel et al. (2020), reformulate it into continuous optimization framework using soft weight variables:
arg min [ w) F(xr, n1)l +A Σ 12h(w,) 11.
(15)
Distortion term LD [
Regularication Lieg
s.t. の clip
w (1==)
+ hh(v,, -2 2 -1-
(16)
where Vi, represents the continuous variable to optimize, and h (VV,,) (..) any differentiable function constrained between 0 and 1, i.e., h(V..) E [0, 1]. Liey acts as differentiable regularizer, guiding h(Vis) to converge towards either or 1, ensuring at convergence ((V..)) E {0, 1). We also anneal 3 in Lreg to facilitate stable convergence. This approach yields an intriguing observation: by minimizing the discrepancy between high-precision teacher model and an initialized student model, we can effectively calibrate the quantized network. However, it's crucial to emphasize that we are different from knowledge distillation (Polino et al., 2018) that requires similar computational and data resources naive training, making it impractical for our variable-rate coding.
3.3 HOw TO ADVANCE INR-VE VC
Our NeuroQuant adjusts QP to manage R-D trade-off where higher rates lead to lower distortion and vice versa. This trade-off can be formally interpreted through variational inference (Kingma, 2013; Ballé et al., 2017). aiming to approximate the true posterior Panja (w)aa æ) with a variational density
q(r0) æx) by minimizing the Kullback-Leible (KI.) divergence over the data distribution Pa:
Feg D K 1g Paja]]
Eaapp Ean- log q x))
log Pae (2: (5)
log Pur (w) + log log))()))) (17)
=0
Distortion Lo
Rate Cn
cons
This leads to the R-D trade-off for INR-VC VC (Appendix B):
L Ln + ALD log Pm ())
20 æ 2l1? s.t. Paju (m/10) (T) = NN(x) o²),
(18)
In coding, w is replaced by the discrete symbol w. which can be losslessly compressed using entropy coding techniques, such arithmetic coding (Rissanen & Langdon, 1981).
7

---

Published as a conference paper at ICLR 2025
Remark (NeuroQuant vs. ICR- Popular INR-VCs (Chen et al., 2021a; Li et al., 2022b; He et al., 2023: Zhao et al., 2023; 2024) focus on min log Pa jo (xlw) without considering the impact of weight quantization, creating mismatch between representation and compression objectives. ie., p():)w) and p(z)(t), which degrades performance after quantization. In contrast, the proposed NeuroQuant directly optimizes log Pw to (a) ()). bridging this mismatch and yielding superior results.
While QAT (Ladune et al., 2023; Kim et al., 2024) can also optimize the same objective, Neu- roQuant's strength lies in transforming R-D optimization into a post-training process, particularly advantageous for efficient variable-rate video coding, thereby reducing encoding costs. Despite NeuroQuant enables a more flexible R-D trade-off than existing approaches, it is not yet a fully joint optimization framework. Recent works (Gomes et al., 2023; Zhang al., 2024) have begun rec- ognize the necessity of joint R-D optimization in INR-VC, but they lack principled explanations and typically underperform compared generalized codecs. For example, they often assume Gaussian prior or i.i.d. context model (Minnen et al., 2018; He et al., 2021), which is not entirely correct for ICR-VC. Future advancements will require the development of customized context models to better capture weight characteristics, enabling true joint R-D optimization for INR-VC systems.
4 EXPERIMENTS
In this section, conduct thorough experiments to verify the effectiveness of NeuroQuant, Detailed implementation and additional experiments are available in Appendix E.
INR-VC Baselines. We select three representative INR-VCs as baselines to evaluate various quanti- zation methods. NeRV (Li et al., 2022b) is a pioncering model that first maps frame indices to video frames. HNeRV (Chen et al., 2023) first introduces encoder to generate learnable embeddings instead of the positional encoding used in NeRV. HiNeRV (Kwan et al., 2024a) further optimizes the network architecture, achieving start-of-the-an performance.
Quantization Benchmarks. We first introduce naive PTQ in HNeRV, a widely adopted approach across various INR- We also include four leading task-oriented PTQs: AdaRound (Nagel al., 2020), BRECQ (Li et al., 2021b), QDrop (Wei et al., 2022), as well as RDO-PTQ (Shi et al., 2023), which is specifically designed for R-D optimization. Two QATs tailored for INR-VCs derived from FFNeRV (Lee et al., 2023) and HiNeRV (Kwan et al., 2024a) are also evaluated.
4.1 QUANTIZATION
We summarize the quantization performance in Table 1. Starting with direct quantization in HN- eRV, the results confirm Remark 2: optimizing reconstructive representation without considering weight quantization (through training awareness or post-calibration) significantly degrades per- formance. When compared with leading task-oriented ATQs-AdaRound, BRECQ, QDrop, and RDOPTQ-NeuroQuue consistently demonstrates superior performance, with the advantage grow- ing as bitwidth decreases. The results support our analysis: non- non-generalized INR-V VC exhibits sig- nificant inter-layer dependencies. making network-wise calibration necessary. Our NeuroQuant also outperforms QAT's designed for INR-VC (i.e., FFNeRV and HiNeRV) across all listed baselines, particularly at lower bitwidths, achieving gains of more than 3dB. Its adaptability to varying preci- sion highlights superior bitrate flexibility, indicating its potential for variable-rate video coding.
4.2 VARIABLE-RATE CODING PERFORMANCE
Figure 4 depicts the R-D curves for the evaluated methods. Compared to NeRV and HNeRV equipped with direct 8-bit quantization, NeuroQuant demonstrates significant compression effi- ciency gains of 27.8% and 25.5%, respectively. This improvement primarily stems from their lack of optimization for the compression objective. In contrast, HiNeRV achieves superior compression performance by incorporating QAT, as discussed in Sec. 3.3. Despite this, NeuroQuant also brings another 4.8% gains by replacing the built-in QAT. A key advantage of NeuroQuant is its ability to support variable-rate coding without training separate weights for each target bitrate, offering both flexibility and efficiency in practice. Moreover, NeuroQuant outperforms the network coding tool DeepCABAC (Wiedemann et al., 2020), highlighting the benefits of task-oriented QPs. Incorporat- ing its advanced entropy coding with NeurQuant an exciting avenue for future research.
8

---

Published as a conference paper at ICLR 2025
Table 1: Reconstruction quality comparison of different quantization methods (vertical) across var- jous INR-VCs with different model sizes (horizontal) in terms of PSNR on UVG (Mercat et al.,
2020). Bold values indicates the best results. denotes QAT strategies and represents mixed pre-
cision. All implementations are based on open-source codes.
Methods
W-bits
NeRv
NeRV
HNeRV
HNeRV
HiNeRV
Avg
Full Prec. (dB)
32
31.39
32.30
32.49
33,80
35.09
33.01
Param.
3.1M
6.2M
3.0M
6.2M
3.1M
HNcRV (Chen al., 2023)
30.68
31.56
32.05
33.29
32.10
31.94
FFNeRV (Lee al., 2023)f
31.10
32.02
32.15
33.34
34.03
32.53
HiNeRV (Kwan et al., 2024a)f
31.20
32.09
32.25
33.48
34.54
32.71
AdaRound (Nagel et al., 2020)
31.03
31.96
32.10
33.26
33.92
32.45
BRECQ (Li et al., 2021b)
31.11
32.05
32.18
33,42
34.10
32.57
Qrrop (Wei al., 2022)
31.15
32.10
32.20
33.44
34.27
32.63
RDOPTQ (Shi et al., 2023)
31.15
32.06
32.16
33.39
34.23
32.60
NeuroQuant (Ours)
31.31
32.22
32.38
33.61
34.67
32.84
HNeRV (Chen et al., 2023)
27.02
27.86
28.14
28.60
24.30
27.18
FFNeRV (Lee et al., 2023yf
30.14
30.90
31.11
32.13
32.37
31.33
HiNeRV (Kwan et al., 2024a)f
30.37
31.32
31.46
32.67
32.95
31.75
AdaRound (Nagel al., 2020)
30.12
30.65
31.02
31.98
32.10
31.17
BRECQ (Li et al., 2021b)
30.22
30.93
31.26
32.24
32.56
31.44
Qrrop (Wei et al., 2022)
30.28
31.05
31.33
32.45
32.68
31.56
RDOPTQ (Shi et al., 2023)
30.25
30.96
31.24
32.25
32.60
31.46
NeuroQuant (Ours)
30.85
31.77
31.64
32.81
33.33
32.08
HNeRV (Chen et al., 2023)
14.81
15.50
13.32
13.06
13.30
14.00
FFNeRV (Lee al., 2023)f
22.52
22.89
23.84
24.14
21.25
22.93
HiNeRV (Kwan et al., 2024a)
24.08
25.30
25.11
26.51
23.89
24.98
AdaRound (Nagel et al., 2020)
22.51
23.44
23.70
24.56
22.10
23.26
BRECQ (Li et al., 2021by
24.05
25.17
25.40
26.32
25.87
25.36
QDrop (Wei al., 2
25.32
26.14
25.94
26.85
26.60
26.17
RDOPTQ (Shi al., 2023)*
24.33
25.75
25.57
26.50
26.14
25.66
NcuroQuant (Ours)*
27.39
28.48
28.02
29.05
28.92
28.37
4.3 DIVING INTO NEUROQUANT
In this subsection, we perform series of evalua- tions to gain deeper insights into how our Neuro- Quant functions.
X265
DCVC
NeRV
Encoding Complexity. Table presents the en- coding time evaluation for 960 x 1920 videos
35
25.5%
NeRV-VR
34
with approximate 3M parameters. From NeRV HiNeRV, compression performance is improved at the cost increased encoding complexity. Without support for variable bitrates, generating a new bitrate point typically requires retraining the model, leading an encoding time that generally unacceptable (e.g., exceeding 22 hours for HiNeRV). In this context, NeuroQuant pro- vides practical solution by leveraging pre- trained model, enabling adaptation to variable bi- trates while ensuring efficient encoding, achiev- ing speedups of up to 7.9 times.
HNeRV
-278%
HNeRVAvR
HiNeRViR
0.000
0.025 0.050
0.075
0.100
0.125
0.150
0.175
Bit Rate (bpp)
Figure Compression efficiency comparison. Variable-rate models are labeled with VVR suffix using the solid line. * INR- VC using QAT.
Table Encoding time required support bitrate. Note that pretrained model is shared for all bitrates in range.
Furthermore, we select the Jockey sequence from UVG as representative example to have an anal- ysis of variable-rate coding in Fig 5. The left sub- plot presents the available points of mixed preci- sion versus unified precision. The middle sub-
Baselines
Naive
NeuroQuant
Pretrain
NeRV HNeRV HiNeRV
1.8
0.4 (x4.5)
1.8 h
4.7 h
1,0 (x4.7)
4.7 h
22.2 h
2,8 (x7.9)
18.9

---

Published as a conference paper at ICLR 2025
Rufe-Control and Mixed Precision
Unified R-D Curves ncorss Bits
Variable Rate Range
32.3
32.23
35
Naive abit
35.0
Naive
NcRV-PSNR (dB)
32.2
34
34.5
NeuO-L
NeuQ.M
24.00
$2.05
NcuQ 4.20t
NeuQ-XL
32.0
21.98
33
33.0
Neuq-2 2. Sbit
34.0
4bit
Sbit
33.5
33.50
31.8
31.82
Mixed Precision
Unifiid Precision
33.25
32.0
33.00
10
3.2
3.4
3.6
3.8
5
10
20
25
10
15
20
25
32.8
32.75
32.80
Malve
Naive
HNeRY-PSNR (dB)
36
NeuQ-S.8bil abit
36.5
NeuQ-M
NeuQ-4.80 9bil
Nauq.L
32.6
32.58
2.553
35
Neu 4. Sbit
Neug-XL
35.2
34
35.5
32.4
33
33.0
35.0
35.0
Sbit
32.27
Unified Precision
32
22.5
34.9
32.2
Mixed Preci sion
34.0
HAWQ-4
22.0
30
Model Size (MB)
Model Size (MB)
Model Size (MB)
Figure 5: Quantitative comparison of NeRV (up) and HNeRV (bottom).
plot depicts the RD curves for different quantization bitwidths, while the right subplot displays the variable-rate range for models of varying sizes. More results can be found in Appendix E.3. We have the following conclusions:
Rate Control and Mixed Precision. Mixed precision supports finer-grained rate control compared unified precision, making NeuroQuant more flexible for various bitrate. Additionally, it enables better bit allocation, resulting in improved coding efficiency.
Unified R-D Curves across Bits. The coarseness of quantization affects both rate and distortion, leading to a R-D trade-off. Within certain range, different bitwidths exhibit unified R-D charac- teristics, providing a foundation for variable-rate coding through adjusting quantization parameters. While we demonstrate state-of-the-ar performance, the degradation is still noticeable below INT3. As result, we do not recommend INT3 and below for variable-rate scenarios.
Variable-Rate Range. While NeuroQuant significantly reduces the need for exhaustive individual weights training, achieving infinite rate range through quantization alone is impossible due to the inherent lower bound of bitwidth. We believe this limitation can be addressed by introducing prun- ing, which can be interpreted as a special case NeuroQuant with bitwridth 0. This approach opens up possibilities for further exploration based the foundations laid by this work.
CONCLUSIONS
Conclusion. In this work, we introduce NeuroQuant, novel approach for variable-rate INR- VC that adjusts the QPs of post-training weights to control bitrate efficiently. Our key insight is that non-generalized INR-VC exhibits distinct characteristics for quantization. Through empirical and theoretical analysis, we establish the state-of-the-art weight quantization for INR-VC. NeuroQuant significantly reduces encoding complexity while maintaining leading compression performance, providing an effective solution for variable-rate video coding in neural representation.
Limitations and Future Work. Despite the success of NeuroQuant, there are some limitations. The variable-rate range is inherently constrained by the available bitwidths. A promising solution integrating weight pruning techniques to endow the model weights with greater variability. Addi- tionally, as discussed in Sec. 3.3, NeuroQuant enables variable R-D trade-off's than existing methods, but still fall short of achieving true joint R-D optimization. Future work will explore customized probabilistic modeling for INR-VC, aiming to enable fully joint R-D optimization process.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by National Key Research and Development Project of China (Grant No. 2022 YFF090240)) Natural Science Foundation of Jiangsu Province (Grant No. BK20241226) and Natural Science Foundation of China (Grant No. 62401251, 62431011), I). The authors would like express their sincere gratitude to the Interdisciplinary Research Center for Future Intelligent Chips (Chip-X) and Yachen Foundation for their invaluable support.
REFERENCES
Johannes Ballé, Valero Laparra, and Eero P Simoncelli. End-to-end optimized image compression. In 5th International Conference on Learning Representations, ICLR 2017, 2017.
G Bjontegaard. Calculation of average psnr differences between rd-curves. ITU-T SG/6 Q, 6, 2001.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neu- ral representations for videos. Advances in Neural Information Processing Systems, 34:21557-
21568, 2021a.
Hao Chen, Matthew Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. Hnerv: A hybrid neural representation for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 10270-10279, 2023.
Weihan Chen, Peisong Wang, and Jian Cheng. Towards mixed-precision quantization of neural networks via constrained optimization. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 5350-5359, 2021b.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 293-302, 2019.
Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware trach-weighted quantization of neural networks. Advances in neural information processing systems, 13:18518-1888 2020.
Zhihao Duan, Ming Lu, Justin Yang, Jiangpeng He, Zhan Ma, and Fengqing Zhu. Towards backward-com compatible continual learning of image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 25564-25573, 2024.
Carlos Gomes, Roberto Azevedo, and Christopher Schroers. Video compression with entropy- constrained neural representations. In Proceedings of the IEEF/CVF Conference Computer Vision and Puttern Recognition, pp. 18497-18506, 2023.
Ziyi Guan, Hantao Huang, Yupeng Su, Hong Huang, Ngai Wong, and Hao Yu. Aptq: Attention- aware post-training mixed-precision quantization for large language models. arXiv preprint arXiv:2402. 14866, 2024.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pp. 544-560. Springer, 2020.
Boris Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 711))999, 2019.
Bo He, Xitong Yang, Hanyu Wang, Zuxuan Wu, Hao Chen, Shuaiyi Huang, Yixuan Ren, Ser- Nam Lim, and Abhinav Shrivastava. Towards scalable neural representation for diverse videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 6132-6142, 2023.
Dailan He, Yaoyan Zheng, Baocheng Sun, Yan Wang, and Hongwei Qin. Checkerboard context model for efficient learned image compression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14771-14780, 2021.
11

---

Published as a conference paper at ICLR 2025
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. International Conference on Machine Learning, pp- 4466-4475 PMLR, 2021.
Hyunjik Kim, Matthias Bauer, Lucas Theis, Jonathan Richard Schwarz, and Emilien Dupont. C3: High-performance and low-complexity neural compression from single image or video. In Pro- ceedings of the IEEF/CVE Conference on Computer Vision and Puttern Recognition, pp. 9347- 9358, 2024.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990 2014.
Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Gower, and David Bull. Hinerv: Video compres- sion with hierarchical encoding-based neural representation. Advances in Neural Information Processing Systems, 36, 2024a.
Ho Man Kwan, Fan Zhang, Andrew Gower, and David Bull. Immersive video compression using implicit neural representations. arXiv preprint arXiv:2402.01596 2024b.
Théo Ladune, Pierrick Philippe, Félix Henry, Gordon Clare, and Thomas Leguay. Cool-chic: Coordinate-based low complexity hierarchical image codec. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13515-13522, 2023.
Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-50. Springer, 2002.
Joo Chan Lee, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Ffnerv: Flow-guided frame-wise neural representations for videos. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 7859-7870, 2023.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land-
scape of neural nets. Advances in neural information processing systems, 31, 2018.
Jiahao Li, Bin Li. and Yan Lu. Deep contextual video compression. Advances in Neural Information Processing Systems, 34:181 1111111148115,, 2021a.
Jiahao Li, Bin Li, and Yan Lu. Hybrid spatial-temporal entropy modelling for neural video compres- sion. Proceedings of the 30th ACM International Conference on Multimedia, pp. 1503-1511, 2022a.
Jiahao Li, Bin Li, and Yan Lu. Neural video compression with diverse contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22616-22622 2023.
Jiahao Li, Bin Li, and Yan Lu. Neural video compression with feature modulation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 26099-26100 2024.
Yiming Li, Zizheng Liu, Zhenzhong Chen, and Shan Liu. Rate control for versatile video coding. In 2020 IEEE International Conference Image Processing (ICIP), pp. 1176-1180. IEEE, 2020.
Yuhang Li, Ruihao Gong. Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2 102. 022605466 2021b.
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expe- dite neural video representation with disentangled spatial-temporal context. In European Confer- ence on Computer Vision, pp. 267-284, Springer, 2022b.
Jianping Lin, Dong Liu, Jie Liang, Houqiang Li. and Feng Wu. A deeply modulated scheme for variable-rate video compression. In 2021 IEEE International Conference on Image Processing (ICIP), pp. 3722-3726. IEBE, 2021.
12

---

Published as a conference paper at ICLR 2025
Yang Liu, Zheng Guo Li, and Yeng Chai Soh. Rate control of h. 264/avc scalable extension. 1EEE Transactions on Circuits and Systems for Video Technology, 18(1):116-12) 2008.
Zechun Liu, Kwang- Ting Cheng, Dong Huang, Eric P Xing, and Zhiqiang Shen. Nonuniform-to uniform quantization: Towards accurate quantization via generalized straight- estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 4942-4952, 2022.
Qian Lou, Feng Guo, Lantao Liu, Minje Kim, and Lei Jiang. Autoq: Automated kernel-wise neural network quantization. arXiv preprint arXiv:1902.000 2019.
Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao. Dve: An end-to-end deep video compression framework. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11006-11015, 2019.
Ming Lu, Zhihao Duan, Fengqing Zhu, and Zhan Ma. Deep hierarchical video compression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 8859 8867, 2024.
James Martens. Deep learning via hessian-free optimization. In Proceedings of the 27th Interna- tional Conference on International Conference on Machine Learning, pp. 735-742, 2010.
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
David Minnen, Johannes Ballé, and George Toderici. Joint autoregressive and hierarchical priors for learned image compression. Advances in neural information processing systems, 31, 2018.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference Machine Learning, pp. 7197-7206. PMLR, 2020.
Markus Nagel, Marios Fournarakis, Rana Ali Amjad, Yelysei Bondarenko, Mart Van Baalen, and Tijmen Blankevoort. A white paper on neural network quantization. arXiv preprint arXiv:2106.08200 08295, 2021.
Sejun Park, Chulhee Yun, Jacho Lee, and Jinwoo Shin. Minimum width for universal approximation. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. In International Conference on Learning Representattons, 2018
Mohammad Saeed Rad, Behzad Bozorgtabar, Urs-Vikuar Marti, Max Basler, Hazim Kemal Ekenel, and Phillipe Thiran. Srobb: Targeted perceptual loss for single image super-resolution. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2710-2719, 2019.
Oren Rippel, Alexander Anderson, Kedar Tatwawadi, Sanjay Nair, Craig Lytle, and Lubomir Bourdev. Elf-ve: Efficient learned flexible-rate video coding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14479-14488, 2021.
Jorma Rissanen and Glen Langdon. Universal modeling and coding. IEEE Transuctions Infor- mation Theory, 27(1):12-23, 1981.
Xihua Sheng, Jiahao Li, Bin Li, Li, Dong Liu, and Yan Lu. Temporal context mining for learned video compression. IEEE Transactions on Multimedia, 25:7311-7322 2022.
Junqi Shi, Ming Lu, and Zhan Ma. Rate-distortion optimized post-training quantization for learned image compression. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
13

---

Published as a conference paper at ICLR 2025
Pierre Stock, Armand Joulin, Rémi Gribonval, Benjamin Graham, and Hervé Jégou. And the bit goes down: Revisiting the quantization of neural networks. In ICLR 2020- 2220-highth International Conference Learning Representations, pp. 1-11, 2020.
Pierre Stock, Angela Fan, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. In ICLR 2021- International Conference Learning Representations, 2021.
Gary Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. Overview of the high efficiency video coding (hevc) standard. IEEE Transactions on circuits and systems for video technology, 22(12): 1499-1999990 1668, 2012.
Hanli Wang and Sam Kwong. Rate-distortion optimization of rate control for h. 264 with adaptive initial quantization parameter determination. IEEE Transactions on Circuits and Systems for Video Technology, 18(1):140-144 2008.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Ran-
domly dropping quantization for extremely low-bit post-training quantization. arXiv preprint arXivv2203.05740, 2022.
Simon Wiedemann, Heiner Kirchhoffer, Stefan Matlage, Paul Haase, Arturo Marban, Talmaj Marinč, David Neumann, Tung Nguyen, Heiko Schwarz, Thomas Wiegand, et al. Deepcabac: A universal compression algorithm for deep neural networks. IEEE Journal of Selected Topics in Signal Processing, 14(4):700-714, 2020.
Thomas Wiegand, Gary Sullivan, Gisle Bjontegaard, and Ajay Luthra. Overview of the h. 264/ave video coding standard. IEEE Transactions on circuits and systems for video technology, 13(7): 560-576, 2003.
Jiaxiang Wu, Cong Leng, Yuhang Wang. Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4820 4828, 2016.
Xinjie Zhang, Ren Yang. Dailan He, Xingtong Ge, Tongda Xu, Yan Wang, Hongwei Qin, and Jun Zhang. Boosting neural representations for videos with conditional decoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2556-2566, 2024.
Yunfan Zhang, Ties Van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression, arXiv preprint arXiv:21 2021.
Qi Zhao, M Salman Asif, and Zhan Ma. Dnerv: Modeling inherent dynamics via difference neural representation for videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2031-2040, 2023.
Qi Zhao, M Salman Asif, and Zhan Ma. Pnerv: Enhancing spatial consistency via pyramidal neural representation for videos. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19103-19112, 2024.
A VARIABLE-RATE BACKGROUND
In real-world communication scenarios, practical video codecs pursue variable-rate coding to adapt to complex network environments while reducing storage and computational overhead.
In traditional video codecs (Wiegand et al., 2003; Sullivan al., 2012), variable-rate coding is typically achieved by adjusting quantization parameters (QPs) (Liu et al., 2008; Li et al., 2020). For leamned generalized video codecs, current variable-rate methods can be categorized into two main approaches: feature modulation and multi moltiigranullaityy quantization. Feature modulation focuses on modifying encoding and decoding features achieve representations with varying entropy. For
14

---

Published as a conference paper at ICLR 2025
instance, Lin al. (2021) proposed scaling network that modulates the internal feature maps of motion and residual encoders. Similarly, Rippel et al. (2021) converted discrete rate levels into one- hot vectors, which were then fed into various codec sub-modules. Duan al. (2024) introduced an adaptive conditional convolution method that applies affine transformations to features based on input Lagrangian multipliers (A). In contrast, multi-granularity quantization aims to control bitrate by adjusting the quantization levels of feature maps (also referred to as latents). For example, Li al. (2022a) proposed a coarse-to-fine quantization strategy using three learnable quantization granularity parameters: global levels, channel-wise levels, and apatial-channel-ao
levels.
However, INR-based video codecs (INR-VCs) take fundamentally different approach, converting video signal compression into model weight compression. This diverges from the conventional focus on extracting compact feature representations, making tailored variable-rate coding strategies for INR-VCs essential. While preliminary studies have hegun exploring variable-rate coding in INR-V VCs, their performance remains limited (detailed in Sec. E.2).
In this paper, we reduce this gap by framing variable-rate coding for INR-VCs as mixed-precision bit allocation problem. Our approach achieves variable-rate coding through QP adjustments and demonstrates state-of-the-ard performance.
B VARIATIONAL INFERENCE PERSPECTIVE
In INR- VC. the variational inference framework interprets the rate-distortion (R-D) trade-off as an optimization problem, aiming to approximate the true posterior distribution Paiaa (w> (w)|z) with variational density q( [ x) by minimizing the expected Kullback-Leibler (KL) divergence over the
data distribution Pv. Starting with the KL divergence, we have:
log q(20 [x)
Ex-p DKL(q [eojal Ex-p, lEowg
(19)
Pwo a (t x))
Ex Emion [log g 2 (z) log Papl (w]=)]]
(20)
Based on the Bayes' theorem, the posterior Peilan (10 [)) can be expressed as:
P P ar) Parlii (x) WP (20)
(21)
Pan (x:
Substituting this expression into Eq. 20, we get:
Euz-pp Drt.gllpata Empoa Enngg (log (()(ar))) log Pæt t (ar ()) log Pan (t))
: log Pm (=:)
(22)
Here, we arrive at a form consistent with 17. Let's examine each of terms individually.
First Term. For unknown mapping from timestamps t to video data F(w, f), the inference refers to computing the inverse transformation from input æ (Ballé et al., 2017). Using soft variable similar Eq. 16, have:
( xx) = II [[(ii:w: w. + ) with w = F- xr
(23)
where U/ denotes uniform distribution centered around w; Then:
E Sünngloge q [z] = Eaang log 2 ]
= 0.
(24)
Second Term. For the second term, suppose Pa (m/) (i)
follows a normal distribution:
Pa vo (x) ()) V(xl,, o²), ã F(w, t). Maximizing log Pata (æ) (T) is equivalent to minimizing the squared error term:
(25)
max log Pa to (x) ()) min log.l/////. (xx
(26)
1
(- (
min log
cxp
202 || - -112)
(27)
1
-mn
(28)
2g
15

---

Published as a conference paper at ICLR 2025
Third Term. The third term reflects the cost of encoding the latent variables w, representing the model complexity.
Forth Term. For specific video data, the true distribution p(æ) is constant.
In conclusion, we obtain the objective function as derived in Eq. 17:
1 20
L = LR + D = log (8) +
- 211²
(29)
By matching the variational density with the INR-based video coding framework, we observe that minimizing the KI. divergence corresponds optimizing weights for the vate-distortion perfor- mance.
C APPROXIMATING HESSIAN
While MSE is the most commonly used loss function in NR- it is impractical to assume that all future scenarios encountered by NeuroQuant will use MSE as their objective. To address this limitation, we consider an alternative approach based on the variation inference discussed earlier. The loss function of AN-VVC can be framed as the likelihood estimation for p(z](0), allowing us to define the Hessian as:
2 Ex-p(x)
log mn]]
(30)
The negative expected Hessian of the log- likelihooddd function is equivalent to the Fisher information matrix (FIM) (LeCun et al., 2002). We define the FIM, denoted by F, as follows:
F Empp(()))
2 log p(xp(((()))
(31)
V2 (
[V log w) Va [x))(
(32)
p i
The first term the right side of Eq. 32 is zero, shown in Chen et al. (2021b):
E px w))
((p((()
(33)
p(x
/ p())ū)
V
(34)
/
Vap( wd
(35)
V / ((
(36)
0.
(37)
Therefore,
F Ex=p(x))))
(ogp(()))) (og log log
(38)
(m=
(39)
In Eq. 11, the first term is ignored, and we approximate H using Gauss-Newton matrix form G. Similarly, in the context of likelihood estimation, this can be represented as:
G Ex [V
log ((a((()()) Va log [7
(40)
To summarize, we have the following relationships:
H V2 L E
[ logp(</<))
(41)
G E Ex)p(())
V V
(42)
F Exmpp
v2, i logr(
(43)
V logg(p log
(44)
16

---

Published as a conference paper at ICLR 2025
In INR-VC, the observed data is consistent with the true data. Hence, when the target data distribu- tion p(æ) equals to the fitted distribution p(a) (i), we derive:
H = F
(45)
This approximation, constrained by network capacity (e.g., bitrate) without data distribution, reveals a unique advantage compared to generalized codecs. Therefore, we can get an equivalent objective under the FIM form, though not exactly the same as the Hessian form
( (z)
Oc 2zn
min E
A= diag diag
Az
(46)
For some loss functions, such perceptual loss (Rad et al., 2019), which lack analytical Hessian form, the FIM can serve as a suitable subsitute.
D NEUROQUANT ALGORITHM
In our approach, represents the quantization steps. Once each layer assigned a bitwidth b4 the initial value of is computed channel-wisely using
max(w min (x
(47)
2
where witk is the weights in the k-th channel of the 1-th layer. During the calibration process, is further optimized to minimize task loss.
Algorithm Bit Allocation for Target Bitrate (Mixed Precision, Sec. 3.1)
Input: Pretrained weights W (FP32); Initial bitwidth (e.g., 8 bit); Potential mixed-precision con- figures set S under target rate R
Output: Quantization steps s, Quantized weights W
1: Quantize W to initial bitwidth b and get Wo:
ac 2: Compute the gradient of Wo with backward propagation g "W.
3: for S₁ in S do
4: Get layer-wise bitwidth 8' from S;;
5: for 1= 1,2,..., N-th layer in Wo do
6:
Compute channel-wise quantization parameter {s4*) kEi
> Eq. 47
7:
Compute de-quantized Wi s Round Wo/ s')s)); end for
8:
9:
Compute weight perturbation AW W W:
10: Compute Hessian- Vector Product HAW with VG = v(gAW):
>> Eq. Eq.
Compute criteria Q2 AW HAW;
12: end for
s, W arg min Q
E EXPERIMENTS
E. IMPLEMENTATION DETAILS
Evaluation. We conducted experiments on the UVG dataset?, which consists of videos, each with a resolution of 1920 x 1080 and recorded 120 FPS over or 2.5 seconds. applied a center crop to achieve resolution of 1920 960, similar to the preprocessing used in HNeRV and NeRV. For evaluation metrics, we employed PSNR to measure reconstruction distortion and bits per pixel (bpp) access bitrate. Additionaly, the Bjøntegaard Delta Rate (BD-Rate) (Bjontegaard, 2001) was calculated for each baseline codec.
Beauty, Bosphorus, HoneyBee, Jockey, ReadySetGo, ShakeNDry, YachtRide
17

---

Published as a conference paper at ICLR 2025
Algorithm 2 Encoding for Target Rate (Calibration, Sec. 3.2)
Input: Pretrained weights W (FP32); Potential mixed-preeision configures S under target rate
R: iteration T
Output: Bitstream, hpp
1: Search optimal bit configure and get 8, W:
> Algorithm
2: Forward propagation and collect the FP network output z;
3: for i= 1, T do
b. Calibration
4: Forward propagation and collect the quantized network output 2;
5: Compute L and gradient descent;
b eq. 16
6:
> Update first with few iteration and then update rounding parameters
7: end for
8:
Quantize W integer with calibrated QPs, i.e., s, v;
9: Get bitstream with lossless entropy coding.
Compute hpp
eq. 48
Here, the hpp calculation includes both the quantized network parameters ve and the quantization parameters
E bpp bpp + hpp, HX + Es.b. HXXXT'
(48)
where E denotes lossless entropy coding. For example, on 1080p video sequence with INT4 it and FP16 s, the bpp for HNeRV-3M is approximately: bpp 0.01, bpp, 0.00004. As shown, the contribution s to the overall bpp is negligible, but it is still included calculations for fairness.
Baselines. All baselines were implemented using open-source codes, including NeRV (Chen et al., 2021a), HNeRV(Chen et al., 2023), and HiNeRV (Kwan et al., 2024a).
NeuroQuant. In the mixed- -eexedipreiisonn bit allocation, we targeted the final weights size instead of the bpp, as the actual bpp cannot be accurately estimated after entropy coding, Specifically, the bitwidth was constrained within the range of 3bbt,, 8bit). The weights compression ratio (FP32 weights size/ quantized weights size) maintained within average range of [4.5, 10), where different base- lines and weights size had small difference. We calculated potential bitwidth configurations while allowing 5% error tolerance to avoid empty solution. In current implementation, we conducted search in group of predefined bitwidth configures for given target size, which can be finished in one minute. Integer Programming (Hubara al., 2021) also can achieve same objective.
Once the bits are allocated, we employed the Adam optimizer (Kingma, 2014) to calibrate quanti- zation parameters (e.g. quantization steps, weight rounding) minimize distortion. For frame-wist INR-VC systems like NeRV and HNeRV, the batchsize was set to 2, while for patch-wise INR-VC systems like HiNeRV, the batchsize was set to 144. The learning rate was set 3e 3 with a cosine annealing strategy. QP were be optimized for 2.1 x 10a iterations, although most cases converged fewer than 1.5 x 104 iterations. All experiments were conducted using Pytorch with Nvidia RTX 3090 GPUs.
Our code will be made open-source upon the release of the paper.
Task-Oriented PTQs. Benchmarks were based open-source implmentations of AdaRound, BRECQ, QDrop, and RDOPTQ. These methods were reproduced for INR-VC systems, using the same batchsize and learning rate as NeuroQuant, which provided better results than naive learning rate. Each layer/block was optimized for 2.1 x 10 iterations.
INR-Oriented QATs. For FFNeRV, the QAT process is summarised as follows:
[(2b - 1). tanh (/w))
Forward: i = sign(w)
(49)
26
dC ac
Backward:
(50)
Im
18

---

Published as a conference paper at ICLR 2025
For HiNeRV, the QAT process is based QuantNoise (Stock al., 2021) but forbidding the naive Straight-Three Through Estimator (STE):
Forward:
i = w mask + i (1 mask),
(51)
w (2 1)
2: max(ww) (
where ib
(52)
2 max(w)
2
OC Oc
Backward:
mask,
(53)
dw dw
w (2
2 max(un)
Inference:
(54)
2 max
2
where mask is random binary tensor with the same shape as w. This random dropping is also similar to QDrop.
Generalized Neural Video Codecs. We also included representative generalized Neural Video Coding Systems, DCVC (Li et al., 2021a) and DCVC-DC (Li et al., 2023), to compare with the existing non-generalized INR-VC systems. Pretrained models were used to the UVG dataset, where all frames of each video were evaluated. The group of pictures (GOP) was set to 32, consistent with other learned video coding methods.
Generalized Traditional Codecs. The command to encode using x264 in our paper is:
ffmpeg
-s (width) (height)
feix_fmt yuv444p101e
-framerate (frame rate)
5
-i (input yuv namo)
-c:v 1ibx264
-preset veryslow
-g 32 pp (qp)
10
(bitstream file name)
The command to encode using x265 in our paper is:
ffmpeg
-s (width) (height)
-pix_fme fmt yuv444p101e
-framerate Iframe rate)
-i (input yuv name)
-c:v libx265
-preset veryslow
-x265-params
''p== (qp) :keyint=32''
10
(bitstream file name)
The pre-process followed the suggestions in Sheng et al. (2022); Li et al. (2024), where we used BT,601 color range convert between YUV and RGB.
E.2 VARIABLE- RATE COMPARISON
Here we further compare NeuroQuant with two typical variable-rate techniques: (1) Neural Network Coding (NNC) techniques (Wiedemann al., 2020) and Entropy Regularization (EM) techniques (Gomes et al., 2023; Waan et al., 2024b; Zhang et al., 2021). NNC uses video codec to compress neural network, while EM introduces additional weight entropy regularization. We employ Deep- CABAC (Wiedemann
al., 2020) and Gomes et al. (2023) as the typical represented methods, respectively. We use the NeRV across three video sequences: Beauty, Jockey, and ReadySetGo to compare. The results summarized in the following table:
As shown, NeuroQuant consistently outperforms DeepCABAC and Gomes et al. across different sequences and bitrates. These results demonstrate the effectiveness of our method in improving rate- distortion performance. For one training iteration, Gomes et al. (2023) involves additional overhead due to entropy estimation, resulting approximately x 1.4 encoding time compared to NeuroQuant.
19

---

Published as a conference paper at ICLR 2025
Table 3: Variable-Rate comparison.
Methods
Bpp
Beauty
Jockey
ReadyS
Avg.
Full Prec. (dB)
33.08
31.15
24.36
29.53
DeepCABAC (Wiedemann et al., 2020)
0.016
32.98
30.94
24.24
29.39
Gomes et al. (Gomes et al., 2023)
0.016
32.91
30.66
23.92
29.16
NeuroQuant (Ours)
0.016
33.04
31.09
24.31
29.48
DeepC ABAC (Wiedemann et al., 2020)
0.013
32,43
30.23
23.92
28.86
Gomes et al. (Gomes et al., 2023)
0.013
32.78
30.29
23.61
28.89
NeuroQuant (Ours)
0.013
32.97
30.96
24.18
29.37
DeepCABAC (Wiedemann et al., 2020)
0.011
31.59
28.70
22.85
27.71
Gomes al. (Gomes et al., 2023)
0.011
32.63
29.89
23.26
28.59
NeuroQuant (Ours)
0.011
32.83
30.67
23.85
29.12
Besides, NeuroQuant achieves precise bitrate control by adjusting quantization parameters, as bitrate directly proportional the number of parameters and their bitwidth. In contrast, entropy-based methods can not directly estimate compressed bitrate from the Lagrangian multiplier A. Instead, it requires multiple encoding dependent, reducing its universality and reliability.
fit mapping from À to bitrate. This mapping is sequence-
Additionally, acknowledge that the lossless entropy coding used in NeuroQuant currently is less advanced compared to CABAC-based techniques used in DeepCABAC. However, as a quantization method, NeuroQuant compatible with various entropy coding techniques. In future work, we aim to incorporate CABAC and EM techniques into NeuroQuant as discussed in Sec. 3.3.
E.3 DIVING INTO NEUROQUANT
To analyze the efficiency of NeuroQuant, we conducted series of deeper studies on the three sequences (Beauty, Jockey, and ReadySetGo) using NeRV-3M and 4-bit precision. Below, we sum- marize our findings.
E.3.1 LAYER DEPENDENCE A GOOD PROPERTY FOR INR?
Benefits of Inter-Layer Dependencies: INR models are inherently non-generalized and tailored to represent specific video data. This specialization often leads to strong inter-layer dependencies, where the contribution of each layer the overall representation is tightly coupled. For video representation, such strong coupling can reduce redundancy across the network, leading to better rate-distortion performance in INR-VCs. contrast, excessive independence could indicate poor utilization of the network's capacity.
Challenges of Inter-Layer Dependencies: Strong dependencies complicate quantization, as per- turbations in one layer can propagate across the network. This is where network-wise calibration, as proposed in NeuroQuant, becomes critical. Additionally, dependencies can limit scalability and robustness, architectural modifications disrupt the internal balance of the network.
Simple Experiment: For INR-VCs, the overfitting (dependence) degree increases with the training iterations growing. To measure dependence, quantized the fourth layer using vanilla MinMax quantization without any calibration, and observed its impact on final loss. Results for varying training epochs are shown below:
The results indicate that stronger dependence means larger loss changes with quantization but pro- vides better representation performance.
For future work, exploring a middle ground- where moderate independence is encouraged to bal- ance representation fidelity and quantization robustness-miss be interesting direction.
20

---

Published as a conference paper at ICLR 2025
Beauty
Beauty
32
Jockey
Jockey
Representation (dB)
30
Readys
ReadyS
Avg.
5
Avg.
28
Dift. (dB)
26
3
24
22
Stronger
1
Stronger
20
Dependence
Dependence
100
200
300
0
100
200
300
Epoch
Epech
(a) Representation Capcaity
(b) Quantization Error
Figure The influence inter-layer dependence.
E.3.2
ABLATION STUDIES
Batch Size. We examined the impact of varying batch sizes compression performance. As shown Table 4, increasing the batch size results in diminishing returns PSNR, with larger batch sizes also introducing greater complexity. For NeuroQuant, batch size of 2 chosen the default setting.
Table 4: Batch Size Ablation
2
8
Beauty
32.78
32.79
32.79
32.80
Jockey
30.54
30.58
30,60
30.62
ReadyS
23.75
23.79
23.82
23.84
Avg 29.02
29.05
29.07
29.09
Learning Rate. We experimented with different learning rates, shown in Table 5. A learning rate of 3e 3 was selected as the default NeuroQuant setting.
Table 5: Learning Rate Ablation
1.5e -
2e
3e
3
5e 3
Beauty
32.86
32,81
32.83
32.79
Jockey
30.58
30.62
30.67
30.53
ReadyS
23.88
23.82
23.85
23.79
Avg
29.11
29.08
29.12
29.04
Objective Trade-off. Eq. 14 describes the final calibration objective, which includes both the dis- tortion term Ln and the regularization term LRzg- We explored the trade-off between these two objectives by varying the regularization weight A, shown in Table 6. The results indicate that NeuroQuant maintains robust performance across different regularization weights.
Quantization Granularity. We compared different quantization granularities, focusing on channel- wise (CW) and laser-wise (LW) quantization strategies. As shown in Table 7, while layer-wise granularity significantly degrades performance, NeuroQuant still yields satisfactory results in this context. Channel-wise quantization delivers superior performance, supporting the correctness the proposed method.
21

---

Published as a conference paper at ICLR 2025
Table 6: Objective Trade-off Ablation
= 0.1
= 0.01
À = 0.001
Beauty
32.79
32.78
32.78
Jockey
30.58
30.58
30.56
Ready 23.80
23.77
23.76
Avg
29.06
29.04
29.03
Table 7: Quantization Granularity Ablation
I.W w/o NeuroQuant
L. w/ NeuroQuant
CW w/ NeuroQuant
Beauty
28.90
32.12
32.79
Jockey
22.49
29.08
30.58
ReadyS
19.50
22.51
23.80
Avg
23.63
27.90
29.06
Iterations. We analyzed the performance gains over different numbers of iterations, as illustrated in Fig. NeuroQuant achieves significant improvements in the first 10 iterations, indicating its rapid convergence during calibration.
32.8
30.50
PSNR (dB))
PSNR
30.00
23.25
32.4
29.75
23,00
32.2
29:30
22.75
29:25
22.50
29.00
5000
10000
15000
20000
5000
10000
15000
20000
5000
10000
15000
20000
Iterations
(a) Beauty
(h) Jockey
(c) ReadyS
32.0
30.50
FSNR (dB)
32.6
23.50
30.00
23.25
PSNR
23,00
32.2
20.50
22.75
32.0
29.00
10a
10a
10a
10a
101
10f
10:
302
10a
10a
10P
10-
10a
102
10
Iferations
Iterations
lterations
(d) Beauty (log scale)
(e) Jockey (log scale)
(f) ReadyS (log scale)
Figure 7: Iterations Ablation.
Due to the tremendous encoding complexity HiNeRV (10x compared NeRV, e.g., more than 21GB memory requirement and nearly one GPU day for 3M weights), exclude HiNeRV in next.
Unified R-D Characteristics. Fig. 8 presents the Rate-Distortiinn (R-D) curves for three different video sequences: Jockey, Beauty, and ReadySetGo. In addition to the commonly used Jockey se- quence, we include the less dynamic Beauty sequence, where baseline models tend to exhibit satura- tion (showing less than I dB PSNR improvement across the entire bitrate range). We also consider the highly dynamic ReadySetGo sequence, where all baselines show the worst R-D performance within the UVG dataset.
22

---

Published as a conference paper at ICLR 2025
NeuroQuant significantly outperforms naive quantization techniques in terms of compression effi- ciency. For the Beauty sequence, both NeRV and HNeRV show à saturation effect, where increasing the number of weights does not yield noticeable gains in distortion reduction. In this case, Neuro- Quant performs worse compared to its performance on other sequences..
On the other hand, NeuroQuant exhibits slight difference among different beseliens and video quences. As highlighted the zoomed-in areas of the R-D curves, NeuroQuant-4hin achieves higher PSNR in the NeRV on ReadySerGo. When it comes the NeRV on Beauty, NeuroQuant achieves higher PSNR. Despite these differences, various precision levels (bitwidth) display nearly unified R-D characteristics.
Visualization. We further visualize the lowest bitrate (0.016pp) for all three sequences in Fig. 9, 10 and 11. Compared with naive 32bit floating-point HNeRV (FP32), NeuroQuant can compress weights to INT4 without visual information loss.
NeRV Jockey
HNeRV on Jockey
35
Naive-8bit
34
Naive-4bit
36
Naive-4bit
NeuQ-6bit
NeuQ-6bit
PSNR (dB)
33
NeuQ-5bit
PSNR (dB)
34
NeuQ-Sbit
NeuQ-4bit
NeuQ-4bit
32
31
32.5
33.0
30
30
$2.0
32.5
29
15
Model Size (MB)
Model Size (MB)
NeRV on Beauty
HNeRV on Beauty
34.50
34.0
Nalve-Bblt
Nalve-Bbit
33.8
Nbive-4bit
34.25
Naive-4bit
PSNR (BB)
33.6
NeuQ-6bit 6bit
Sbitt
PSNR (dB)
34.00
NeuQ-6bit
NeuQ-5bit
33.4
NeuQ-Abit
33.75
NeuO-4bit
33.5
34.0
33.50
33.0
33.25
33.3
32.8
33.0
33.2
32.6
32.75
25
Model Size (MB)
Model Size (MB)
NeRV on ReadySetGo
HNeRR ReadySetGo
32
Naive-8bit
Nalve-Bbit
Nalve-4bit
30
Naive-4bit
NeuQ-6bit
NeuQ-6bit
PSNR (dB)
NeuQ-5bit
PSNR (dB)
NeuQ-5bir
28
NeuQ-4bit
NeuQ
26.0
26
25.5
24
10
15
20
25
10
15
20
25
Model Size (MB)
Model Size (MB)
Figure 8: Quantitative comparison of NeRV (left) and HNeRV (right).
23

---

Published as a conference paper at ICLR 2025
FP32
NeuroQuant
HN+RV INT4 PSNR: 32 19dB
PSNR: 33.33IB
PSNR 32:9B
Model Size: 124MB MB
Model Size: 1.5 MB
Model Size: LBMBB
Figure 9: Comparison around 0.01bpp with Beauty sequence in HNeRV.
SARAR
SARAR
FI32
NeuroQunnt
HNeRV INT PSNR: 2684BB
PSNR: 29.3BB
PSNR: 28.66B
Model Size: 124MB
Model Size: 1B
Model Size: 155MB
Figure 10: Comparison around 0.01 with Jockey sequence in HNeRV.
M
Tucky
FP32
NeuroQuant INT4 PSNR: 25.3IB
HINRV INT4 PSNR: 2243dBB
PSNR: 26.3918
Model Sixe: 124MBB
Modl Sio:: 1.55 MB
Model Size: 1.55 MB
Figure 11: Comparison around 0.01bpp with ReadySetGo sequence HNeRV.
24


Reviews:
[{"review_id": "vfpooKcpUa", "reviewer": "Reviewer_M1Dk", "summary": "In this paper, the authors propose a post-training quantization method tailored to implicit neural representation (INR) based image and video compression. They argue that existing post-training quantization methods are not suitable for INR-based image and video codecs, and advance the existing PTQ for this specific task. Furthermore, the authors demonstrate how their proposed method can tackle variable rate coding with INR using a single INR model. They experimented with their method on top of existing INR methods and showed that their method performs better with minimal reconstruction loss.", "questions": "1. How the figure 3 is generated. what is the architecture details of the INR network, and what kind of data is used for fitting. Does the analysis is also true for MLP?\n\n2. What is the significance of the equation 5 and 6, whether this optimization problem is solved in the paper. In the abstract, it was mentioned, PTQ was formulated as the mixed-precision quantization, it was not evident for me where the mixed precision quantization is solved. From the table 1, it seems like the mixed-precision quantization was not used. Also detail how the mixed-precision quantization is used.\n\n3.  For Nagel et.al (2020) which formulation was used to compare? In Nagel et. al (2020) whether the equation (25) or equation (21) is used in their respective paper. It is important to specify, the equation (25) is closer to the loss (network calibration) in the proposed paper.\n \n4. In equation (16), how the $\\mathbf{s}$ is determined, is it learned with respect to the task loss, or is it optimized by the greedy search or is it fixed parameter. \n\n5.  the network-wise calibration might also be applicable to the generalized neural network, did the authors have done any experiments on the generalized neural codec? \n\n6. For the quantization aware training approach, the weights initialization with post-training quantization will improve the convergence and reconstruction quality of the QAT. It would be nice to test this feature on some INR method which uses QAT.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Using one single model for different bit rates with post-training quantization is interesting. This alleviates the need to train a model for each bit-rate, this will decrease the training time.\n\n2) The paper provides the mathematical insights to their proposed method, inspired from the Nagel et. al (2020), and formulates the post-training quantization objective with respect to the network calibration. \n\n3) The experimental results show that the proposed method has a significant gain in the variable-rate coding.", "weaknesses": "1. The authors failed to compare their proposed approach with Neural Network Coding tool (NNC) [1] which also performs post-training quantization, and also can offer variable-bitrate coding by adjusting QP parameters. The authors should compare their method with NNC.\n\n\n [1] S. Wiedemann et al., \"DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks,\" in IEEE Journal of Selected \n          Topics in Signal Processing, vol. 14, no. 4, pp. 700-714, May 2020\n          https://arxiv.org/abs/1907.11900", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fqlt3rf9rA", "reviewer": "Reviewer_Zox4", "summary": "NeuroQuant is a cutting-edge post-training quantization method for variable-rate video coding that optimizes pre-trained neural networks for different bitrates without retraining.", "questions": "How does NeuroQuant differ from traditional quantization methods in terms of bitrate adjustment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method achieves variable-rate coding by adjusting QPs of pre-trained weights, eliminating the need for repeated model training for each target rate, which significantly reduces encoding time and complexity.\n\n2. The method demonstrates superior performance in compression efficiency, outperforming competitors and enabling quantization down to INT2 without notable performance degradation.\n\n3. The paper proposes a unified formula for representation-oriented PTQ calibration, streamlining the process and improving its applicability across different models.\n\n4. The approach is backed by both empirical evidence and theoretical analysis, ensuring its robustness and effectiveness in practical applications.", "weaknesses": "N/A\nActually, I am not very familiar with this field, so please have AE consider the opinions of other reviewers more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s5cVB9Qtnp", "reviewer": "Reviewer_3W48", "summary": "In this paper, the authors propose post-training quantization for INR-VCs, which achieves variable-rate coding more efficiently than existing methods that require retraining the model from scratch. The proposed model realizes variable bitrate by considering the sensitivity of the weights to quantization, while also incorporating better theoretical assumptions for INR-VC compared to other post-training quantization techniques. The proposed method demonstrates both improved RD performance and faster encoding speed.", "questions": "- The encoding runtime for HiNeRV 3M (22 hours) looks significantly longer than reported in the original paper, even accounting for differences in GPUs. Additionally, the reported memory usage seems unusually high. Are there any differences in configuration, such as the use of FP16 precision in these experiments?\n- How much time is required to obtain additional rate points with models like NeRV, HNeRV, FFNeRV, and HiNeRV? Although these models require a pretraining phase, the pretrained model can be fine-tuned to produce multiple rate points by adjusting quantization levels or using entropy coding with regularization [1]. Fine-tuning time is substantially shorter than full training (e.g., 30 epochs for QAT versus 300 + 60 epochs for HiNeRV).\n- What is the computational cost (in terms of MACs and wall time) for the proposed calibration process compared to QAT?\n\n[1] Gomes, Carlos, et al. \"Video compression with entropy-constrained neural representations.\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The results look promising. Although NeuroQuant achieves only a marginal improvement over the current best INR-VC (-4.8%), it provides greater efficiency in obtaining multiple rate points.\n- The experiments comparing different quantization methods are comprehensive, which will be helpful for future work in this area.", "weaknesses": "- In Table 2, excluding the pretraining time for NeuroQuant does not seem appropriate. Even with NeuroQuant, pretraining is still required, and the current presentation may be misleading. The authors should consider reporting the pretraining and fine-tuning times separately for both the baseline models and NeuroQuant.\n- Similarly, the claim of an 8x encoding speedup is also misleading, as it excludes the pretraining time required for INR-VC encoding (even though NeuroQuant avoids full retraining for each rate point).\n- Variable/learnable quantization levels for INR-VC have been explored in related works [1,2,3], so the paper’s claim is inaccurate (e.g., line 43). These methods, which resemble the proposed mixed-precision quantization, also enable fine-tuning for multiple rate points from a single pretrained model (but with QAT). These methods should be discussed and compared in the paper.\n- For a fairer comparison, the comparisons to x264/x265 should avoid using the 'zerolatency' setting, as the INR-VCs in the paper inherently have non-zero latency.\n- For ablation study, more sequences should be use for obtaining a representative result.\n\n[1] Zhang, Yunfan, et al. \"Implicit neural video compression.\"\n[2] Gomes, Carlos, et al. \"Video compression with entropy-constrained neural representations.\"\n[3] Kwan, Ho Man, et al. \"Immersive Video Compression using Implicit Neural Representations.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BrpMwZkfMQ", "reviewer": "Reviewer_i6EH", "summary": "This paper proposes a novel post-training quantization approach designed for INR-VC called NeuroQuant that enables variable-rate coding without complex retraining. It redefines variable-rate coding as a mixed-precision quantization problem. Through network-wise calibration and channel-wise quantization strategies, NeuroQuant achieves SOTA performance compared to popular PTQ and QAT methods.", "questions": "1.\tI think that the RD performance of the PTQ method may be slightly inferior to QAT. Could you explain why the proposed PTQ method has better RD performance compared to FFNeRV/HiNeRV?\n2.\tCould you provide a detailed explanation of the Encoding Complexity section? Does it refer to the encoding complexity of a single bitrate or multiple bitrate?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Compared with existing quantization methods, the proposed method could achieve significant performance improvement, indicating the efficiency of the proposed method.\nThere are some highlights for the proposed post-training quantization method for INR-VC:\n1.\tA criterion for optimal sensitivity in mixed-precision INR-VC was proposed, enabling the allocation of different bitwidth to network parameters with varying sensitivities.\n2.\tThrough network-wise calibration and channel-wise quantization strategies,  NeuroQuant minimize quantization-induced errors, arriving at a unified formula for representation-oriented PTQ calibration.", "weaknesses": "1.\tThe authors did not provide a detailed explanation as to why the proposed PTQ method would be superior to QAT methods such as FFNeRV and HiNeRV.\n2.\tIn the Encoding Complexity section, the authors did not provide a detailed explanation of whether the acceleration brought by NeuroQuant is due to the absence of QAT optimization during training or because NeuroQuant does not require retraining for adjustments at different bitrates.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pYhkzIpFdM", "reviewer": "Reviewer_6keJ", "summary": "In this paper, the authors aim to introduce the variable rate control for INR-based video coding, by simply using PTQ. Therefore, they investigate PTQ approaches with mixed precision on those INR models. They first validate a weak layer independence in such non-generalized INR models. This challenges Hessian-based quantization methods, as they often follow this assumption and adopt diagonal Hessians. Then the authors propose a perturbation-based approach to estimate the intractable Hessian-involved sensitivity criterion (Omega) in the section with eq.9 and eq.10. Therefore, they can perform bit allocation for mix-precision quantization. Then the authors adopt network-wise calibration to further decrease the quantization error. The proposed approach named NeuroQuant achieves a cutting-edge performance w.r.t. both single QP and the whole RD curve.", "questions": "1. I wonder why the improvement is such impressive, compared to the existing quantization approaches. Is the channel-wise quantization conducted on both weight and activation?\n1. It seems like the key point of this approach is to adopt a mix-precision one-block BRECQ on INR video coding models. Despite the story about the unique properties of non-generalized INR models, can we further develop the method to a more general MP PTQ with calibration?\n2. Is this inter-layer independence a good property for evaluated INR models, or an ill pose? I would appreciate if the authors provide some insight about this property.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The authors clearly explain their motivation for adopting mix-precision PTQ for variable-rate INR video coding. The experimental results are impressive, with significant PSNR improvements (*e.g.* 0.2 db @6bit and 3 db @2bit for NerV) on all the experimental settings.", "weaknesses": "1. The comparison in Table 1 may be unfair. If I understand correctly, the proposed approach involves mix-precision quantization, which helps bit allocation among layers. Therefore, it introduces extra quantization step parameters (${s}$ in eq.17) to store. I wonder whether the bpp calculation in Fig.4 and Fig.5 considers this quantization parameter. On the other hand, AdaQuant and BRECQ are fix-precision methods so this parameter can be omitted. The authors should clarify their evaluation details, especially the calculation of bpp.\n1. The calibration objective derivation in section 3.2 is similar to the *Network-wise Reconstruction* situation discussed in the existing BRECQ paper (Li et al. 2021b, section 3.2). And the authors are also aware of this prior approach. Intuitively, intra-network independence can be seen as one-block intra-block independence, and BRECQ covers this. In behavior, both the calibration methods adopt an MSE-form objective. Thus, I cannot easily recognize those analyses as the contribution of this paper. I request the authors further clarify their contribution against the existing approaches.\n2. It would be better to provide more intuitive explanations of the proposed approach, e.g. diagram figures and pseudo algorithm. Considering that not all experts in the video coding community are familiar with model quantization, the math formulations are somewhat confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)