Scalable Neural Video Representations with Learnable Positional Features
Authors: Subin Kim
Sihyun Yu
Jaeho Lee
Jinwoo Shin
Venue: NeurIPS 2022 Accept (2022)
Keywords: coordinate-based neural representations

Abstract: Succinct representation of complex signals using coordinate-based neural representations (CNRs) has seen great progress, and several recent efforts focus on extending them for handling videos. Here, the main challenge is how to (a) alleviate a compute-inefficiency in training CNRs to (b) achieve high-quality video encoding while (c) maintaining the parameter-efficiency. To meet all requirements (a), (b), and (c) simultaneously, we propose neural video representations with learnable positional features (NVP), a novel CNR by introducing "learnable positional features" that effectively amortize a video as latent codes. Specifically, we first present a CNR architecture based on designing 2D latent keyframes to learn the common video contents across each spatio-temporal axis, which dramatically improves all of those three requirements. Then, we propose to utilize existing powerful image and video codecs as a compute-/memory-efficient compression procedure of latent codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality as 34.07$\rightarrow$34.57 (measured with the PSNR metric), even using $>$8 times fewer parameters. We also show intriguing properties of NVP, e.g., video inpainting, video frame interpolation, etc.


Scalable Neural Video Representations with Learnable Positional Features
Subin Kim*
Sihyun Yu-
Jaeho Lee²
Jinwoo Shin
Korea Advanced Institute of Science and Technology (KAIST)
PPohang University of Science and Technology (POSTECH) (subin-kim, sihyun. yu, finwoos](kkistt ac kr jaeho. leedpostech. ac ac.kr kr
Abstract
Succinct representation of complex signals using coordinate-based neural rep- resentations (CNRs) has great progress, and several recent efforts focus extending them for handling videos. Here, the main challenge is how to (a) alleviate compute-inefficiene in training CNRs to (b) achieve high-quality video encoding while (c) maintaining the parameter-efficiency To meet all requirements (a), (b), and (c) simultaneously, we propose neural video representations with learnable po- sitional features (NVP), novel CNR by introducing "learnable positional features" that effectively amortize video as latent codes. Specifically, we first present a CNR architecture based on designing 2D latent keyframes to learn the common video contents across each spatio-temporal axis, which dramatically improves all of those three requirements. Then, we propose to utilize existing powerful image and video codees as a compute-/memory-effii codes. We demonstrate the superiority of NVP on the popular UVG benchmark; compared with prior arts, NVP not only trains 2 times faster (less than 5 minutes) but also exceeds their encoding quality
compression procedure of latent
(measured with the PSNR metric), even using >8 times fewer parameters. We also show intriguing properties of NVP. e.g., video inpainting, video frame interpolation, etc."
1 Introduction
Recent advances in coordinate- based neural representations (CNRs) [9, 13, 17, 40, 46] have shown great promise in the field
paradigm for representing complex signals, including gigapixel images [29, 34], audios [40], 3D scenes [30, 33, 35], and even large city-scale street views [47]. Instead of storing signal outputs coordinate grid (e.g., image pixels), whose memory requirement scales unfavorably in terms resolution and dimension, CNRs represent each signal as a compactly parameterized, continuous neural network; they interpret signal as coordinate- -toeeee---vllll to-value function and train a neural network to approximate this mapping. CNRs enjoy numerous appealing properties, including data compression [11, 12, 59], super-resolution [6], novel view synthesis [18, 22, 33, 54], and generative modeling of complex, high-dimensional data [14, 24, 41, 42, 57, 60), while being parameter-efficient interpretation of a given signal in various scenarios.
In particular, several works have attempted to exploit CNRs to interpret video signals [5, 23, 42, 57] by learning a neural network f 3 + R23 with f(z, y.t) (r, 9) b) and exhibited their potential as a succinct representation videos, as well as providing numerous applications including video generative modeling [42, 57], video compression [5, 59], and video super-resolution [7]. They
"gqual contribution.
Videos are available at the website https::/subin-kttee wimbbiii-kim--ve ceriimmvv.itthuii github. io/MVP.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

NeRV(BPP: 0.938)
Instant-agp (BPP: 6.489)
(Vurs BPP: 0.189)
Ground Truth
Figure 1: Reconstruction results on Jockey in UVG-HD after training each model for "I minute" with single NVIDIA V100 32GB GPU. NVP can capture the detail of video containing dynamic motions, e.g., the legs of running horse, while the prior methods generate blurry artifacts.
observe that conventional CNR architectures [40, 46] often fail to encode large-scale videos due to their complex temporal dynamics accompanied by large spatial variations, but the problem can be remarkably mitigated by designing CNR architectures specialized for videos. For instance, Chen al. [5] proposes a CNR structure that focuses on the continuous modeling of the video signal only along the temporal dimension, allowing for more radical variations along spatial axes; it exhibits comparable encoding quality to existing powerful video codecs (e.g.. H.264 [51], HEVC [43] while enjoying lots of intriguing properties (e.g., denoising and video frame interpolation).
However, CNRs suffer from severe compute-inefffcienne: limiting their scalability to encode real- world, large-scale videos despite their advantages. To alleviate this issue, several works [26, 34, 37] have proposed new CNR architectures by separating CNR f into two parts;
h go for coordinate-to-later mapping (2, y, t) z and a latent-to-RGB mapping (z) = (r, g. 6). They construct ge as an embedding function defined with latent grids Ug in which the shape resembles the grid interpretation of a given signal (e.g., a 2D array of C-dimensional latent codes U, E IR H xC for image pixels) rather than a neural network. These approaches have shown promise in due to the strong locality induced by a grid structure of Ug; however, they result another problem: these architectures severely sacrifice the sarfamter--fffcieneeyy since the parameter size of 0 can be very large, growing proportionally to both the input coordinate dimension and the signal resolution. In this paper, we focus on developing video CNRs that are the best of both worlds: achieving high-quality encoding and the mampute/parrmmete--ehee
simultaneously.
Contribution. We introduce neural video representations with learnable positional features (NVP), a novel CNR for videos. NVP avoids requiring a single giant full-dimensional 3D array in ge by presenting learnable positional features that effectively amortize given video as "2D and 3D" latent grids with succinct parameters. Specifically, we decompose the coordinate-to-later mapping as
go = 99 X 98 X g0yr X 90 1
for 0 = (0xy, Oxt, Oyr, Oxyz),
keyframes
spurse features
where we present two types of latent grids for constructing these mappings (see Figure 2).
Latent keyframes: We first design "image-like" 2D latent grids U0,,- Uo Ua for go 90..: 90,y (respectively) that learn the representative video contents across each spatio-temporal axis and dramatically improve the parameter efficiency of NVP.3
Sparse positional features: We then introduce a "video-like" 3D latent grid U0 for 90 whose size is much smaller than the original video pixels, but effectively encodes video details locally.
MMeasured with a single NVIDIA V100 32GB GPU, NeRV [5] takes at least 15 GPU hours to encode single video 600 frames with 1920 x 1080 resolution for desired quality. "Such a deterministically across only the temporal direction.
consideration is different from conventional approaches for specifyying keyframes

---

(rig: 6).
Lanant keyframes
(1)
Moduletion
Sporse positional feetures
Usus
Overall illustration of our NVP. At given space-time coordinate, NVP computes a latent vector from latent keyframes (see Figure in Appendix for details) and sparse positional features. The latent vector is passed through neural network to compute the corresponding RGB outpur.
Figure 2:
Moreover, we propose compute-/menoory--fiiiienn compression procedure to further reduce the parameter by incorporating existing image and video codecs, e.g., JPEG [49] for images and HEVC [43] for videos, respectively. In particular, we treat 2D and 3D latent grids like the image or video pixels and utilize powerful compression pipelines for them. Our compression scheme does not require any re-training of trained parameters, which significantly increases compute-efficieet prior approaches to compressing CNR parameters but remarkably maintains the encoding quality. We also remark that such compression approach is not applicable to the hashing- latent grid of the prior method [34], while ours is "collision-fe free"" and maintains the video- or image-like structures. Finally, for the choice of hgg we suggest using a modulated network (with respect to the temporal coordinate) to improve the encoding quality of videos that contain dynamic motions.
to
We verify the effectiveness of our method the popular UVG benchmark [32]. In particular, NVP achieves the peak signal-to-noise ratio (PSNR; higher is better) metric of 34.57 in 5 minutes (with single NVIDIA V100 32GB GPU): it is achieved >2 times faster, even with using >8 times fewer parameters than the state-of-the- art on compute-effciinncy that reaches 34.07 in 10 minutes. Moreover, compared with prior arts on encoding quality, our method improves the learned perceptual image patch similarity (LPIPS [58]; lower is better) as 0.145 102 (9.7%% with a similar number of parameters while requiring 72.55% less training time. We also show numerous compelling properties of NVP, e.g., video inpainting, video frame interpolation, super-resolution, compression, and consistent frame-wise encoding results without deviating quality.
2 Related work
Coordinate-based neural representations. Coordinate-based neural representations (CNRs), also known as implicit neural representations neural fields, have emerged new paradigm for representing complex, continuous signals. They propose to encode signals through a neural network, typically a multilayer perceptron (MLP) combined with high-frequency sinusoidal activations [40] or Gaussian activations [9]. Prior works have focused on utilizing neural fields on various complicated data, e.g., gigapixel images [29, 34], 2D videos [5, 42, 57], 3D static scenes, [30, 33, 35], and 3D dynamics scenes [23, 39, 52]. In particular, most approaches have focused on constructing CNR architectures for encoding 3D scenes [3, 15, 47] and exhibited how employing specialized prior knowledge for given signal domain in the architecture can remarkably boost the encoding quality. Despite these successes, extending CNRs for videos is yet exploee...ore. In this paper, we aim to move toward developing CNR for videos by exploiting their unique temporal properties.
Hybrid CNRs. Rather than solely designing neural network of coordinate-to-RCC mapping, hybrid CNRs incorporate learnable latent codes that follow a grid structure, e-g., image CNRs combined with 2D latent spatial grids [6. 31], and 3D scene (or shape) CNRs with latent cubic grids [4, 8, 19, 26, 29, 37]. Specifically, they compute a latent vector using the grid-structured latent code and pass through neural network to compute the signal output given input coordinate. Such approaches have shown significant efficiencies in training time and encoding quality due to the powerful locality induced by -iddsshapdd latent codes. However, the number of parameters required for latent brid-based representations easily grows proportionally to the input coordinate dimension or data resolution, limiting the scalability of hybrid CNRs. Remarkably, some of the recent approaches have exhibited this inefficiency can be significantly mitigated by considering multi-level (or progressive) structures for latent grids [26, 29, 44, 45). While prior works primarily focus on encoding images or 3D scenes, we aim to design parameter-efffcient hybrid CNRs for videos.
3

---

3 NVP: Neural video representations with learnable positional features
We first formulate our problem setup follows. Given a video signal == (f,, f2...... fr) consisting of T video frames, the goal is to find a compact neural representation fw with parameters w, from which the original video v can be reconstructed with high quality. Here, the quality can be defined using various distortion metrics, e.g,, peak signal-to-noise ratio (PSNR) [16] and LPIPS [58], for evaluating a reconstruction quality and a perceptual similarity, respectively.
To achieve this goal, we take an approach based on coordinate- neural representations (CNRs)-
paradigm where each datum (e.g., video) is parameterized mapping. In particular, we aim to represent the given video using a neural network fw : R23 + R which maps the space-time coordinates (I, 2) of the video to corresponding RGB values (r.g. b). where fw is optimized with reconstruction objectives, e.g., mean-square error. Such an approach has significant potential, CNRs have shown to encode other continuous, complex signals (e.g-, 3D scenes) [40, 46] compactly while enjoying lots of intriguing properties, e.g., super-resolution [6] and denoising [5]. However, CNRs have suffered from tremendous time costs for training, and even in the case of videos, it is difficult to achieve high-quality encodings if one utilizes conventional CNR architectures that overlook the complex spatio-temporal dynamics of videos [5]. Our contribution lies in resolving these issues by designing "learnable positional features" that succinctly encode a video as latent codes with high quality and keeping their
neural network of coordinate
intact.
In the rest of this section, we provide a detailed description of each component in NVP. In Section 3.1, we explain the architecture of NVP. We then describe compression procedure in Section 3.2.
3.1 Architecture
We design our video CNR fw as a composition of two functions with parameterization w = (0, d): a coordinate- to-latnnt mapping ge and a latent-to-RGB mapping h2- Here, we decompose the coordinate-to- mappinng as ge = 90 X ge,, x g0,, x 90.,, with go (z, y. = (Zzy. Zet, Zyt Zzyt) (for 0 (0xy. xx: Byt, xyt)), where each 90,, I6.. 90,, is formalized with image-like 2D latent spatial grids URz,, Ua_.. Uø,. (respectively) and 98.r is designed with a video-like sparse 3D latent grid U We then present the latent-to-RGB mapping he (zxy. Zxt. Zye. Zzyr) = (r, g. b) to be a multi-layer perceptrion (MLP) modulated by another neural network. To explain our architecture, we assume all the input coordinate (xr y, 1) of go (and fw) is in [0, 1]3 C R without loss of generality.
Learnable latent keyframes. At a high level, learnable latent keyframes Ug.,- UR. Ug, are image- like 2D latent grids learned to capture the common representative contents in a given video v across each t-, y/, x-axis, respectively. For given input coordinate (x, y,t, we compute latent vectors Zay. Ext. Zyt from Ug.,, Uo.., U0 individually. We explain our keyframe only with Ua,, by letting U = Uyz for simplicity, but note that other keyframes Uo.y Ua.. operate in the similar manner.
Formally, U is L 2D spatial grids of C-dimensional latent codes dij, whose resolution Hi x Wi:
U == (U1,...,U') U2),
Ui= == (uly) E #ixxxx for 1,...,,, L,
Vij E R for 1 sii Hi, 1 i Wt.
Here, the keyframe follows an L-level multi-resolution structure, i.e., for each level 1, the height H and the width Wj become different H; - | H1 and W W L-'- W, with fixed 7 >> I and H1, W1 O, where |-] indicates the floor function of the input. Since U Ua,, is shared over the rraxis, we compute the latent vector Zye == (zt,,.. ... 244 by considering only the value of y and at a given coordinate (a, y. t). Specifically. for = 1...... L, each 2/2 is a linearly interpolated vector of four vectors in the spatial grid Uj. where the indices of these vectors chosen as the closest ones to the relative position of the input coordinate (y. () E [0. 1j2:
(m, n) = (LyHi), [tW]]) for [ I....,L
2/c 1erp (HH, m, tWr - mn); u/ um u/ 1, 2
where lerp indicates linear interpolation operation the input coordinate between given vectors.
Note that we learn the keyframe as the latent codes, unlike conventional approaches that specify the keyframe in deterministic manner from T' vidco frames (f,, .......,ff7. fr); it encourages to capture

---

the representative contents of the video over each spatio-temporal direction better. We also remark that our architecture involves two keyframes U Un, that are considered across spatial axes. Considering these keyframes may not be beneficial in the RGB space as the spatial variation of video pixels often large. However, in our approach, these keyframes are learned under a more flexible, continuous latent space, and thus the representative frames can even be found in these spatial directions in such a space while encoding the RGB outputs the video accurately.
Finally, recall that UZg. U0 Ua,. consist of multi-resolution spatial grids, i.e., the resolution of spatial grids in each latent keyframe grows from coarse to fine. Since natural scenes often include repeating patterns in various scales, e.g., a scene of flowers of different sizes, this multi-resolution architecture promotes learning common patterns with reduced memory and computation costs, which is also validated in Müller et al. [34].
Sparse positional features. Given space-time coordinate (ar, y; l), we compute the latent vector Zxyt with Ug that represents the local details of the video at this input position. Here, U0, = (usgk) E IR Hf XSx is 3D sparse grid of D-dimensional latent codes, i.e., the 3D grid size H W x S is much smaller than the size of the video pixels of 3D RGB grid:
Up = (uijk) E R #XXxDDD Mijk E ERD for 1 S i < H, S S W, 15k58.
To evaluate the latent vector Zzyt, we concatenate h x latent codes in Uy that their indices are near the relative position of the input (x,, y, 1): (m, n. k) [[=H], [yW] [SS])
Zeyt (Ummk Um 1 1.++-11)
where h, 8, 8 >> O are given as hyperparameters.
The locality of U0 as 3D latent codes dramatically alleviates the compute-inefficiency in fitting videos, in contrast to conventional CNRs where the entire parameters are shared (as a neural network) for arbitrary input coordinates (z, y. t) and thus require significant training cost. Moreover, recall that we construct Ue as sparse 3D grid of latent codes; remarkably, efficienty captures the video details even if the size is smaller than the number of video pixels since the common contents of a given video are effectively encoded with the latent keyframes U U U0
Note that we design U0,,. as sparse 3D grid; each single Uijk should represent wide area of videos solely without concatenation. As the single latent vector may lack expressive power to represent such wide range and may result in non-smooth transition as the CNR output. Hence, we mitigate this issue by concatenating multiple vectors near each other (see Figure 10 in Section 4.3).
Instead of directly selecting near latent codes from U Jeg. one may consider upsampling of Ue using linear interpolation before selecting the close latent codes at a given coordinate. Such an interpolation further helps each latent code to learn smoother representations and also generalizes representing video frames at unseen coordinates during training. Mcanwhile, this upsampling requires more computing time compared to the computational cost of other modules in NVP, and thus faces trade-off between the smoothness and compute-effiiiency in training (see Table 4 in Section 4.3).
Modulated implicit function. With the latent vector z [Zay, Zxt. Zyt. Zxyz] evaluated from go- a naïve design choice of the latent-to-RGB mapping ha is to utilize a MLP that maps z to the corresponding RGB output (r, g, b). However, if video contains temporally dynamic motions, we found such a simple MLP architecture occasionally lacks expressive power and can be difficult to capture the complex dynamics of the given video, even with the large network size of hgs
To cireumvent this issue, we design he to be K-layer MLP (coined as a synthesizer network) modulated by another modulator network [31]: where the latent vector z and the time coordinate t are passed through the modulator and the synthesizer, respectively. Here, the modulator network utilizes piecewise linear activations (e.g.. ReLU), where the synthesizer uses sinusoidal activations. Specifically, an RGB output (r: g, b) of the latent vector z (from (1. y, f)) is computed as follows:
0gg = L,
O3 Z O s by) for I,...,K 1-1,
(r.g,b) = Ax +b.
where Ag, by are weights and biases of k-tth layer of the synthesizer, is k-th hidden feature of the modulator, and & denotes an element-wis product. It helps achieve high-quality encoding rapidly in carly training iterations and often at the convergence than naive MLP (See Table 3 for details).
5

---

Table 1: PSNR, FLIP, and LPIPS of different CNRs encode videos in UVG-HD under each encoding time. t and I denote higher and lower values are better, respectively. Subscripts denote standard deviations, and bolds indicate the best results. indicates applying the method without the corresponding compression scheme. We report the BPP values of NeRV without compressing parameters if the encoding time is I 1 hour since the NeRV's compression requires longer time. On the other hand, the compression procedure of NVP only takes less than 1 minute, but for fair comparison, we do not apply to NVP as well whenever NeRV not compressed.
Encoding time
Method
BPP
PSNR (t)
FLIP (4)
LPIPS (4)
Instant-ngp [34]
7.580
33.15=3.19
0.090110.034
0.226:10.113
5 minutes
NeRV-S* [5]
1.072
24.16=5.17
0.219+00.097
0.542:+0.180
NVP-S" (ours)
0.901
34.57=2.62
0.075:10.021
0.190=10.100
Instant-ngp [34]
7,580
34.07=3.01
0,082++0.030
0.204+001105
~0 minutes
NeRV-S" [5]
1.072
26.53=5.92
0.176:10.088
0.460=1.1184
NVP-S* (ours)
0.901
35.79-2.31
0.066
0.160+00008
Instant-ngp [34]
7.580
35.69-2.72
0.071+00002
0.162++0.090
~1 hour
NeRV-S* [5]
1.072
33.26-4.31
0.094.10.038
0.240
NVP-S" (ours)
0.901
37.61=2.20
0.052++0.011
0.145:+0.106
SIREN [40]
0.284
27.20=3.77
0.169:+0.059
0.409=0.112
FFN [46]
0,284
0.153++0.055
0.44
~55 hours
Instant-ngp ngp [34]
0.229
28.81
0.155:+00057
0.390::..135
NeRV-S [5]
0.201
14.97
0.067++0.023
0.163++0.101
~ hours
NVP-S (ours)
0.210
36.46 2.
0.067+000017
0.135++0088
SIREN [40]
0.284
26.09=3.88
0.175-00008
0,486++0118
FFN [46]
0.284
29.53=3.44
0.135110.052
0.391
>40 hours
Instant-ngp [34]
0.436
29.98=3.39
0.138++0.051
0.358+t0.140
NeRV-L [5]
0.485
35.00=3.31
0.079-10.020
0.145::0.100
~1 hours
NVP-L (ours)
0.412
37.47
0.062-10.017
0.102:L0.01
3.2 Compression procedure
Recall that we aim to find "compact" video CNRs; several works have focused on reducing the number of coordinate-based neural representations parameters (or bits) after training while maintaining their performance. In particular, they have relied on exploiting existing well-known techniques for neural network compression, e.g., exploiting magnitude pruning [5, 21] or quantization [5, 59], and exhibited considerable results. However, these approaches mainly involve a re-training of CNR parameters, requiring severe computation costs, and thus are not suitable for practical scenarios.
Instead, we propose a compression pipeline for NVP, which does not require re- re-training, yet signifi- cantly reduces the number of bits while preserving the video quality. Our main idea is to incorporate existing image and video codecs that have shown their promises for the compression of given pixels. In particular, we focus on compressing keyframes Uo,- Uo... Uo and sparse positional features U as the parameter size of the modulated implicit function h& is neglectable compared with those. Specifically, we quantize Ua_, and UØ,,- Up_.- U0,, as 3D/2D grids of 8-bit latent codes and regard them as video and image pixel grids, where the number of the channel becomes the dimension of latent codes. Based on these interpretations, we compress these latent codes by utilizing existing video and image codecs, e.g., HEVC [43] for videos and JPEG [49] for images. Intriguingly, we found this procedure can significantly reduce the parameters while notably maintaining the video quality without any fine-tuning of the MitettttoRRGD mapping h& (See Section 4.3).
4 Experiments
We verify the effectiveness of our framework on UVG-HD [32], a representative benchmark for
evaluating video encodings. Experimental results demonstrate that our neural video representations with learnable positional features (NVP) simultaneously improves the overall performance by (a)
alleviating a compute-inefficiency in training, (b) achieving high-quality video encoding, and (c) maintaining parameter-efficiency. We also show applications of our NVP, including video inpainting and spatio-temporal interpolation. Finally, conduct ablation studies to validate each component.

---

Guond
Choadd Irith
N BP
Neky BPP-0.550
FLIP
FLIP
SVP& BAPLO Het.
NYP (OURS, BET
FLIP
FLIP
100
100
200
Figure 3: Illustration of reconstructions Yachtride (left), and ShakeNDry (right) in UVG-HD. FLIP indicates the output of evaluating its metric. The red box is zoomed in as the image at the right.
Evaluation. We follow similar setups on prior work [5] proposing coordinate- tations (CNRs) for videos. All reported numbers are averaged over 7 videos in UVG-HD: Beauty, Bosphorus, HoneyBee, Jockey, ReadySetGo, ShakeNDry, and Yachtride, along with the standard deviations unless otherwise specified. We also use the Big Buck Bunny video, which is used in NeRV [5]. For quantitative evaluation,
neural represen-
the following metrics: peak signal-to-noise ratio (PSNR) for reconstruction quality, LPIPS [58], FLIP [2], and SSIM [50] for perceptual similarity, where all metrics evaluated in frame-wise manner and averaged over the whole video. We evaluate these metrics on different bits-per-pixel (BPP; lower is better) to evaluate the parameter efficiency, see Appendix A. for more description of the evaluation.
Implementation details. All main experiments, including baselines, are processed with a single GPU (NVIDIA V100 32GB) and 28 instances from virtual CPU (Intel Xeon Platinum 8168 CPU @ 2.70GHz), where it takes at most ~11 hours to run our method and ~~2 days to run other baselines. Moreover, we denote NVP-S and NVP-L the model with latent code dimensions of sparse positional features to be 2 and 4, respectively; see Appendix A.2 for more details.
Baselines. We compare our method with SIREN [40], and FFN [46], which are well-known signal- agnostic CNR architectures, Instant-ngp [34] for sfateoof-the-ar CNRs on compute-efficien efficienfff, and NeRV [5], which is CNR specialized for videos. For all of the baseline methods, we follow their reported experimental setups. In particular, for NeRV [5], we use two configurations provided in the official implementation: NeRV-S and NeRV-L for a small and large model, respectively. See Appendix B for a detailed description of baseline methods.
4.1 Main results
Figure 1, Figure 3, and Table summarize quantitative and qualitative results of NVP and baselines. Remarkably, as shown in Figure 1, NVP can accurately capture dynamic motions and high-frequency details of a video, e.g., the legs of running horse, while prior state-of-the-art CNRs fail to achieve and show a blurry artifact. We also emphasize such a high-quality encoding is accomplished in "less than minute". which supports the superior compute-efficiency of NVP in training.
Moreover, Table I verifies the effectiveness NVP with quantitative evaluations, which outperforms all other baselines at varying encoding times from ~5 minutes to >40 hours. In particular, NVP significantly improves LPIPS compared with previous methods, demonstrating how the encoded videos are perceptually similar to ground-trutt videos. Such result is also confirmed in Figure 3; NeRV shows distortion that some pixels significantly deviate from the ground outputs, while our method does not suffer from such artifacts. We also note that the variance of NVP is relatively small compared with other baselines, which shows the robustness of videos with diverse scenes and motion. See Appendix C and D for video-wise results and discussion on decoding time, respectively.
7

---

Table Quantitative interpolation result of different methods on Big Buck Bunny measured with PSNR, LPIPS, and SSIM metrics. Bold indicates the best result.
Metric
NeRV [5]
NVP (ours)
Original
Mask
Inpainting Result
PSNR (1)
23.05
33.76
LPIPS (
0.480
0.311
Figure 4: Video inpainting result of NVP on the drift-chicane video in DAVIS 2017 [38] to remove the masked car.
SSIM (1)
0.690
0.960
IL354
NNP(ous)
0.6
0.2
Po99 31
LPTS
0.4
Nearest
Billnnar
0.8
0.2
52
0.50 Bla placi (BPP)
0.25
0.1
0.3
0.3
0.20
0.50
0.25
0.50
iioiturk
Bils per piuel (3FF))
Bits piael (3FFP)
BiG
plaxt (3FF)
(A) SlakeNDey
(b) Boauty
Bicubic
NVP (ours)
Figure PSNR and LPIPS values NVP and well-known video codecs over different BPP values computed on (a) the ShakeNDry video and (b) the Beauty video in UVG-HD.
Figure 5: Super-resolution result (x8) of NVP (HoneyBee).
4.2 Applications
In this section, we provide several applications of our method, NVP, as video CNRs. For better, playable illustrations and qualitative results, please refer to our project page.
Video inpainting. Intriguingly, our method has the capability video inpainting, i.e., the desired moving object in the video be naturally removed by capturing shared video contents with learnable keyframes. Figure 4 visualizes the illustration of inpainting results from NVP; as shown in this figure, one the car is removed where such parts are filled with natural background.
Video frame interpolation. Since video CNRs approximate videos as temporally continuous signals, they should interpolate among two different frames at an arbitrary time, even if such frame does not exist in training dataset. To validate the interpolation capability of NVP, we provide quantitative results Big Buck Bunny sequences; our method shows better interpolation results measured with various metrics, such as PSNR and LPIPS.
Video super-resolution. We remark that NVP encodes a given video as spatio-temporall continu- ous signal. Thus, our method can interpolate the frames across spatial directions, i.e., frame-wise super-resolution. Figure 5 exhibits how well NVP smoothly interpolates video frames across spatial directions while preserving the sharp edges, compared with naïve upsampling methods.
Video compression. Recall that one of the major advantages of CNRs is their succinct encoding to represent given signal; one may consider utilizing CNRs for video compression [5, 11, 12). To verify the potential of our method on video compression, we compare the quality of compressed videos from NVP with the ones from the current state-of-the-ar video codecs. As shown Figure compressed videos from NVP show the comparable reconstruction quality (measured with PSNR metrics) while outperforming perceptual similarity (measured with LPIPS metrics).
4.3 Ablation studies
Effect of architecture components. To verify the effectiveness of each component, we train our model with all the videos UVG-HD by removing each component while maintaining the total number of parameters, then measure PSNR metrics from these models. Table 3 summarizes the effect of three different architecture components. Without any of the components consisting of our positional features, the reconstruction quality gets dramatically worse, which validates how NVP

---

Table 3: PSNR values of each component of NVP: learnable keyframes, sparse feature, and modula- tion at 1,500 (1.5K) and 150,000 (150K) iterations. Bold indicates the scores within one standard devi- ation from the highest average score.
PSNR
30-
0.2
0.4
0.6
0.8
Keyframes Sparse feat.
Module. Params.
1.5K
150K
Bits per pixel (BPP)
136M
29.95
31.21
29.88
32.442448
Figure 7: Rate-distortion plot of different
X
147M
32. a5x
38.04:32
/
/
136M
34,85 1..
38.89 ..11
compression strategics on ReadySetGo.
37.5
37
35.0
PSNR
PSNR
36-
32.5
35-
30.0
XVP. 3PP 6.396
34
27.5
NVP wio Day Uyt
1L564 3FF-6.576
500
1000
1500
2000
200
400
600
Iterations
Frame index
Figure 8: Convergence plot of NVP under differ- ent keyframe choices on the Jockey video.
Figure 9: Frame-wise encoding quality of NVP and existing video codecs on Beauty.
succinetly encodes given video as latent codes. We also note that the modulation not only improves the final encoding quality but also notably achieves high-quality encoding in its early training epochs.
Compression procedure. To validate the effectiveness of our compression scheme, compare the encoding quality of (a) NVP with our compression scheme and (b) NVP compressed through based pruning. Figure shows the proposed compression pipeline outperforms conven- pruning under various BPP values. We also remark that our compression method does not require re-training and is thus much more time-efficient.
magnitude- basedd
tional magnitude-
Analysis of non-temporal keyframes. Recall that we additionally design keyframes across spatial directions, unlike conventional approaches that designate the keyframe
only the temporal direction. To validate the effect of such keyframes, we compare NVP with (a) NVP without spatial keyframes and (b) NVP without all of the keyframes in Figure 8, where the number of parameters is equally set for a fair comparison. While utilizing latent learnable keyframes only over temporal direction is already fairly effective for high-quality encoding, one observe the consideration of keyframes across other directions provides a further improvement.
Consistent frame-wise encoding quality. Existing keyframe- based compression approaches often suffer from inconsistent encoding quality: the frame-wise quality of the compressed video highly depends whether it is the designated keyframe not. In contrast, NVP learns the keyframes and does not have this problem. Figure validates the result: our method exhibits consistent encoding performance while conventional popular video codecs show several peaks that the reconstruction quality (measured with PSNR metrics) highly deviates from others.
Effect of the concatenation of latent codes in sparse positional features. To validate the effective- ness of design choice on extracting latent features from sparse positional features U we compare the reconstructions from NVP with and without concatenation of Uo As shown in Figure 10, the concatenation of latent codes Uijk in Ug.,. indeed mitigates non non-smooth transitions between latent codes and captures sharp details in given video better. In particular, without the concatenation, it results in undesirable artifacts (e.g., showing discontinuous borders), validating our concatenation scheme for constructing latent representations from sparse positional features.
Effect of upsampling sparse positional features. We also examine the effect of upsampling of the sparse positional features U Figure 11 shows the result: linear interpolation of Ua.,. exhibits more smooth patterns for unseen coordinates during training. Meanwhile, we note that the upsampling requires 1.61 times more training time per iteration due to the additional computation bottleneck (see Table 4); however, regardless of upsampling, we remark that NVP still achieves notable compute- efficiency compared with prior state-of-the-ar methods (such Chen al. [5]).

---

Without Concatenation Us..
With Concatenation of Us...
Ground Truth
Figure 10: Reconstruction results on ReadySetGo in UVG-HD. Concatenation of sparse positional features captures sharp details (e.g., a fence) better.
Table 4: Training time per iteration with/without up- sampling of sparse posi- tional features.
Upsampling
Time
Upsampling Us.,
Upeampling U....
X
0.291s
0.469s
Figure Comparison of super-resolution result (x8) on HoneyBee.
Discussion and conclusion
We proposed NVP, a new coordinate- neural representation (CNR) to encode videos as suceinet latent codes. Our main idea is decompose video into "image-like" and "video-like" structures to learn coordinate-to-Ban
mapping efficiently. Extensive experiments have verified the effectiveness of NVP on all the paaametrrccompuue-effciceeee and the encoding quality. We hope our method will facilitate various future research directions the CNR area.
Limitations and future works. Each video contains different scenes and motions so that it can be either static or dynamic, yet we utilize the same hyperparameters and architectures for encoding any video. Although such video-agnostic design is fairly effective and outperforms prior works, we believe the video-wise consideration of the architecture and hyperparameter can remarkably boost the performance further. Moreover, we have shown the potential of utilizing powerful image and video codes for compressing latent codes NVP; extending such codecs be specialized for the compression of latent codes should be an interesting direction.
Negative social impacts. A side effect CNRs is their potential unexpected behavior on encoding: they may cause undesirable artifacts in representing the given signal but are challenging to predict due to the under-explored behavior of training CNRs. Furthermore, in the case of representing videos, the encoded videos may suffer from severe distortions and conceivably cause ethical problems. In this respect, such behaviors should be extensively and carefully investigated and mitigated exploit CNRs as the standard for encoding videos in real-world situations.
Acknowledgments and Disclosure of Funding
We would like thank Younggyo Seo, Jihoon Tack, Jongheon Jeong, Sukmin Yun, Jongjin Park, Junsu Kim, Seong Hyeon Park, Seojin Kim, Changyeon Kim, Jaehyun Nam, and anonymous reviewers for their helpful feedbacks and discussions. We also appreciate Max Ehrlich for providing the exact results of prior video compression methods.
This work was mainly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) NN.22021-0-02666 Artificial Intelligence Innovation Hub; No. 20.2009---0005. Artificial Intelligence Graduate School Program (KAIST); No.2019-00 9-96001906, Artificial Intelligence Graduate School Program (POSTECH). This research was partly supported by the Challengeable Future Defense Technology Research and Development Program (No.91502)) (50.91502720) of Agency for Defense Development in 2022. This work was partially supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. 2No.2022RI FFIII111117777
10

---

References
[1] E. Agustsson, D. Minnen, N. Johnston, J. Balle, J. Hwang, and G. Toderici. Seale-space flow for end- to-end optimized video compression. IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[2] P. Andersson, J. Nilsson, T. Akenine-Möller, M. Oskarsson, K. Aström, and M. D. Fairchild. FLIP: Difference Evaluator for Alternating Images. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 3(2):15:1-15:23 2020.
[3] Barron, B. Mildenhall, M. Tancik, Hedman, Martin Bartin-aa and P. P. Srinivasan. Mip-NeRF: multiscale representation for anti- anti-aliasing neural radiance fields. In IEEE International Conference on Computer Vision, 2021.
[4] R. Chabra, E. Lenssen, E. llg, T. Schmidt, J. Straub, S. Lovegrove, and R. Newcombe. Deep local shapes: Learning local SDF priors for detailed 3d reconstruction. In European Conference on Computer Vision, 2020.
[5] H. Chen, B. He, H. Wang, Y. Ren, S. N. Lim, and A. Shrivastava. NeRV: Neural representations for videos. In Advances in Neural Information Processing Systems, volume 34, 2021.
[6] Y. Chen, S. Liu, and X. Wang. Learning continuous image representation with local implicit image function. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[7] Z. Chen, Chen, Liu, Xu, Goel, Wang, Shi, and X. Wang. VideoINR: Learning video implicit neural representation for continuous space-time super-resolution. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[8] J. Chibane, T. Alldieck, and G. Pons-Moll. Implicit functions in feature space for 3D shape reconstruction and completion. In JEEE Conference on Computer Vision and Pattern Recognition, 2020.
[9] S.-F. Chng. S. Ramasinghe, Sherrah, and S. Lucey. GARF: Gaussian activated radiance fields for high fidelity reconstruction and pose estimation. arXiv e-prints, 2022.
[10] Deng, W. Dong, Socher, L.-J. Li, Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, 2009.
[11] E. Dupont, A. Golinski, M. Alizadeh, W. Teh, and A. Doucet. COIN: COmpression with Implicit Neural representations. In ICLR Workshop on Neural Compression: From Information Theory to Applications. 2021.
[12] F. Dupont, Loya, M. Alizadeh, A. Golinski, Y. W. Teh, and A. Doucet. COIN+ Data agnostic neural compression. arXiv:2201. /2904, 2022.
[13] R. Fathony, A. K. Sahu, D. Willmott, and J. Z. Kolter, Multiplicative filter networks. In International Conference on Learning Representations, 2021.
[14] Gu, Liu, P. Wang, and C. Theobalt. StyleNeRF: A style-based 3d aware generator for high-resolution image synthesis. In International Conference on Learning Representations, 2022.
[15] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and Debevec. Baking neural radiance fields for real-time view synthesis. In IEFE International Conference on Computer Vision, 2021.
[16] A. Hore and D. Ziou. Image quality metrics: PSNR vs. SSIM. In International Conference Pattern Recognition, 2010.
[17] Z. Huang, S. Bai, and J. Z. Kolter. Implicit:: Implicit layers for implicit representations, In Advances in Neural Information Processing Systems, 2021.
[18] A. Jain, M. Tancik, and P. Abbeel. Putting NeRF on diet: Semantically consistent few-shot view synthesis. In IEFE International Conference on Computer Vision, 2021.
[19] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nießner, T. Funkhouser, al. Local implicit grid representations for 3D scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012.
[21] J. Lee. J. Tack, N. Lee, and J. Shin. Meta- learning sparse implicit neural representations. In Advances in Neural Information Processing Systems, 2021.
11

---

[22] T. Li. M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele. and Z. Lv. Neural 3D video synthesis from multi-view video. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[23] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space- time view synthesis of dynamic scenes. In IEEE Conference on Computer Vision and Pattern Recognition. 2021.
[24] C. H. Lin, H.-Y. Lee, Y.-C. Cheng. S. Tulyakov, and M.-H. Yang. InfinityGAN: Towards infinite-pixel image synthesis. In International Conference on Learning Representations, 2022.
[25] J. Liu. S. Wang, W.-C. Ma, M. Shah, R. Hu, P. Dhawan, and R. Urtasun. Conditional entropy coding for efficient video compression. In European Conference on Computer Vision. Springer, 2020.
[26] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. In Advances in Neural Information Processing Systems, volume 33, 2020.
[27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Reprexentations, 2019.
[28] G. Lu, W. Ouyang, D. Xu, X. Zhang, C. Cai, and 7. Gao. Dve: An end-to-end deep video compression framework, IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[29] N. Martel, D. Lindell, C. Lin, R. Chan, Monteiro, and G. Wetzstein. ACORN: Adaptive coordinate networks for neural representation. In ACM Trans. Graph. (SIGGRAPH), 2021.
[30] R. Martin-Brualla N. Radwan, M. S. Sajjadi, T. Barron, Dosovitskiy, and D. Duckworth. NeRF in the wild: Neural radiance fields for unconstrained photo collections. IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[31] Mehta, M. Gharbi, C. Barnes, Shechtman, R. Ramamoorthi, and M. Chandraker. Modulated periodic activations for generalizable local functional representations. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[32] A. Mercat, M. Viitanen, and J. Vanne. UVG dataset: 50/120fps 4K sequences for vidco codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, MMSys '20, page 297-302, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450368452. doi: 10.1145/3339825.33999
[33] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In European Conference on Computer Vision, 2020."
[34] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. Graph., (444:102:11102:: July 2022. doi: 10. 10.11455/5282233331127
[35] K. Park, U. Sinha, J. Barron, S. Bouaziz, D. Goldman, M. Seitz, and Martin Brrulla Deformable neural radiance fields. In IEEE International Conference on Computer Vision, 2021.
[36] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chaman, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[37] S. Peng, M. Niemeyer, L. Mescheder, M. Pallefeys, and A. Geiger. Convolutional occupancy networks. In European Conference on Computer Vision, 2020.
[38] Pont-Tuset, F. Perazzi, S. Caelles, P. Arbelácz, A. Sorkine- Hornung, and L. Van Gool. The 2017 DAVIS challenge on vidco object segmentation. 304::1704.00677 2017.
[39] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer D-NeRF: Neural radiance fields for dynamic scenes. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[40] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. Implicit neural representations with periodic activation functions. In Advances in Neural Information Processing Systems, 2020.
[41] I. Skorokhodov. S. Ignatyev, and M. Elhoseiny. Adversarial generation of continuous images. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[42] 1. Skorokhodov, S. Tulyakov, and M. Elhoseiny. StyleGAN V: A continuous video generator with the price, image quality and perks of StyleGAN2. In IEEE Conference on Computer Vision and Pattern Recognition. 2022.
12

---

[43] G. J. Sullivan, J.-R. Ohm. W.-J. Han, and T. Wiegand. Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on Circuits and Systems for Video Technology. 66462) 2012. doi: 10.11099TSST22022222111.
[44] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[45] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level detail: Real-timg time rendering with implicit shapes. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[46] Tancik, P. P. Srinivasan, Mildenhall, S. Fridovich Keil, N. Raghavan, U. Singhal. R. Ramamoorthi. J. T. Barron, and R. Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, 2020.
[47] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar. Block-NeRE Scalable large scene neural view synthesis. arXiv:2202.0522 052663, 2022.
[48] S. Tomar. Converting video formats with ffmpeg. Linux journal, (006(146):1)))) 2006.
[49] G. K. Wallace. The JPEG still picture compression standard, IEEE transactions on consumer electronics. 38( Ixxiiiiixxxx,, 1992.
[50] Z. Wang, C. Bovik, H. R. Sheikh, and E, P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(())600-612, 2004.
[51] T. Wiegand, G. Sullivan, G. Bjontegaard, and Luthra. Overview of the H.264//VCC video coding standard. IEEE Transactions on Circuits and Syxtems for Video Technology. 13(7):560-576, 2003. doi: 10110//CSVTT.200..0 115115.
[52] W. Xian, J.-B. Huang. J. Kopf, and C. Kim. Space-time neural irradiance fields for free-viewpoint video. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
[53] B. Xu, N. Wang. T. Chen, and M. Li. Empirical evaluation of rectified activations in convolutional network. arXiv: 1505,00853, 2015.
[54] D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang. SinNeRF: Training neural radiance fields on complex scenes from a single image. arXiv:2204.00 00928, 2022.
[55] R. Yang, F. Mentzer, I., V. Gool, and R. Timofte. Learning for video compression with hierarchical quality and recurrent enhancement. In JEEE Conference Computer Vision and Puttern Recognition, 2020.
[56] R. Yang, Y. Yang. J. Marino, and S. Mandt. Hicrarchical autoregressive modeling for neural vidco compression. 20Xiv:2010.10258 2020.
[57] S. Yu, J. Tack, S. Mo, H. Kim, J. Kim, J.-W. Ha, and J. Shin. Generating videos with dynamics-aware implicit generative adversarial networks. In International Conference on Learning Representations, 2022.
[58] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unrcasonable effectiveness of deep features as perceptual metric. In IEEE Conference Computer Vision and Pattern Recognition, 2018.
[59] Y. Zhang, T. van Rozendaal, J. Brehmer, M. Nagel, and T. Cohen. Implicit neural vidco compression. In ICLR Workshop Deep Generative Models for Highly Structured Data, 2022.
[60] P. Zhou, L. Xie, B. Ni, and Q Tian. CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally- Independent Pixel Synthesis. arXiv:2/ 10.00988, 2021.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section
Did you include the license the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section and supplementary material.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and supplementary material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.
4. If you using cxisting assets (e.g., code, data, models) curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.
(b) Did you mention the license of the assets? [Yes] See Section 4.
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
See the supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [Yes] See the supplementary material.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "sNf6xpmXSPy", "reviewer": "Reviewer_kkNU", "summary": "The paper proposes a coordinate-based architecture for video representation. The key element of the architecture is the introduction of keyframes (positional encodings of the features). The approach is validated on the UVG benchmark showing benefits over prior art.", "questions": "How does the method scale with video size in terms of both the spatial resolution of the frame as well as the number of frames?", "limitations": "The limitations are discussed but are centered around the general idea of CNR for video representation. The discussion of the possible limitation of the model design could be extended. For example, one might expect that the keyframe idea (amortizing over the video) would not work well for long videos composed of multiple possibly dynamic scenes.\n\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is in general well-structured and well written. The model design choices are motivated, and the reported results looks solid. The idea is moderately original and the quality of the work is good enough to recommend the acceptance. However,  he clarity of the presentation could be improved and the authors could highlight better the significance of the work. See detailed comments and questions below.\n\n**Abstract:**\n- Abstract could be strengthened to reflect better the paper contribution and its importance.\n\n**Introduction:**\n- The introductory section positions well the work w.r.t prior art.\n- Introduction does not motivate why research on CNRs for video representation is an important research avenue. Adding such motivation would make the paper stronger.\n- The connection between learnable positional features and keyframes is now well exposed in the intro and in the abstract. It is hard to understand why keyframes could play the role of learnable positional features.\n- Fig 2 could be a bit more informative; the key element of the paper seems to be the construction of the keyframes however, the figure does not give details on how the keyframes are constructed.\n- L24 “… numerous appealing properties …“- only one property is mentioned.\n- L36 “… while enjoying lots of intriguing properties …” – could the authors enumerate the properties\n- L71 “… 34.43 in 5 minutes …” – can the authors add details here? What is the hardware? How is the time measured?\n\n\n\n**Methodology:**\n- In general methodology section is well written and easy to follow. One exception is the construction of the keyframes. This part could benefit from some re-writing. Also, adding a figure explaining the process could ease the understanding. Moreover, it is unclear why U_l is represented as a set.\n- Did the authors consider different types of or designs of keyframes?\n- Modulation, in L174 the authors write: “… we found such a simple MLP architecture lacks an expressive power and fails to capture the complex dynamics… “. However, based on ablation study the benefit of modulation seems marginal (in terms of PSNR). Could the authors comment on this?\n\n**Experiments:**\n- The ablation table is missing std, could the std information be added to the ablation table?\n- Adding additional dataset would make the validation and observations stronger.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dqgAN-6_b_", "reviewer": "Reviewer_MBhC", "summary": "This paper proposes a new CNR framework by introducing “learnable positional features” and decomposing the video into 2D and 3D latent representations. Besides, a compute- and memory-efficient compression procedure is introduced to further reduce the parameters and training cost. Experiments are conducted on UVG-HD benchmark for evaluating video encodings.", "questions": "Here are some concerns to be addressed:\n\n1. My major concern about this work is the experiment results shown in Table. 1, in which the experimental setting is different from NeRV. Maybe the results on “Big Buck Bunny” sequence should be given to present a more intuitive comparison.\n2. Also in Table. 1, why is instant-NGP not compared after encoding time > 15hours?\n3. The results of video compression evaluation should be more detailed. Figure. 4 only shows the results on ShakeNDry and Beauty. More comprehensive experimental results should be presented.\n4. Lack of comparison of decoding time.\n5. Why is the grid size selected as H×W×S? It is better to show the influence of different grid sizes.\n6. More experiments comparing with traditional coding frameworks such as VVC and HEVC are suggested.\n", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \nThis work proposes a new CNR framework to solve the dilemma of quality and efficiency in video encoding. The experimental results have been improved on both PSNR metrics and parameter-efficiency. The overall clarity of the paper is good.\n\nWeakness: \nThis work is mainly built based on the existing CNR framework, but the overall originality is not high. The experimental part is not sufficient, and some experimental settings are also unfair to some extent.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DAFO2_y9QS", "reviewer": "Reviewer_LDYR", "summary": "The authors proposed a scalable implicit neural network for reconstructing video signals in a parameter-efficient and computationally efficient manner. In order to do this, they proposed learnable positional features inspired by the existing work in the literature (Instant-ngp). In general, implicit neural network maps from input co-ordinates to RGB pixels directly using one mapping function, here the authors maps the pixel co-ordinates to intermediate latent space, and from latent space to RGB pixels using another mapping function with modulation function. In the latent space, the authors learn the key frames of the given video in the latent grids, for xy, xt, yt, xyt. These are compressed using existing standard codecs HEVC and JPEG, and interpolation is used to find the latent vectors for any given (x,y,t), and then latent to RGB mapping function. The authors show their proposed method trains faster, and reconstructs the video better in a parameter efficient way.", "questions": "1) In learning latent keyframes, what kind of information do the keyframes contains, and in terms of visualization how does it look like?. does it learn image like structure?\n2) how the BPP is calculated? whether the weights of the network are used in full precision or half precision to compute the BPP. In the total filesize, how much information is from latent grids, and how much information is from the latent to RGB mapping function.\n3) I visualized a few videos in the dataset used in the paper, the temporal motion in the videos is not much dynamic, it's almost static. Maybe this is the reason, the method is able to have good reconstruction results. I suspect whether the proposed might have a similar performance with high temporal variations.\n ", "limitations": "the limitations are discussed in the paper", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper proposes the parameter and computational efficient implicit neural network to reconstruct videos from pixel coordinates. The method has the advantage of learning keyframes automatically from the videos, whereas the other compression methods do not have the capability to learn the keyframes.\nLearns the grid representations in the latent space for different directions (xy, xt, yt, xyt). The authors claim that it improves parameter efficiency of Neural video representation (NVR)\n\nWeakness:\nFor the compression of the latent spatial grids, the HEVC and JPEG compression methods are used. For the video compression applications, the combination of traditional codecs with learning-based compression is not interesting in my opinion. The traditional codecs are bound to have artifacts that the proposed method incorporates into the proposed method \nThe performance of the proposed method are below the HEVC and H264, and the datasets used to evaluate the proposed method is not a good benchmark. I see that in the honeybee video, the video is almost static and it is difficult to conclude whether the proposed method can capture the dynamic motions in the temporal dimensions.\nThe proposed method is not compared with other state-of-the-art methods in learning-based video compression methods.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept