PNeRV: A Polynomial Neural Representation for Videos
Authors: Sonam Gupta
Snehal Singh Tomar
Grigorios Chrysos
Sukhendu Das
Rajagopalan N Ambasamduram
Venue: Accepted by TMLR ()

Abstract: Extracting Implicit Neural Representations (INRs) on video data poses unique challenges due to the additional temporal dimension. In the context of videos, INRs have predominantly relied on a frame-only parameterization, which sacrifices the spatiotemporal continuity observed in pixel-level (spatial) representations. To mitigate this, we introduce Polynomial Neural Representation for Videos (PNeRV), a parameter-wise efficient, patch-wise INR for videos that preserves spatiotemporal continuity. PNeRV leverages the modeling capabilities of Polynomial Neural Networks to perform the modulation of a continuous spatial (patch) signal with a continuous time (frame) signal. We further propose a custom Hierarchical Patch-wise Spatial Sampling Scheme that ensures spatial continuity while retaining parameter efficiency. We also employ a carefully designed Positional Embedding methodology to further enhance PNeRV's performance. Our extensive experimentation demonstrates that PNeRV outperforms the baselines in conventional Implicit Neural Representation tasks like compression along with downstream applications that require spatiotemporal continuity in the underlying representation. PNeRV not only addresses the challenges posed by video data in the realm of INRs but also opens new avenues for advanced video processing and analysis.

Published in Transactions on Machine Learning Research (02/2024)
PNeRV: A Polynomial Neural Representation for Videos
Sonam Gupta
es @s18800050csee.immmooo
Deportment of Computer Science Engineering, IIT Madras
Snchal Singh Tomar
mehalstomarrlgmaate com
Department of Electrical Enginecring, IIT Madros
Grigorios G Chrysos University of WVsconsinn-Madinn
chrysos@ruisc.ett edu
Sukhendu Das
sdas@üimm.. oc
Department of Computer Science Engineering, IIT Madras
A. N. Rajagopalan Department of Electrical Engineering, ITT Madros
rajuBee. mjjuee..itm.com oc
Reviewed on OpenReview: https: opw.reviee.e....e... net/ forum? id= oCBs=Cov2g
Abstract
Extracting Implicit Neural Representations (INRs) video data poses unique challenges due to the additional temporal dimension. In the context of vidcos, INRs have prodominantly relied on a frame-only parameterization, which sacrifices the spatiotemporal continuity ob- served in pixel-level (spatial) representations. To mitigate this, we introduce Polynomial Neural Representation for Videos (PNeRV), a parameter-wise efficient, patch-wise INR for videos that preserves spatiotemporal continuity. PNcRV leverages the modcling capabilities of Polynomial Neural Networks to perform the modulation of continuous spatial (patch) signal with continuous time (frame) signal. We further propose a custom Hierarchical Patch-wise Spatial Sampling Scheme that ensures spatial continuity while retaining param- eter efficiency. We also employ a carefully designed Positional Embedding methodology to further enhance PNeRV's performance. Our extensive experimentation demonstrates that PNeRV outperforms the baselines in conventional Implicit Neural Representation tasks like compression along with downst applications that require spatiotemporal continuity in the underlying representation. PNeRV not only addresses the challenges posed by video data in the realm of INRs but also opens avenues for advanced video processing and analysis.
1 Introduction
Implicit Neural Representations (INRs) have become the paradigm choice for modelling discrete sig- nals such as images and videos using a continuous and differentiable neural network, for instance, a multi layered perceptron. They facilitate several important applications like super-resolution, inpaint- ing, and denoising (Niemeyer al.) 2019 Park al.) 2021 Pumarola et al. [202]] Tretschk al., 2021 Xian et al.] 2021: Li et al.] 2021 Du al.) 2021] for images. They offer various important ben- efits over discrete representations particularly in terms of them being agnostic to resolution. Recent advancements have extended INR to video signals, but early methods relied on utilizing 3 dimensional spatiotemporal coordinates (I, as input and RGB values as outputs. Such straightforward exten- sions of INRs to vidcos are inefficient during inforence since they noed to sample x H x W times reconstruct. the entire video. For high resolution videos, this behavior becomes more prominent.

---

In this work, we enhance INRs for videos along the following three directions. Firstly, we adopt temporal as well as spatial parameterization (illustrated in Fig. [(]) in our light-weight representation. We achieve this by replacing the dense pixel-wise spatial sampling with carefully designed Hierarchical Patch-wise Spatial Sampling approach. Our scheme (elaborated upon in section 3.1 breaks video frame into patches and samples coordinates from sub-patches in recursive fashion across different levels of hicrarchy. Secondly, we leverage the properties of PNNs to build a parameler-wise efficient. decoder backbone that yields better metric performance. PNeRV also inherits some important properties of PNNs such as robustness to the choice of non-lincar activation functions. Finally, we improve the positional embedding of input signals to align well with our PNN backbone and achieve peak metrie performance. Our claims are backed by consistent qualitative and quantitative results on video reconstruction and four challenging downstream tasks i.e. Video Frame Interpolation, and Denoising. The key contributions of this paper Published in Transactions on Machine Learning Research (02/2024)
Also, simple multi layered perceptron is unable to model the complex spatio-temporal relationship in video pixels well. To address this issue and maintain parameter efficiency, current seate-of-tho-art meth- ods in the field use a framee-nly parameterivation as depicted in Fig- (a) and (b). These representations take the time index of a frame as input and predicts output. Although state-of- the- art INRs on video data exhibit impressive results on tasks such as video denoising and compression, they sulfer from two fundamental issues. Firstly, the lack of spatial parameterization renders the representa- tion less suitable for conventional INR applications such as video super-resolution Secondly, they are not equipped to capture the information pertaining to pixel-wise auto and cross correlations across time explicitly. Hence, resulting in a suboptimal metric performance to model size ratio. Only recently, |Sen et al. [022) have attempted to explore a spatiotem- porally y continuous neural representation based hy- pernetwork for generating videos. However, their approach and the tasks they enable are fundamen- tally differen:]o ours.
(a)
the entire frame
r(t)
71%)
r(
((P)
P
04
(
Figure 1: PNeRV when compared to its coun- terparts: (a) NeRV: An INR for videos with only frame-wise parameterization that leads to loss of spa- tial continuity. (b) E-NeRV: A step-up over NeRV with a parameterization that employs fixed Spatial Con- text (SC). The fixed SC does not support spatial con- timuity. (c) PNeRV: An efficient INR for videos with a PNN backbone (signified by the usage of Hadamard Product @) that supports varying SC while retaining spatial continuity.
We utilize the following key insights to build a spatiotemporally continuous Neural Representation while keeping the model size in check: (1) Achieving spatiotemporal continuity doesn't always require dense per- pixel sampling. A well-designed patch-wise sampling approach (Tretschk et al. 2020 Yuval Nirkin 2021 can yield comparable results for downstream tasks while processing less data. (2) To achieve better efficiency in handling gigher-dimensionll inputs with fewer learnable parameters and maintaining performance, we
consider using Polynomial Neural Networks (PNNs) (Chrysos et al., 2021b 2019 our preferred function approximator. PNNs model the auto and cross correlations within their inpul feature maps. (3) We also
propose a Positional Embedding (PE) methodology to aid the PNN backbone in learning a faithful repre-
sentation using the sampled inputs. Carefully designed PEs (Vaswani et al. 2017 Wu et al. 2021 Deng
ct al.
2022 Sitzmann et al. 2020b are proven to boost the performance of Deep Neural Networks.
Compression, can be summarized as:
1. We introduce a Hierarchical Patch-wise Spatial Sampling approach in our formulation which makes PNcRV continuous in space and time while retaining parameter efficiency.
2. We design PNN for temporal signals. We build a Higher order Multiplicative Fusion (HMF) module that learns parametric embedding.
We highlight these differences section 2
2

---

Published in Transactions on Machine Learning Research (02/2024)
Posiiionl Emtledlnng
A13 12
Embadiing Fusion Block
IMR Decoder
(R
(thwa)
& Par
R+
HMF
Ag
PPE
Output
Ras222
Comoeiaton
Figure 2: The PNeRV Architecture: The PNeRV pipeline consists of three modules. First, the PEs of time index t, coarse patch coordinate Xij and the fine patch coordinate Aú computed in the Positional Embedding Module. Second, these embeddings are fused effectively in the Embedding Fusion Block. Finally the PNN-based INR decoder reconstructs the frame patch, given a fused Positional Embedding 2, Here FC denotes a fully connected layer of appropriate input-output dimcnsions.
3. We propose new positional embedding scheme to encode and fuse spatial and temporal signals. The scheme brings together both parametric (learnable) and functional (deterministic) embeddings, a first in Neural Representations for videos. We show that both the embeddings complement each other to align well with the PNN based backbone and altain peak metrie performance.
2 Related Work
Implicit Neural Representations. INR is a method to convert conventionally discrete signal represen- tations such as images (discrete in space) and videos (discrete in space and time) into continuous represen- tations. Originally motivated as an alternate representation for images (Park et al. 2019
Mescheder et al. 2019 Chen & Zhang, 2019), INR has been pushing the envelope terms of performance on a wide array of tasks images such denoising and compression (Zhu et al. 2022 Huang et al.
2022 Li et al. 2022a
Chen et al., 2022a). INR for videos extends INR for images by simple reparameterization in terms of
rideo-framce indices as well [Nemmeyer et al., 2019
Park et al.] 2021 Pumarola et al. 2021 Tretschk et al.
2021 Xian et al. 2021 Li et al.) 2021 Du et al., 2021 Chen et al. 2022b Saragadam et al.) 2022 Mai
& Liu 2022). The approach of choice for such architectures entails learning an embodding for pixels and timestamps, which are passed on to decoder network. To expedite model training and inference with large video tensors in such INR formulations, state-of-the-are literature in INR for videos (Chen al. 2021 Li
et al. 2022b Chen al. 2023) has introduced parameterization over frame indices only. While such for- mulations are lighter and faster, they compromise spatial continuity. We aim to bring the best of both these formulations together in this work by employing parameterization over patches as well as frame indices, with a PNN backbone. Consequentially, the spatial continuity achieved while keeping model parameters in chock, an essential attribute for a faithful INR and is critical for applications such as super-rosolution.
(Sn al. 2022 have recently attempted to build a spatiotemporally continuous INR based hypernetwork for generating videos. Their proposed method differs from ours in two key aspects. First, theirs is a video generation pipeline and the INR is only a component of their model. Whereas ours is vanilla INR that serves an alternate representation for videos while enabling interesting downstream tasks. Second, since their model is hypernetwork, it is not well equipped to tackle high resolution videos such as the ones found in the UVG dataset (Mercat et al. 2020 The authors attribute this behaviour to the unstable training routines large hypernetworks.
Polynomial Neural Networks (PNNs). PNNs model their outputs as higher-degree polynomial of the input. A full polynomial expansion be expressed as follows (Chrysos et al. 2021b):
3

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Overview of the nomenclature used in section 3
All PEs E IR where is a hyperparameter.
Nomenelature Pertaining to Spatial Sampling (Section 3.1
Nomenclature Portaining to PFs (Section 3.2
Symbol Dimeusion
Definition
Symbol
Definition
I 3
Complete Video
Hyperparamete:: Erequency.
H x
(th frame
Hyperperanster: longth of she PE.
H x D
Global spacisl (pisel) courdiastes.
Frsa(Ag.
Fauctional embedding luse space and time.
Grid coordinate in correspanting Pay.
PNN driven fusion of Frne(f) and (eeg(Ag) INR Decoder ispu:: (Tysa (1as)) + [Ag
Ag
K x 2
Tensor with fine courdiuntes in thast correspond to Par- ( Coarse patch. x such patches are sumpled % (k.4) fine patch. L such fine patches sumpled Pg.
Pue
Ps
2
$ x
Tepe(t)
Functionul PE
Pac
MA
Nr. x3
Fpee(Ag)
Parametric embedding of Asj
x == ==o 222 + X2 +b.
(1)
where, F; 2, G, and b represent the output, input vector, non-linear activation and bias. Wi represents the weight tensor for the 7th order, and X; represents the mode-f produc The PNN paradigm's 's elegance lies in the utilization of tensor factorization techniques to prevent exponential increase in model parameters with an increase in the polynomial order. We examine only the Nested Coupled CP Decomposition (NCP P since our model implementation is based on its sequential polynomial expansion. Considering a 3rd order polynomial governed by Eq. [ the decomposed forward pass can be expressed as the following recursive relationship:
I, (A] @ (S% b(n))
(2)
for n E (2,3]. Here x = Cxg + is the output of the 3rd order polynomial, @ represents Hadamard product and x1 = (A/ ) @ bp:). The learnable parameters in this setup are C E Book An] E ma Sin E R Br) E Rexk, and bld] E I, and q E Rº The symbols d, o, E, and k represent the docomposition's input dimensions, output dimensions, implicit dimension, and rank. The rise of PNNs has seen their application to array of important deep learning regimes such as generative models (Chrysos let al. 2021b Choraria et al. 2022, Singh et al. 2023), attention mechanisms (Babiloni et al.) 2021 and classification models (Chrysos et al. 2022a by 2023 Chen et al. 2024). However, their direct application to temporal signals has not emerged, and they have only been used in a single variable sctup in unconditional modeling regimes. PNeRV builds along these new directions in its INR decoder and HMF.
Rich Positional Embeddings. PEs based on a series of sinusoidal functions much like the Fourier series, have become an integral part of INRs. Several works (Mildenhall et. al. 2021 Sitzmann et al. 2020a Tancik
et al. 2020) have shown that in the absence of such embeddings, the output of the INR is blurry i.e. misses the high frequency information. Thus, PEs enable INRs to capture finc-details of a signal making them indispensable for image applications (Wu et al.) 2021 Deng et al. 2022 Skorokhodov et al. 2021). INR methods for videos have also sought to capitalize upon the advantages of an efficient PE (Sitzmann et al. 2020b; Mai & Liu 2022). However, state-of-the-art in the domain (Li et al.) 2022bb Chen et al. 2021) has only explored functional (deterministic) embeddings in one input variable. In contrast, PNeRV employs both parametric (learnable) and functional embeddings. We also introduce PNN based fusion strategy combine the functional and parametric embeddings.
3 PNeRV: Polynomial Neural Representation for Videos
Overview: Let us now introduce our method. The notation and definitions for the various elements used in this section is summarized in Table D We denote tensors by calligraphic letters, matrices by uppercase boldface letters and vectors by lowercase boldface letters. To enable spatial continuity while keeping the model size in check, we propose Hicrarchical Patch-wise Spatial Sampling approach for the input coordinates.
As shown in Fig. 2 the PNeRV architecture comprises three key components, namely, a Positional Embed- ding Module, an Embedding Fusion Block, and the PNN-based INR decoder. Each frame U2 in an input
Defined in appendix A.2
Definition adopted from Chrysos et 2021b
4

---

Published in Transactions on Machine Learning Research (02/2024)
video = [u is recursively divided into coarse patches and fine sub-patches. Coordinates sampled from both the patch and sub-patch instances along with their respective frame index (1) serve as inputs to the INR decoder. In nutshell, the PNeRV formulation can be represented as:
Pi- Fo(Aij. A
(3)
where, Fe denotes the complete PNeRV model (having parameters )). As defined in Table [
Aij
denotes a fine coordinate Tensor, Xij is a coarse patch coordinate, and is the frame index. We present a detailed discussion on cach of our model's constituent clements in the subsoctions that follow.
3.1 Hierarchical Patch-wise Spatial Sampling
State-of-the-ar methods Chen et al. 2023 2021
et al. (0021) have drifted away from spatial param- eterization of their representation to ensure faster inference. They resort to a temporal-only param- eterization. In contrast, PNcRV uses a spatiotem- poral parameterization whilst having fewer param- eters by employing our efficient sampling approach (depicted in Fig. 3). We observed that a pixel- wise formulation increases the computational com- plexity manifold. Hence, we opt for a hierarchical patch-wise formulation. A primitive method to sam- ple spatial patch coordinates would be to assign a scalar coordinate to each patch (similar to frame indices). However, the pitfalls of such an approach are twofold. Firstly, scalar patch indices lack spatial context. They do not convey any sense of spatial lo- calization. Secondly, PEs obtained from scalars have lower variance, which is not idcal for training. Our analysis in Table g unddrssores these pitsalls. We have designed our sampling strategy to enrich the input to our INR decoder with spatial information of the patches. Instead of associating just a scalar index to each patch, we associate cach patch Pi with a coarse 2D index Xij and a fine index Aij E Rxx22
21
80 Au
Figure 3: Hierarchical Patch-wise Spatial Sam-
pling: (a) A Global coordinate grid C with input val- ues normalized to range [0. 1] is constructed for each frame. (b) The grid is divided into M N coarse patches of equal size. For coarse patch P.j, its cen- troid is used as a 2D coordinate Xij. (c) Each coarse patch is further divided into K x L fine patches and a collection of the centroids of these smaller patches is used as the fine patch coordinate tensor Aij-
The process
of computing Aij and Aig is illustrated in Fig. 3 Like traditional INRs Niemeyer et al. (2019); Park et al.
(2021) bs Pumarola el al. (2021), we first. build global coordinate grid C of size H x D normalized to range 0, (Fig. 3 (a)). Next, each frame is divided into M X N coarse patches. The coordinates Xij for these coarse patches Ptj are found by computing their centroids (Fig. 3 (b)). Further, cach coarse Pij is divided into K x L fine sub-patches. The K x L x 2 dimensional lensor formed by the centroids of each of these sub-patches used as the fine coordinates of Pij (Fig. g (c)). It is imperative to note that, although we divide a frame into patches, the normalized coordinate values are sampled from C in all cases for computation of centroids. In effect, the manner in which the patch-coordinates are sampled our scheme is hierarchical in nature. This ensures sense of spatial locality in all patches. Intuitively, the coarse coordinate captures global context whilst the fine coordinates of patch capture the local context. Algorithm in Appendix sumunarizes hierarchical pasch-wise spltial sampling.
A.3
3.2 Positional Embedding Module
Literature on INRs (Sitzmann al.] 2020bb Tancik et al. 2020)) dictates that rich positional em- beddings (PEs) are central to the performance of INR methods. Fourier series like PEs are posi- tively correlated with the network's ability to capture the high frequency information [aacik et al. (2020). Although the field has witnessed several advances toward the development of optimal func- tional (fixed) embeddings of signals and their parametric (learnable) fusion, functional fusion and para- metric embeddings remain under explored. In this work, we exploit the combination of functional PEs, parametric PEs, functional PE fusion, and parametric PE fusion to learn superior INR for videos.
5

---

Published in Transactions on Machine Learning Research (02/2024)
We propose embedding scheme wherein per- form temporal functional embedding in i, a spa- tial embedding via functional fusion, and a para- metric (multiplicative) fusion of all PEs to yield rich spatiotemporally aware PE. We elaborate upon each of our embeddings and their parametric fusion in the sections that follow.
Topx(()
And
Ad
A Acl
3.
Trze(Ag)
Positional Encoding of Frame Index (FPE) Given frame index t, normalized between [0, 1] as input, we adopt the widely used Fourier series based positional encoding scheme similar to the ex- isting methods Chen al. (2021]: Li ct al. (022b)).
Figure 4: The HMF architecture at a glance: All linear transformation matrices represent the terms in Eq. 9 Here, denotes the Hadamard Product, ① represents feature addition, black arrows represent in- puts, and blue arrows represent the fused entities.
This embedding given as:
(t) (sin(Tv't)
(4)
where, denotes the frequency governing hyperpa- rameter and 1 governs the number of sinusoids.
Paramctric Embedding of Fine Coordinates (PPE) We employ a paramctric positional embedding scheme (PPE) to encode the spatial context available in the fine patch coordinates given by tensors Aij. The PPE block in Fig. g illustrates the same. First, Eq. g is applied to each element. of Aij to map it to 12x21 dimensional vectors. These resultant embeddings are arranged side by side in spatial order to obtain a feature map of size 1xxx1x11 Notice that each value in the K grid has a 2D coordinate value correspouding to x and y. Eq. is applied individually the and y coordinates and the resulting vectors are fused across the channel dimensions. Resulting in channel dimension of 41. To merge these fcatures we use Non-Local Block |Wang et al. (2018) followed by a linear layer. This spatially aware attention based fusion mechanism encourages weighted feature fusion between varions spatial regions where the weights are governed by the Non-Local Block. We refer to this parameterized embedding as I PPP((Ai)).
Time Aware Spatial Embedding (TSE) video can be seen as time modulated spatial signal. There- fore, ideally, the spatial positional embedding should be dependent on the frame-index (time) as well patch coordinates. To this end, we design Time Aware Spatial Embedding which inspired from Angle modulation. In analog communication, Angle Modulation refers to the technique of varying a carrier signal's 's phase in accordance with the information content of a modulating signal. The general expression for the same is given by
(t) = (cos(2o((2 + ((cos(2
(5)
where, le is the modulated signal, Amp. is the amplitude of the carrier signal, <(.) is the phase governing function. fe and fm are the frequencies of the carrier and modulated signals, respectively. We design the embedding to perform functional fusion of Xij and f. We model a video as time (t) modulated spatial signal (Aiz). The proposed embedding (denoted by TTSE) is governed by Eqs. and
TrsE(Aig; (c0)( sin( ...2==0
(6)
wherein,
sin sin(2TrAais 8a) sin sim(2m))nn sin(2nAyt)
23 fT n +
(7)
g
Our ablations (Table substantiate that functional fusion (Trsæ) complements parametric fusson of I TFPE(() and (Ap)(Ai)) boost performance.
3.3 Embedding Fusion Block
Effective fusion of all our positional embedding elements critical the performance of our method. We opt for hybrid functional and parametric fusion module to bring together the positional embeddings obtained

---

Published in Transactions on Machine Learning Research (02/2024)
via the (ppE()) and (TSE() functions. Our fusion mechanism is split over two stages. First (FPE(() and (ppe(Aij) are fused using our proposed Higher-order Multiplicative Fusion (HMF) block. Then, PrsE(Aij, t) is added to the resulting vector, resulting in new embedding that acts as input to the INR. decoder.
Higher-order Multiplicative Fusion (HMF) We introduce the HMF which a Nested-CoPP (Chrysos et al. 2021a) inspired fusion mechanism, to fuse (F)e(t) and P (Atg). As shown in Fig. E HMF entails additive fusion of the linearly transformed fusion entities to capture first-order correlations. The additive fusion blocks are followed by Hadamard product operation with the previous additive fusion output in recursive fashion for three iterations. The recursive structure ensures that cross-correlations are captured well by the fused output. The fusion in effect translates to the following recursive relationship:
In = ((A mm.) ( [FPE(t) + A
(m A (PPE(As)) 3 xn-1) + In-1,
(8)
wherein,
= A A (PPE(AA))) + AT. FFPE(t).
Here, E {2, (): 23 represents the fused embedding (output of HMF block), and @ represents Hadamard product. The learnable paramcters in HMF are A E R2xk and Am. E 22* The rank of the decomposed weight matrices k, taken to be 160. As highlighted in Chrysos et al. (2021a), the adopted approach for fusing the frame-tinestamps and patches has an advantage over a standard approach that employs concatenation followed by downsampling. In that, concatenation amounts to the additive format of fusion which fails to capture cross-terms in correlation. That is, multiplicative interactions of order 2 more are essential for capturing both auto and cross-correlations among the entities to be fused.
3.4
INR Decoder
The literature on PNNs Chrysos et al. (2021b) has shown that stacking two more polynomials in multiplicative fashion leads to desired order of the underlying polynomial with much lesser parameters. Such an approach is vermed as ProdPoly (Product of Polynomials). As defined by (Chrysos et al. 2021b), ProdPoly implementation entails the Hadamard product of outputs of sub-modules in the architecture to obtain a higher order polynomial in the input. Since the order of a polynomial is directly correlated with its modelling capabilities, the ProdPoly approach is suitable for designing our lightweight INR decoder. The proposed INR. decoder decoder as product of three polynomials. Per our formulation, the output of the y polynomial given input to the (r (r+1)* block. The advantage of such stacking is that it leads to exponential increase in order of the polynomial.
modified derivative of the ProdPoly formulation. In that, we design the TNR
Specifically, we have three ProdPoly blocks in a hierarchy. The first ProdPoly block accepts as the fused embedding z as input. The other two ProdPoly blocks take the output feature map from their preceding ProdPoly block, 0,-1 as their input (Fig- 2 Each ProdPoly block INR decoder is an adapted implemen- tation of an NCP decomposed PNN variant tailored to our modcl's requirement. The NCP-polynomial in each ProdPoly block implemented using two convolutional blocks F. The design of these blocks is inspired 1; Li et al. (2022b). Each F block entails an Adaptive Instance Normalization layer (AdaIn) Karras et al. (019)), Convolution, pixel shuffie operation and a GeLU Hendrycks & Gimpel (2016) activation layer. This operation is denoted as F(.). The AdaIn layer takes 2 as input and normalizes the feature distribution with spatio-temporal context embedded in the input vector z. Tn essence, we adapt Eq. the following, for our decoder where and A are implemented as F and $-
by Chen et al. (2021):
yrm = Frim Fy O (I Tmm) {1,2],
wherein,
yr = (Fr
(r z)
0, Mr2 is the output of hh ProdPoly block. U set of three transpose convolutional layers applied only before the first ProdPoly block to obtain 2D feature map from the input vector z. 0g is the final output
7

---

Published in Transactions on Machine Learning Research (02/2024)
Table 2: Quantitative comparisons in terms of PSNR (dB) with respect to reconstruction on the Scikit-
Bunny video and the UVG datset. PNcRV achieves state-of-the performance while maintaining signifi- cantly fewer parameters and being up 4x faster in terms of rate of convergence.
Method
Params (M).
Bunny
Beanty
Bosphorus
Bee
Jockey
SetGo
Shake
Yacht
NeRV-L
12.57
39.63
36.06
37.35
41.23
38.14
31.86
37.22
HNeRV
11.90
36.23
36.17
30.20
41.58
28.55
29.67
32.44
25.50
E-NoRV
12.49
42.87
36.72
40.06
41.74
39.35
34.68
39.32
35.58
Ours
44.90
39.8
41.86
43.98
39.84
35.82
41.37
36.93
Gain over E-NeRV
0.6
1 2.03
1 3.08
1 1.8
1 2.24
1 0.49
1.14
2.05
1.35
(i.c. reconstructed patch Pij) of the INR decoder. Vim 's in the first ProdPoly block are implemented as linear layers. In the remaining blocks, transpose convolution layer is used with appropriate padding and strides. To remove the redundant parameters, similar to Li et al. (2022b), we also replace the convolutional kernel in F1 with two consecutive convolution kernels with small channels. The optimal rank for our resultant polynomial's decomposition per the NCP (Eq. 2 was found be 324. Appendix [A.4 presents a detailed study pertaining to the choice of optimal rank for the decomposition, alongside elaborate architecture details.
3.5 Training
To train our network, we randomly sample a batch of frame patches Ptj along with their normalized fine coordinates, coarse coordinates, and the time indices (Aij. A), These indices are then given as input to PNeRV to predict the corresponding patches P.y. The model is trained by using a combination of the L1 and SSTM Wang al. (2004) losses between the predicted frame patches and ground truth frame patches,
governed by Eq.
10
T
MxN Poilli + (1 - (S.M P.j))
L(Pij, Pi)) MXNXT
1
Σ
P Σ
(10)
where, MM x N is the total number of patches per frame, T denotes the total number of frames, and y is a hyper-parameter lo weigh the loss components. We set y to 0.7. We infer frame patches at all the locations and concatenate them in a consistent manner to reconstruct the original videos. Since the model learns non- overlapping patches independently, the intensity changes near the patch edges may cause the reconstructed frames to have boundary artifacts. We apply Gaussian blur to the reconstructed video to mitigate these subtle artifacts. No further post-processing is required for continuity and coherence in the generated frames.
4 Experiments
We split our experimental analysis of PNeRV into (1) evaluation of the representation ability using Video Reconstruction task (2) testing the efficacy on the proposed downstream tasks (3) performing appropriate ablation studies to assess the contributions and salience of individual design elements. The downstream tasks we perform include (i) Video Compression to assess the applicability PNcRV as an alternate lightweight video representation (ii) Video Super- Resolution to assess the spatial continuity of PNeRV (iii) Video Inler- polation assess the temporal continuity of PNeRV (iv) Video Denoising as an interesting application of PNcRV. We also compare the rate of convergence (during training) of PNcRV vis-à-vis prior art.
Experimental Setup: We train and evaluate our model the widely used UVG dataset (Mercat et al. 2020) and the "Big Buck Bunny' (Bunny) vidco sequence from scikit-vidco. The UVG datasct compriscs 7 videos. Each UVG video is resized to 720 x 1280 resolution and every 4th frame is sampled such that the entire video contains 150 frames. All 132 frames of the Bunny sequence are used at a resolution of 720 x 1280. For all our experiments, we train cach model for 300 epochs with a batch size (unless specified otherwise) with up-scale factors set to 5, 2, The input embeddings T TFPE, PrSE, and TpPE are computed with 1.25. We set I = 80 for Trre and TrSE- Whereas, ITSE uses 40. The network is trained using
8

---

Published in Transactions on Machine Learning Research (02/2024)
Adam optimizer (Kingma & Ba 2014 with default hyperparameters, learning rate of 5e and cosine annealing learning rate scheduler [Loshchilov & Hutter 2016). Following F-NeRV's evaluation methodology, we use PSNR (Wang et al. 2003) to evaluate the quality of the reconstructed videos.
4.1 Video Reconstruction
High fidelity vidco reconstruction assumes utmost importance when it comes to building INR. We compare PNeRV with several slate-of-Uhe-ard methods, namely NeRV-L (Chen et al. 2021), E-NeRV (Li et al. 2022b) and HNeRV (Chen et al. (023) on videos belonging to the UVG dataset and the Bunny video. The PSNR values obtained for reconstructed videos are reported in Table 2 We observe that our model consistently outperforms existing methods on a diverse set of videos, while employing significantly lesser number of learnable parameters shows improvements on videos with slow moving objects like Beauty, Bee, Shake as well as dynamic videos like Bunny, Bosphorus and Yacht. Hence, validating that the PNN-backed PNeRV is lightweight INR that captures the necessary spatiotemporal correlations needed to better represent videos. We present qualitative comparisons with state-of-the-art for the task in Fig. 10 (Appendix A.6 (left column). Appendix A.5 presents additional qualitative results.
4.2
Downstream Tasks
4.2.1 Video Compression
Recent video compression algorithms follow a hybrid approach where a part the compression pipeline consists of neural networks while following the tradi- tional compression pipeline (Agustsson et al. 20200] Yang et al.] 2020 Wu et al., 2018). An INR. en- codes a video as the weights of a neural network. This enables the use of standard model compression tech- niques for video compression. Following (Chen et al.) 2021), we employ model pruning for video compres- sion. We present experimental results for the same on the 'Big Buck Bunny' sequence from scikit-video in Figure 5 It can be observed that PNeRV model of 40% sparsity achieves results comparable to the full model, in terms of reconstruction accuracy and per- ceptual coherence. Fig. 10 (Appendix A.6) (middle column) presents qualitative comparisons with state- of-the-art for the task. For sparsity values less than 45%, our model outperforms NeRV and E-NeRV. How- ever, beyond 45% sparsity, PNeRV's performance de- grades rapidly. This behaviour can be attributed to the of multiplicative interactions in PNeRV which cause model performance to increase rapidly with increase in model parameters. We provide additional qualitative results, quantitative results on the UVG dataset, and comparisons with HNeRV in Appendix A.7 From Fig. 17 it can be observed that the frames predicted by HNeRV are blurred typical property of autoencoder type of an architecture whereas our method able to preserve the fine details well.
44
42
PSWR
38
36
34
32
Sparsity
NeRVAL
N-NeRV
PNeRV
Figure Model pruning results on NeRV-L, E- NeRV and PNeRV trained for 300 epochs on "Big Buck Bunny" video. Sparsity represents the ratio of pruned parameters.
4.2.2 Video Super-Resolution
We present qualitative results for x4 Super-Resolution in Fig. 6 As reported in Table 3 for Super- Resolution, we compare our results with bicubic interpolation, ZSSR (Assaf Shocher 2018 and SIREN [Sizzaan et al. 2020b). PNeRV outperforms these baselines in each case, which confirms that PNeRV is a generic spatiotemporal representation that lends itself well to various downstream tasks that require spatial continuity without the need for task-spocific retraining or fino-tuning. We also provide reasons for not comparing our results with VideoINR (Chen al. 2022b), an important contemporary INR based method in Video Super-Resolution in Appendix A.9

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Quantitative comparisons for x4
GT
ZSSR
SIREN
Ours
super-Resolution
PSNR (dB) f
Method
Bunny
Beauty
Bicubic
29.82
34.03
ZSSR
27.53
31.96
Figure 6: Qualitative Results for x4 Super- Resolution. PNeRV's superior performance be observed in the high frequency regions.
SIREN
21.68
29.61
Ours
31.74
36.48
Table 4:
PSNR (dB) metrics for Vidco
Table 5: PSNR (dB) metrics for Video De-
Frame Interpolation
noising
Seen Frames
Unseen Frames
Type of Noise
Method
Bunny
Beauty
Bunny
Beauty
Method
white
salt & pepper
NeRV-L
39.3
36.16
28.58
23.98
NeRV-L
38.41
39.83
E-NeRV
42.52
36.96
33.77
26.41
E-NeRV
37.73
38.98
Ours
43.10
38.66
33.91
28.63
Ours
39.62
41.89
4.3 Video Frame Interpolation
The temporally continuous nature of PNeRV, allows us to perform the task of Video Frame Interpolation. We train and evaluate PNeRV on the "Bunny" and 'Beauty" videos for this task. We report the quantitative and qualitative comparisons for the task in Table and observe that our method achieves better metric performance than prior art, and excellent perceptual quality of the predicted 'unseen' (interpolated) frames. Hence, we infer that PNeRV better captures spatiotemporal correlations in videos with respect to prior art. We present additional results for the task in Appendix A.
and Fig. 10 (Appendix (.66) (right column), respectively. We
4.3.1 Video Denoising
INRs have been shown to be better attuned to filtering out inconsistent pixel intensities i.e. noise and perturbations. Hence, making it suitable for denoising videos without being explicitly trained for the task. To test the performance our representation on noisy videos, we applied white noise and salt and pepper noise separately to the original videos. PNeRV was then trained on these perturbed videos for reconstruction. Comparisons between the reconstructed videos and the original videos reveal that the representation learned by PNeRV is robust to noises. It implicitly learns a regularization objective filter out noise better than existing methods. Quantitative comparisons with prior art (reported in Table 5 assert the superiority of our method. We also provide qualitative results and detailed analysis of the same in Appendix A.10
4.4 Ablation Studies
4.4.1
Varying the polynomial attributes of the INR Decoder
We study the impact of varying the rank and order of the polynomial formed by the PNN-based INR Decoder architecture.
Rank of the Polynomial: NCP-Polynomial formulation, the rank of the polynomial can be varied by modifying the number of channels of the F.m module in each ProdPoly block. In general, it is expected that polynomial with banker decomposition (i.e. more channels) would perform better due to the incrcased expressivity of the representation learned by the model. To understand the effect of this, we modify the rank of the first ProdPoly block in the INR-Decoder while keeping the ranks of the second and third ProdPoly block fixed. These results are reported in Tab. 0 It can be seen that the rank of the polynomial is positively correlated to the quality of the reconstructed video.
Order of the Polynomial: Each ProdPoly block in the proposed architecture has an order of 2. Thus, the effective order of INR-Decoder is 2 where R is the total number of ProdPoly blocks in the decoder.
10

---

Published in Transactions on Machine Learning Research (02/2024)
Effect variation the rank (controlled by the number of channels) of individual ProdPoly decompositions in terms of PSNR. with re-
Table Ablation:
Table 7: Ablation: Effect of variation of the order (controlled by the number of ProdPoly blocks) in terms of PSNR. for re- construction "bunny' video.
spect to reconstruction on "bunny" video.
Rank of the Polynomial Component
# ProdPoly Blocks
PSNR (dB)
# Params
PSNR
ProdPoly: ProdPoly:
ProdPoly:
321
96
96
44.9
2
11.50 M
43.78
212
96
96
42.05
3
11.89 M
44.90
112
39.23
12.29
44.65
Table 8: Ablation: Effect of the individual PE components formulation on PSNR (dB) for re- construction.
Table 9: Ablation: Characterizing the effect of varying patch sizes in terms of # #Parameters and PSNR (dB) for reconstruction.
Tpestl TRMEC)
Prse(]
Bunny
Beauty
Patch-size
# Parameters
Bunny
Beauty
Haseline
21.85
35.3
H/S, D/8
12.37
42.02
37.21
Centroed
H/4, D/4
11.89
44.90
39.80
Parumetric PP only
43.83
39.70
H/2, D/2
12.56
44.27
33.82
H. D
12.94 M
42.65
32.46
Hence, we vary the number of ProdPoly blocks to change the order of INR-Decoder polynomial and report our findings in Table 7 It can be seen that the performance drops when the order is reduced. Interestingly, the PSNR value decrcases when the order is increased beyond certain range. We also present an analysis of PNeRV's independence to the choice of non-linear activations in Appendix A.12 property it inherits from the PNN paradigm.
4.4.2 Efficacy of Positional Embeddings
We demonstrale the contribution of each Positional Embedding (PE) with respect to its individual contribu- tion toward the reconstruction quality achieved. To this end, we first propose two simple baselines as shown in Table S wherein each patch is assigned a coordinate from 0 to M x N - in a row-wise fashion (row 1) or each patch is assigned its centroid value (row 2). Then I НFPЕ is used to compute the patch embeddings. It evident that the performance drops considerably in both these settings. Hence, motivating the need of carefully designed positional embeddings. Next, add the parametric PE (TPPE) (row 3) followed by addition of functional PE (TTSE). The results show that both PpPE and ITSE contribute to the overall network performance. For this ablation study, t is encoded using FFPE and fused with the spatial embedding using the HMF block in all the experiments. Our well-designed PE scheme greatly enhances our model's 's performance by leveraging the high-frequency information preferred by the PNN paradigın.
PSNR (0B)
45.40
205
30
PSBR
NeRU
Our
1949
zae
Epochs
Epochs
(4) POWR (8B) venus No. Taindin Ejocks "Bansyy (Adeo.
So POME (ES) versas No. Triiking Epoaiss the "Goule" (1VV0) video
Figure 7: Rate of Convergence (PSNR (dB) for reconstruction versus #training epochs) compared to state- of-the-art.
11

---

Published in Transactions on Machine Learning Research (02/2024)
4.4.3 Varying the Input Patch-Size
The patch-wise formulation is the key idea that enables us to model spatial continuity, Thus, we delve into PNeRV's performance obtained for different patchs sizes in Table 9 We found that a patch size of ( B) performs the best. This suggests that neither a pixel-wise (dense spatial) nor frame-wise representation (temporal-only) is optimal. We hypothesize that the surge in parameters (oxer-parancterization) in the pixel-wise approach might be the limiting factor that inhibits learning in such cases. We find this result particularly insightful since we found sweet-spot between the two parameterization methodologies. Another interesting trend to observe from Table 9 is that of the relationship between patch size and the number of parameters. We discuss the reasoning behind this in detail in appendix A.13
4.4.4 HMF versus other fusion strategies
We compare the proposed PNN-backed Higher-order Mul- tiplicative Fusion (HMF) of space and time embeddings with other fusion mechanisms as given in Table 10 As ex- pected, conventional concatenation, addition, or multipli- cation operations features fail to capture the auto and cross-correlations of the inputs. Hence, causing a drop in performance. We observe that the dip in PSNR is more pro- nounced for the 'bunny' video than the "baaut"" video. We attribute this observation to the 'bunny" video having more temporal variations. The results of this study indicate that the proposed HMF scheme models both the structural and the perceptual vidco attributes better than the prior art.
Table 10: Ablation: Assessing the efficacy of our HMF versus other parametric PE fusion strategies in terms of PSNR (dB) for recon- struction.
PE Fusion Strategy
Bunny
Beauty
Concat + Linear
43.76
39.39
Linear Elementwise Addition
43.28
39.39
Linear + Hadamard Product
43.06
38.92
Ours
44.9
39.80
4.5 On PNeRV's rate of convergence
Following E-NeRV's sctup, we perform reconstruction experiments with PNcRV models trained for different number of training epochs the 'Bunny' and "Yacht" (UVG dataset) videos and report our findings in Fig. 7 It can be seen that training for more number of epochs boosts the performance with upto 4x faster convergence than baselines. PNeRV's performance surpasses that of the baselines at 600 epochs on the 'Bunny" and 1200 epochs on the "Yacht'. We also provide comparisons with state-of-the-art with respect to inference time in Appendix A.8
5 Conclusion
In this work, propose and validate the efficacy of PNeRV, light-weight spatiotemporally continuous, fast, and generic neural representation for vidcos with versatile set of practical downstream applications. We do so by building on two principal insights. First, a well-designed patch-wise spatial sampling scheme can perform just as as good as a pixel-wise sampling. Second, replacing popular function approximators by the more efficient PNNSs and designing other model components to aid its learning can lead to superior performance. We provide conclusive results to support our claims with analysis on several downstream tasks and consistent ablation studies. We believe our work shall serve as a primer Loward building spatiotemporally continuous light- light-weight INRs for videos. As future work, it would be interesting to examine PNN based PEs to further improve INR for vidcos. Please find our broader impact statement in the following subscction.
5.1 Broader Impact Statement
As one of the most widely consumed modality of data, videos are central to several important tasks in the modern socio-technical context. In such a scenario, PNeRV brings in a fresh approach to tackle the ever growing costs involved in handling such massive data by providing a method restore and compress videos efficiently. In effect, PNcRV can potentially have a lasting positive impact on several video streaming, communication, and storage services. As with any nascent technology, the largely positive impact areas are accompanied by few unforeseeable ones which are beyond the scope of this work.
12

---

http:/ppcoedd.....eeeeeeeesee/pp Published in Transactions on Machine Learning Research (02/2024)
References
Eirikur Agustsson, David Minnen, Nick Johnston, Johannes Balle, Sung Jin Hwang, and George Toderici. Scale-space flow for end-to-end optimized vidco compression. In Proccedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 8503 8512, 2020.
Michal Irani Assal Shocher, Nadav Cohen. 'zero-shot" super-resolution using deep internal learning. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, June 2018.
Francesca Babiloni, Ioannis Marras, Filippos Kokkinos, Jiankang Deng, Grigorios Chrysos, and Stefanos Zafeiriou. Poly-nl: Linear complexity non-local layers with 3rd order polynomials. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10518 10528, October 2021.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. NeRV: Neural representations for videos. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL
net/formmfid
BbikqBW2TGB
Hao Chen, Matthew Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. HNeRV: Neural representations for videos. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
Xi Chen, Zhiyan Zhao, Yilei Zhang, Manni Duan, Donglian Qi, and Hengshuang Zhao. Focalelick: Towards practical interactive image segmentation. In Proceedings of the IEEE/CV /CVF Conference Computer Vision and Pattern Recognition, pp. 1300-1309, June 2022a.
Yixin Chen, Grigorios Chrysos, Markos Georgopoulos, and Volkan Cevher. Multilinear operator networks. In International Conference on Learning Representations (ICLR), 2024.
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Vidcoinr: Learning vidco implicit ncural representation for continuous space-lime super-resolution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022b.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1932-5941, 2019. doi: 10.1 CCPPR. 2019.00609.
Moulik Choraria, Leello Tadesse Dadi, Grigorios Chrysos, Julien Mairal, and Volkan Cevher. The spectral bias of polynomial neural networks. In International Conference on Learning Representations, 2022. URL
httpss:/opeeneviiee http:///pppppeoeeeeoooooooooooop
Grigorios Chrysos, Stylianos Moschoglou, Yannis Panagakis, and Stefanos Zafeirion. Polygan: High- order polynomial generators. ArXiv, abs/1908.06571, 2019. URL https://apl sww.ntcsccoooar..n.m org/
2021122D
Grigorios Chrysos, Markos Georgopoulos, and Yannis Panagakis. Conditional generation using poly- nomial expansions. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 28390- 28404. Curran Associates, Inc., 2021a. URL aEddBiiaa66:50a02101555333 Pwp.r.
Grigorios Chrysos, Stylianos Moschoglou, Giorgos Bouritsas, Jiankang Deng, Yannis Panagakis, and Stefanos Zafeiriou. Deep polynomial neural networks. IEEE transactions on pattern analysis and machine intelligence, 44(8)-4021-4034, 2021b.
Grigorios G. Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classificrs with polynomial neural networks. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 692 716, Cham, 2022a. Springer Nature Switzerland. ISBN 978-3-031-1988669.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Grigorios G Chrysos, Markos Georgopoulos, Jiankang Deng, Jean Kossaifi, Yannis Panagakis, and Anima Anandkumar. Augmenting deep classifiers with polynomial neural networks. In European Conference on Computer Vision (ECCV), pp. 692- 716, 2022b.
Grigorios G Chrysos, Bohan Wang, Jiankang Deng, and Volkan Cevher. Regularization of polynomial networks for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Yingying Deng, Fan Tang, Weiming Dong, Chongyang Ma, Xingjia Pan, Lei Wang, and Changsheng Xu. Stylr2: Image style transfer with transformers. In Proceedings of the PEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11316 11326, 2022. doi: 10.1109 10.1109/CVPR52200 CWWP522688202220111222
Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua B. Tenenbaum, and Jiajun Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEFE/CVF International Conference on Computer Vision, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint MrXXi:1606.084155 2016.
Cong Huang, Jiahao Li, Bin Li, Dong Liu, and Yan Lu. Neural compression-based feature learning for video restoration. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1872-5887 June 2022.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Diederik P Kingma and Jimmy Ba. arXiv:1412.6980. 2014.
Adam: A method for stochastic optimization. arXiv preprint
Kaidong Li, Ziming Zhang, Cuncong Zhong, and Guanghui Wang. Robust structured declarative classi- fiers for 3d point clouds: Defending adversarial attacks with implicit gradients. In Proceedings of the ICEF/CVF Conference Computer Vision and Pattern Recognition, pp. 15294-15304, June 2022a.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view syuthesis of dynamie scenes. In Proceedings of the IEEE/CVT Conference Compuler Vision and Pattern Recognition, pp.
June 2021.
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expedite neural video representation with disentangled spatial-temporal context. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022, pp. 267-284, Cham, 2022b. Springer Nature Switzerland. ISBN 978-3-031-19833-5
Πya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
Long Mai and Feng Liu. Motion-adjustable neural implicit video representation. In Procoedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10738 10747, June 2022.
Alexandre Mercat, Marko Viitanen, and Jarno Vanne. Uvg dataset: 50/120fps 4k sequences for video codec analysis and development. In Proceedings of the 11th ACM Multimedia Systems Conference, pp. 297-302, 2020.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowazin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communccations of the ACM, 65(1):99-100) 106, 2021.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Occupancy flow: 4d reconstruction by learning particle dynamics. October 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proccodings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, June 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Scitz, and Ricardo Martin-Brualla Nerfies: Deformable neural radiance fields. Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10318 10327, June 2021.
Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard G. Baraniuk, and Ashok Veeraraghavan. Miner: Multiscale implicit neural representation. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (ods.), Computer Vision ECCV 2022, pp. 318-333 333, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20050-2.
Bipasha Sen, Aditya Agarwal, Vinay P Namboodiri, and C.V. Jawahar. INR-v: A continuous representation space for video-based generative tasks. Transactions on Machine Learning Research, 2022. ISSN 2835-8856. URL https:/ www.:et/ooooummmdaaaoaoeeeeeeeee
Rajhans Singh, Ankita Shukla, and Pavan Turaga. Polynomial implicit neural representations for large di- verse datascts. In Proceedings of the IEEE/ /CVF Conference on Computer Vision and Pattern Recognition, 2051, June 2023.
pp. 2041-2051,
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. Advances in neural information processing systems, 33: 7462-7473, 2020a.
Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NeurIPS, 2020b.
Ivan Skorokhodov, Savva Ignalyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. Proceedings of the TEFE/CV /CV Conference on Computer Vision and Pattern Recognition, pp.
June 2021.
Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. NeurIPS, 2020.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Carsten Stoll, and Christian Theobalt. Patchnets: Patch-based generalizable deep implicit 3d shape representations. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 293 309, Cham, 2020. Springer International Publishing. ISBN 978-3- 030055517-4
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (ods.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL pdf
15

---

Published in Transactions on Machine Learning Research (02/2024)
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEFE conference computer vision and pattern recognition, pp- 7794-7803, 2018.
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assess- ment. In The Thirth-Seventh Asilomar Conference Signals, Systems & Computers, 2003, volume 2, pp. 1398-1402. Icce, 2003.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE Transactions on Image Processing, 13(4):600-6)) -11660006111,- 2004. doi: 10.1109/TIP. 2003.819561.
Yuan Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 2018
Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative position encoding for vision transformer. In Proceedings of the LEEE/CVF International Conference on Computer Vision, pp. 10033 10041, October 2021.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Spaco-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogmition, pp.
2021.
Ren Yang, Fabian Mentzer, Luc Van Gool, and Radu Timofte. Learning for video compression with hier- archical quality and recurrent enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6628 6637, 2020.
Tal Hassner Yuval Nirkin, Lior Wolf. Hyperseg: Patch-wise hypernetwork for real-time semantic segmenta- tion. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021. URL https://talhassser gi oitttti.oohootteeeeeeooeoooo http:///h//////plllcccoooo///22 CVPR_ 2
Zihan Zhu, Songvou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Mare Pollefeys. Nico-slam: Neural implicit scalable encoding for slam. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12786-12796, June 2022.
16

---

Published in Transactions on Machine Learning Research (02/2024)
A Appendix
A.1 Abbreviations
INR
Implicit Neural Representation
PNN
Polynomial Neural Network
HMF
Higher-order Multiplicative Fusion
PE
Positional Embedding
FPE
Positional Encoding of Frame Index
PPE
Parametric Embodding Fine Coordinates
TSE
Time Aware Spatial Embedding
A.2 The mode-n product
The mode-n (matrix) product of tensor Bixxxx....xx with matrix U RJxx is denoted by X and is of size I x In.1 J Iat1 X In. Elementwise, have
(X' Xn U) i Hatz ...iN
Tiybg.. Ujsnn
Each mode- fiber [of X is multiplied by the matrix U.
A.3 The Hierarchical Patch-wise Spatial Sampling Algorithm
Algorithm Hierarchical Patch-wise Spatial Sampling
Input: C, H, D, K, L, M, N, Paj. Pkt Output: Xig, Aij
1: function HPSS(C, Pay. Plt. H. M,
Xij (L topí P.z))-ottonn
hottom P(j) (T)(Ps))+rghhtt
Aij + A matrix of dimensions K x L with k and [ being the row and column index, respectively.
4: x == top( Pkt), y = left( h
H d = D NKK
NL
5: for k: < 0 to - do
6:
for - to
do
Aiylh]0 < (1 20 2yth
2
== y+h
end for y left
10:
11:
== x + d end for
12:
13: return Aij Aij
14: end function
Convolution
Pixel (scale)
activation (GeLU)
Figure 8: Detailed architecture of the F' blocks.
17

---

Published in Transactions on Machine Learning Research (02/2024)
Figure Detailed diagram for the U block in the INR. Decoder. Yellow blocks represent the transpose convolutional layers, whereas the green rectangles are fully connected layers.
Output Size C' x H x W
Layer
Modules
Upscale Factor
U
MLP & TransposeConv2D & Reshape
324 x 16 X 9
11
MLP
160 x 324
Fi1
F block
324 x 80 45
+ 12
MLP
160 x 162
F12
F block
2
162 160 x 90
P21
TransposeConn22
2
384 320 X 180
F21
F block
2
384 x 320 x 180
222
TransposeConv2D
96 x 320 180
F22
block
96 x 320 x 180
Pai
TransposeConvv21
96 x 320 180
Fs1
F block
96 x 320 180
432
TransposeConv2D
96 320 180
F32
F block
96 x 320 180
ToRGB Layer
Convolution
3 x 320 x 180
Table 11: INR-Decoder Architecture.
A.4 The INR Decoder architecture in detail
In this section, we provide the finer dotails of the PNcRV architecture. We then provide more details about the implementation and training of the proposed method. PNeRV consists of three components: the Positional Embedding Module (PE), the Embedding Fusion Block, and the INR-Decoder. Given the coarse patch coordinate Ay, fine patch coordinate Aij and the time index, first compute the positional embeddings TrSE (Ay,, t), (PP)))i)) and (F)E(t). The embeddings TPPE (Az)) and TFPE(t) are fused using Polynomial Neural Networks (PNN) based fusion module HMF. HMF consists of series of linear transformations followed by Hadamard product and addition, as shown in Fig 4 of the paper. Each linear layer, namely, Al,,), A A3,)),
AlIAA) A2), Ag.Aw. is of dimension 80 x 160 The resulting embedding is added elementwise to TrsE Ay: 1) obtain the fused embedding z which given as input to the TNR. Decoder. vector of dimension 160.
The INR-decoder consists of stack of 3 prodpoly blocks. Each prodpoly block in turn is 2nd order NCP- Polynomial implemented using convolutional blocks Frm, where is the index of the prodpoly block and is the index corresponding to the F-block. The structure of F is illustrated in Fig. g To limit the increase in the number of parameters of the model, following (Li et al. (022b), we employ the following design for Fi block: Conv(C1, Co x & x s) + pixel-shuffle(s) > Conv(Co, C2). Where, C₁ = 324, Co = 81, C2 = 324 and s 5. The input vector mappod to feature map using a 3rd-order polynomial implemented using transpose convolutional layers as depicted Fig. g This is referred U in Fig. Table. provides the complete architecture details for INR-decoder.
18

---

Published in Transactions on Machine Learning Research (02/2024)
A.5
Qualitative results for Video Reconstruction
We provide additional comparisons with state-of-the-art in Fig, [6 and additional qualitative results for our method illustrated in Fig. 14 and Fig. [5 Owing to the ensemble of design elements, PNeRV outperforms
state-of-tho-art convincingly on this task.
A.6
Qualitative Comparisons
Figure 10
Appendix
A.6 presents qualitative comparisons with prior art on Reconstruction, Compression,
and Interpolation.
Reconstruction
Compression
Interpolation
GT
NeRV
E-NeRV
Ours
Qualitative comparisons with prior art Reconstruction, Compression, and Interpolation. The specific regions where method predicts significantly better outputs are highlighted in red boxes.
Figure 10:
A.7 Additional Results for Video Compression
Table 12 (a) provides quantitative comparison with state-of-the-art video compression in different sparsity (denoted by p) settings. Our model outperforms prior art convincingly. Fig. 17 wherein we present qualitative comparisons with state-of-the-art on the task with sparsity 0.2, further underscores PNeRV's superior performance.
Table 12: (a) Averaged Comparison Model Compression (b) Quantitative Comparison Inference Time.
(a) Compression PSNR (dB)
(b) Inference Speed Inference Time (ms)
Method
== 0.2
== 0.4
NeRV
32.30
31.20
153.81
E-NeRV
32.54
32.28
34.11
Ours
33.86
33.60
28.32
19

---

Published in Transactions on Machine Learning Research (02/2024)
A.8 Quantitative Comparison: Inference time per forward pass
Table 12 (b) provides a quantitative comparison with state-of-the-art in terms of time taken (ms) to perform
one forward pass of the model on NVIDIA GeFORCE RTX 3090 GPU. Results clucidate that our light-weigg weight model faster then prior art.
A.9
Super-Resolution using PNeRV
On the comparison with VideoINR: VideoLNR (Chen et al. 2022b) has two core differences from our work. Firstly, VideoINR uses ground truth High-Resolution (HR) video frames for training, while ours is fully unsupervised approach utilizing only the low-resolution video for training. Secondly, our method is a multifunctional INR. In that, it learns to represent signal (video) as model weights. In contrast, VideoINR is an autoencoder trained specifically for Super-Resolution. Wherein, the claimed INR components function as non non-linear transformations in the intermediate feature space. Therefore, we do not compare with VideoINR. Instead, we show qualitative results for Super-Rcsolution (Fig. and quantitative comparison with bicubic interpolation, ZSSR, and SIREN (Table which are unsupervised models.
A.10
Video Denoising: Qualitative Results
Noisy
NeRV
Salt
Pepper
Nolsy Frame
NeRV
E-NeRV
Ours
Gaussian
Noise
Figure 11: Qualitative Comparison of denoising results obtained on 'honcybec' video. (a) Salt and Pepper
Noise, (b) Gaussian Noise.
Figure 11 shows the qualitative comparison of the output of our method with the two INR baselines NeRV (Chen et al. 2021) and E-NeRV (Li et al. 2022b). Notice that E-NeRV fails to reconstruct the honeybee, thus regularizing the video such that the original content is lost. NcRV can generate the honeybee but it lacks clarity. PNcRV preserves all the content of the frames including honeybee and generates superior-quality video. These results confirm that PNeRV learns more robust video representation.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Table 13: Metrics analyze the effect of absence of non- linear activations in terms of PSNR (dB) for reconstruction.
NeRV
E-NARV
Method
Bunny
Beauty
NeRV-L 31.71
27.53
E-NeRV 27.57
27.49
Ours
39.84
38.50
Visualization of frames reconstructed by models trained without non-lineat activation functions. The highlighted regions illus- trate our method's robustness to the choice activation employed.
Figure 12:
Seen Frame
Interpolated (Prediteed
GT Frame
Seen Frame
Figure 13: Qualitative results for Video Frame Interpolation on the 'Bunny" (rows 1 and 2) and "Beauty" (rows 3 and 4) videos. Columns 1 (seen, previous) and 4 (seen, next.) show the seen frames used to interpolate (predict) the unseen frame illustrated in column 2. The closeness of predicted frames (column 2) to the ground truth frames (column 3) underscores the faithfulness of our interpolation.
A.11 Video Frame Interpolation
Following F-NeRV's setup, we divide the training sequence in 3:1 ("seen:useee"") ratio such that for every four consecutive frames, the fourth frame is not used training. This "unscen" frame is interpolated during inference to quantitatively evaluate the model's performance.
Table 14: MS-SSTM metrics for Video Frame Interpolation
Method Beauty
NeRV-L 0.8161
E-NeRV 0.9745
Ours 0.9775
21

---

Published in Transactions on Machine Learning Research (02/2024)
Figure 14: Visualization few frames of the reconstructed videos on "bunny" (first row), 'beauty' (second row), 'honeybee' (third row) and "bosphorus' hrmphoous'(furth row) videos of UVG dataset (Mercat et al. 2020
Fig. 13 provides the qualitative results for Video Frame Interpolation task on 'bunny' and 'beauty' videos. It can be observed that the perceptual quality of the interpolated frame is similar to that of the ground truth for the bunny video. We also assess the quality of the interpolated video using multi-scale structural similarity (MS-SSIM) metric. MS-SSIM accounts for luminance, contrast and structure of each frame and hence better correlates with human perception. Table 14 reports the MS-SSTM numbers 'beauty' video. PNcRV outperforms SOTA methods on this metric as well suggesting the superior quality of the interpolated video.
A.12 Robustness to the choice of activation function
Since PNNs (Chrysos et al.) 2021bb have built-in non-linearities, they do not rely on the usage of popular hand-crafted non-linear activation functions to yicld best performance. To highlight this aspect of method, test the effect of training network and the baselines without any activation functions "bunny" dataset by removing activation functions from all the network layers except for the output layer. The quantitative and qualitative results for the same are reported in [able[Iaand Fig. 12 It can be observed that performance the baselines NeRV (first row) and E-NeRV (second row), dropped significantly. In contrast, the performance of our model remains comparable. Tt is notable that NeRV fails learn high-frequency information such as that the face of the bunny (highlighted in red boxes), resulting in worse qualitative performance.
A.13 Relationship between batch-size and number of learnable parameters
Intuitively, one would expect that number learnable parameters would increase with an increase in patch size. However, as observed Table D the number of parameters docreases when transitioning from patch size of ( 8 to (.. ) because of a lesser number of channels convolutional layers of the prodpoly blocks for the latter.
22

---

Published in Transactions on Machine Learning Research (02/2024)
TuRpe Tr AUIIN
Pirpe Teb
Figure 15: Visualization of few frames of the reconstructed videos on "jockey' (first row), "shakeandry' (second row), 'yachtride' (third row) and 'readysetgo' (fourth row) videos of UVG dataset Mercat et al. 2020
To be specific, for (5) W) configuration, the number of channels in F22, Fai and F32 (sce Table 11 in Appendix and Figure 2 are 384 whereas for the ( ( 4) scenario, the number of the channels are all set to 96. We arrive this design choice empirically. The F' blocks comprise pixel shuffle layers which are responsible chosen to be 96, channels in the previous layer divided by the upsampling factor square, whichever is minimum. For palch sizes greater than or equal to H W the channels in the F2z, Fai and F3z layers become 96. The increase in parameters after these layers is due to the rm layers that upsample the input feature map to appropriate dimensions. This contributes to the extra parameters.
upsample the size of the generated output. In design, the number of channels
23

---

Published in Transactions on Machine Learning Research (02/2024)
GT
Highuric
NeRV
E-NeRV
HNeRV
PNeRV (Ours)
Figure 16: Qualitative comparisons with stato-of-tho-art with respect to vidco reconstructing on the "shake" (columm 1), "bosphorus" (column 2), and "beauty" (column 3) videos of the UVG dataset (Mercat el al. 2020) 'GT' denotes the ground truth frames. As is evident, PNeRV outperforms thee--ff particularly regions with high frequency information content.
24

---

Published in Transactions on Machine Learning Research (02/2024)
GT
NeRV
E-NeRV
HNeRV
PNeRV (Ours)
Figure 17: Qualitative comparison with state-of-the-art with respect to video compression (p = 0.2) the "Bunny" video. "GT" denotes ground truth frames. The highlighted regions depict regions where PNeRV oulperforms state-of-the-arl evidently.
25


Reviews:
[{"review_id": "qjW5n0Nvhe", "reviewer": "Reviewer_xD4r", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The motivations of the proposed methods are presented clearly. I have little expertise in NeRV and PNNs, but the authors did a good job of helping me understand the context of this work and the reasons why they proposed these methods.\n    \n- The authors did extensive ablation studies to show the effectiveness of each proposed component.\n    \n\nW\n\n- There are quite a few minor issues in the present paper (see Requested Changes below for several). I request the authors recheck the whole paper and try to fix the minor issues as much as possible, so the paper can be read more easily.\n    \n- The qualitative results are not very impressive to me. For example, the interpolation result in Figure 5 might be too sharp to be desirable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper provides a novel solution to balance between inference efficiency and spatiotemporal continuity for video modelling with INRs by hierarchically sampling patch-wise spatial coordinates.\n    \n2. The work applies PNNs to process temporal signals and make adaptations to leverage PNNs’ strength in parameter efficiency.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Minor issues: \n- Two places refer to the same footnote 3 in Section 2. Please delete one of the references and leave no space between the text and the index.\n    \n- There’s no Section 8 but Section 3.1 and 3.2 refer to it.\n    \n- I find no description of NeRV**-L**.\n    \n- In Figure 6, ”Poly-INR” should be changed to “PNeRV” or “Ours” for consistency.\n    \n- Section 4.2.2: “As reported in Table 3, for SR, we compare our results with bicubic interpolation, INR-V (Sen et al., 2022), ZSSR (Assaf Shocher, 2018), and SIREN (Sitzmann et al., 2020b).” However, INR-V’s results are not presented in Table 3.\n    \n- Section 4.2.2: ”We also provide reasons for not comparing our results with VideoINR (Chen et al., 2022b), an important contemporary INR based method in Video SR in Appendix (Chen et al., 2022b).“ The referencing is incorrect.", "broader_impact_concerns": "I have no concerns about the ethical implications of the work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lHF7jDGvv0", "reviewer": "Reviewer_LSEQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- ### Strenghts\n\t- Novelty of adapting polynomial neural networks for implicit neural representations. Using the hierarchical sampling\n\t- Results for compression and frame interpolation seem to be good, outperforming previous methods.\n\t- The method is evaluated on multiple tasks and on the standard benchmarking dataset UVG and on the Big buck bunny Video.\n\t- Ablation studies\n- Weaknesses\n\n\t- Writing quality and presentation of the work can use a lot of improvement. (See requested changes below)\n\t- The methodology seems very complicated. There are a lot of minor engineering details. It is good that the authors described all of these but it makes the paper hard to follow.\n\t- The validity of the novel embedding scheme and the hierarchical sampling procedure seem questionable to me. I want to clarify that it could be due to me not understand the concepts clearly. Something that the authors could try to address by improving the writing quality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors propose a novel network architecture for neural representation for videos. The architecture consists of the following components:\n\t- 1. A positional embedding module, the authors combine Fourier features, angle modulation and fusion.\n\t- 2. A polynomial neural network as the backbone, instead of a \"conventional\" neural implicit network.\n\t- 3. A Hierarchical sampling scheme, consisting of coordinates for coarse and finegrained samples.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Writing\n\t- The paper contains many unusual formulations, please carefully proofread the paper to find and fix them. Examples:\n\t\t- \"such as **non-reliance** on the employment\"\n\t\t- \"We further propose a PNN based Higher order Multiplicative Fusion (HMF) module that is responsible for learning parametric embedding.\"\n\t\t- \"In contrast, PNeRV **gravitates** back to a spatiotemporal parameterization\"\n\t\t- \"The HPSS methodology is **encapsulated** by Algorithm 1 in Appendix A.2.\"\n\t- Many different abbreviations are introduced (HMF, PPN, PNN, INF, PE, TSE, PPE, EFB) and used instead of the original nouns in the text. This makes many sentences very hard to read. Please use these abbreviations sparingly.\n- Formatting\n\t- Text in tables is way too small\n\t- Weird parts of the text are in red. (around eq. 8, Table 6. and Tabl 9.)\n\t- PSNR is crossed in Table 9. What is this supposed to indicate?\n\t- Please use larger images in Figure 5. These images are not suitable for qualitative inspection at all. If in need for space, only show specific crops of the image.\n- Questions about validity of the Hierarchical Sampling scheme.\n\t- As can be seen in equation 3, the proposed model outputs a patch $P_{ij}$ based on coarse $\\Lambda$ and fine $\\lambda$ input coordinates. It is not clear to me what the hierarchical component of this scheme is. Given that both $\\Lambda$ and $\\lambda$ represent different frequencies of the single patch centroid, any conventional neural implicit representation should include $\\Lambda$ already as a low-frequency embedding of $\\lambda$ .\n\t- The authors should make more clear in the text, or even with experiments what is the added value of their \"hierarchical\" sampling scheme because it is not clear to me.\n\t- In paragraph 3.1 the authors contrast this with a naïve path-index. I think this is not a realistic baseline since neural implicit representations always map embedded 2d coordinates to the desired output.\n- Questions about the complexity of the embedding scheme.\n\t- The authors propose a very complicated coordinate embedding scheme. They are somewhat motivated by the ablation results in Table 8. But I wonder if there are not simpler methods with the same performance.\n- Ablation on the key contributions.\n\t- The paper seems to consist of three main components:\n\t\t- 1. A positional embedding module, the authors combine Fourier features, angle modulation (similar to Mai & Liu 2022)\n\t\t- 2. A polynomial neural network as the backbone, instead of a \"conventional\" neural implicit network.\n\t\t- 3. A Hierarchical sampling scheme, consisting of coordinates for coarse and finegrained samples.\n\t- While the authors do provide a multitude of ablations, it would be nice if they could include a single simple ablation that shows the performance of each of these three contributions individually. This would allow me to better judge the contribution to the (good) final performance.\n\t- As a reference, the E-NeRV paper itself has a very nice example of such an ablation in Figure 3.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "I42Uf5PEpV", "reviewer": "Reviewer_65SM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The overall model architecture is explained in detail\n+ The design choices that the authors made, including how to fuse positional embeddings and the architecture of INR decoder, improved the model's performance over the baselines, which was proved by the extensive experimental results and ablation studies.\n\nWeakness\nOverall, the writing and presentation should be improved.\n- There are many typos in equations and tables. For example, where does H in the first paragraph on page 7 come from? The HMF contains addition between x_i and x_{i+1} according to Figure 4 while Eq. 8-9 do not. The first paragraph on page 11 says that the higher-ranked decomposition (more channels) perform better while Table 6 contains the opposite.\n- The paper lacks some important details for understanding the content. For example, why does the number of parameters decrease and then increase again when the patch size decreases? (Table 9)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel neural representation (NR) for videos based on two principals: patch-wise spatial sampling that enables modeling spatial continuity while ensuring fast inference and leveraging Polynomial Neural Network (PNN)s for modeling high resolution videos with fewer parameters. The proposed NR, Polynomial Neural Representation for Videos (PNeRV), is equipped with the new positional embeddings (PEs) that utilizes both parametric and functional PEs, which enhances the model's ability to capture the high frequency information and both the auto and cross-correlations of the inputs. The experimental results on video reconstruction and other downstream tasks (video compression, video super-resolution and video denoisig) showcase the superiority of the proposed model compared to previous NR approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please fix or explain the weaknesses I mentioned above.", "broader_impact_concerns": "The authors adequately addressed the broader impact of the work in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers recommended acceptance after the author responses. The reviewers do believe that presentation of the core ideas can be improved. So the decision is acceptance, but we strongly encourage the authors to not only incorporate the responses in the text, but also try to improve the clarity of the presentation of the method.
Decision: Accept (poster)