Progressive Fourier Neural Representation for Sequential Video Compilation
Authors: Haeyong Kang
Jaehong Yoon
DaHyun Kim
Sung Ju Hwang
Chang D. Yoo
Venue: ICLR 2024 poster (2024)
Keywords: Fourier Neural Implicit Representation
Seqeuntial Video Compilation
Video Continual Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: Motivated by continual learning, this work investigates how to accumulate and transfer neural implicit representations for multiple complex video data over sequential encoding sessions.

Abstract: Neural Implicit Representation (NIR) has recently gained significant attention due to its remarkable ability to encode complex and high-dimensional data into representation space and easily reconstruct it through a trainable mapping function. However, NIR methods assume a one-to-one mapping between the target data and representation models regardless of data relevancy or similarity. This results in poor generalization over multiple complex data and limits their efficiency and scalability. Motivated by continual learning, this work investigates how to accumulate and transfer neural implicit representations for multiple complex video data over sequential encoding sessions. To overcome the limitation of NIR, we propose a novel method, Progressive Fourier Neural Representation (PFNR), that aims to find an adaptive and compact sub-module in Fourier space to encode videos in each training session. This sparsified neural encoding allows the neural network to hold free weights, enabling an improved adaptation for future videos. In addition, when learning a representation for a new video, PFNR transfers the representation of previous videos with frozen weights. This design allows the model to continuously accumulate high-quality neural representations for multiple videos while ensuring lossless decoding that perfectly preserves the learned representations for previous videos. We validate our PFNR method on the UVG8/17 and DAVIS50 video sequence benchmarks and achieve impressive performance gains over strong continual learning baselines.

Published as a conference paper at ICLR 2024
PROGRESSIVE FOURIER NEURAL REPRESENTATION FOR SEQUENTIAL VIDEO COMPILATION
Haeyong Kang, Jaehong Yoon, DaHyun Kim, Sung Ju Hwang, and Chang D. Yoo Korea Advanced Institute of Science and Technology (KAIST) [haeyong. kang, jachong. dahyun.kim, sjhwang82, cd_ cd_yoo| | @ ac.
ABSTRACT
Neural Implicit Representation (NIR) has recently gained significant attention due to its remarkable ability to encode complex and high-dimensional data into representation space and easily reconstruct it through a trainable mapping func- tion. However, NIR methods assume a one-to-one mapping between the target data and representation models regardless of data relevancy or similarity. This results in poor generalization over multiple complex data and limits their efficiency and scalability. Motivated by continual learning, this work investigates how accumulate and transfer neural implicit representations for multiple complex video data over sequential encoding sessions. To overcome the limitation of NIR, we propose a novel method, Progressive Fourier Neural Representation (PENR), that aims find an adaptive and compact sub-module in Fourier space to encode videos in each training session. This sparsified neural encoding allows the neural network to hold free weights, enabling an improved adaptation for future videos. In addition, when learning representation for new video, PFNR transfers the representation of previous videos with frozen weights. This design allows the model to continuously accumulate high-quality neural representations for multiple videos while ensuring lossless decoding that perfectly preserves the learned repre- sentations for previous videos. We validate our PFNR method on the UVG&/17 and DAVIS50 vidco sequence benchmarks and achieve impressive performance gains over strong continual learning baselines. The PFNR code is available https: / / git.bb. com/ httpeaaaaaeonaaeeeeeeeeaa/aeeeee PPPN/.
INTRODUCTION
Neural Implicit Representation (NIR) (Chen et al., 2021 Li et al., 2022; Chen et al., 2023; Mehta et al., 2021) is a research field that aims to represent complex data, such as videos or 3D objects, as continuous functions learned by neural networks. Instead of explicitly describing data points, NIR models compress high high-dimensiooal data from a oiw-dimensional embedding space. This process enables efficient data storage, compression, and synthesis. However, there's 's challenge: when compressing multiple pieces of data, each high-dimensional data needs to be encoded in the neural network, increasing linear memory requirements. To address this, Neural Video Representation proposes a solution, as explored in studies by Chen et al. (2021a; 2022b). This approach combines different videos into single video format and then reduces the model size through techniques like weight pruning and quantization post-training. While this method is effective for current data compression, it has significant limitation: it restricts the model's ability to adapt to new videos as they are added. To overcome this limitation, inspired by incremental knowledge transfer and expansion continual learning, we investigate a practical implicit representation learning scenario with video data, which aims accumulate neural implicit representations for multiple videos into single model under the condition that videos are incoming sequentially.
Continual Learning (CL) (Thrun, 1995; Rusu et al., 2016; Zenke et al., 2017; Hassabis et al., 2017) a learning paradigm where a model learns over multiple sequential sessions. It seeks to mimic human cognition, characterized by the ability to learn new concepts incrementally throughout lifetime without the degeneration of previously acquired functionality. Yet. incremental training of
"oorrepooniigg author

---

Published as a conference paper at ICLR 2024
NIR is challenging problem since the model detrimentally loses the learned implicit representations of past session videos while encoding newly arrived ones, a phenomenon known as catastrophic forgetting (McCloskey & Cohen, 1989). This issue particularly matters as neural representation methods for videos encode and reconstruct the target data stream conditioned to its frame indices. Then, the model more easily ruins its generation ability while learning continuously encode new videos due to the distributional disparities in holistic videos and their individual frames. Furthermore, the compression phase of neural representation makes it wayward to transfer the model to future tasks. Various approaches have been proposed to address catastrophic forgetting during continual learning, methods (Kirkpatrick et al., 2017; Chaudhry et al., 2020; Jung et al., 2020; Titsias et al., 2020; Mirzadeh et al., 2021) aim to keep the learned information of past sessions during continual training aided by sophisticatedly methods (Yoon et al., 2018 8; Mallya et al., 2018; Serrà et al., 2018; Wortsman et al., 2020; Kang et al., 2022a;b) propose to minimize the inter-task interference via newly designed architectural components, and (3) Rehearsal- based methods (Rebuffi et al., 2017; Chaudhry et al., 201 9b; Saha et al., 2021; Yoon et al., 2022; Sarfraz et al., 2023) involves replaying real synthesized data from previous sessions. However, these methods are less suitable for video data in CL due to the substantial memory and computational costs required to store and revisit high-dimensional samples. While conventional architecture-basee methods offer solutions to prevent forgetting, they are unsuited for sequential complex video processing as they reuse few or all adaptive parameters without finely discretized operations.
which are often conventionally classified as follows: (1) Regularization-bute
designed regularization terms, (2) Architecture-buse
To enhance neural representation incrementally on complex sequential videos, we propose a novel sequential video compilation method, coined Progressive Fourier Neural Representation (PFNR) to identify and utilize Lottery tickets (i.e., the weights of complicated oscillatory signals) in frequency space. To achieve this, we define Fourier Subnetwork Operator (FSO), which breaks down a neural implicit representation into its sine and cosine components (real and imaginary parts) and then selectively, identifies the most effective Lottery tickets for representing complex periodic signals. In practice, given a backbone and FSO architecture, our method continuously learns to identify input-adaptiee subnetwork modules and encode each new video into the corresponding module during sequential training sessions Our approach draws inspiration from the Lottery Ticket Hypothesis (LT'H) (Frankle & Carbin, 2019), which suggests that sparse subnetworks can maintain the performance of a dense network and from the Fourier Neural Operator concepts developed in studies by Li et al. (2020a;b); Kovachki et al. (2021); Tran el al. (2021). A challenge in this domain the inefficiency of continually searching for optimal subnetworks in Fourier space. This process typically requires iterative training steps with repeated pruning and retraining for each new task. To address this, PENR introduces a parametric score function. This function learns to produce binary masks for the real and imaginary components, enabling the identification of adaptive substructures for video encoding in each training session by selecting the top-pereentage weights based on their ranking scores. This allows PFNR to discover the optimal subnetwork during training, through joint training of weights and structure, thus avoiding the laborious processes of iterative retraining, pruning, and rewinding inherent in LTH. Crucially, PFNR permits overlapping subnetworks with those from previous sessions during training. This overlap allows the transfer of learned representations from earlier videos when relevant while keeping the weights for these earlier sessions fixed. As result, model can continuously expand its representation space across successive video sessions, ensuring that it maintains the encoding and generation quality of previous videos without any degradation (i.c.. remaining forgetting- free). This is achieved without needing a replay buffer to store multiple nigh-dimensional frames, a significant advancement the field.
Our contributions can be summarized as follows:
We suggest a practical learning scenario for neural implicit representation where the model encodes multiple videos continually in successive training sessions. Earlier NIR methods suffered from poor transferability to new videos due to the distributional shift of holistic video and frames.
We propose a cutting-edge method referred to as the Progressive Fourier Neural Representation for complex sequential video compilation. The proposed method continuously learns a compact subnetwork for each video session given a supernet backbone while preserving the generative quality of previous videos flawlessly in Fourier space.
We demonstrate the effectiveness of our method on multiple sequential video sessions by achiev- ing superior performance over conventional baselines in average PSNR and MS-SSIM without
2

---

Published as a conference paper at ICLR 2024
any quantitative qualitative degeneration in reconstructing previously encoded videos during sequential video compilation.
2 RELATED WORKS
Neural Implicit Representation (NIR). Neural Implicit Representations (NIR) (Mehta et al., 2021) are neural network architectures for parameterizing continuous, differentiable signals. Based on coordinate information, they provide way to represent complex, high-dimensional data with a small set of learnable parameters that can be used for various tasks such as image reconstruction (Sitzmann et al.. 2020; Tancik et al., 2020), shape regression (Chen & Zhang, 2019; Park et al., 2019), and 3D view synthesis (Mildenhall et al., 2021; Schwarz et al., 2020). Instead of using coordinate- based methods, NeRV (Chen et al., 2021a) proposes an image-wise implicit representation that takes frame indices inputs, enabling fast and accurate video compression. NeRV has inspired further improvements in video regression by CNeRV (Chen et al., 2022b), DNeRV (He et al., 2023), E-NeRV (Li et al., 2022), and NIRVANA (Maiya et al., 2022), and HNeRV (Chen et al., 2023). A few recent works have explored video continual learning (VCL) scenarios for the NIR. To tackle non-physical environments, Continual Predictive Learning (CPL) (Chen al., 2022a) learns mixture world model via predictive experience replay and performs test-time adaptation using non-parametric task inference. PIVOT (Villa et al., 2022) leverages the past knowledge present in pre-trained models from the image domain to reduce the number of trainable parameters and mitigate forgetting, CPL needs memory to replay, while PIVOT needs pre-training and fine-tuning steps. In contrast, along with the conventional progressive training techniques (Rusu et al., 2016; Cho et al., 2022), we introduce a novel neural video representation referred to as "Progressive Fourier Neural Representation (PFNR)", which utilizes the Lottery Ticket Hypothesis (LTH) to identify an adaptive substructure within the dense networks that are tailored to the specific video input index. Our PFNR doesn't use memory, a pre-trained model, or fine-tuning for sequential video representation compilation.
Continual Learning. Most continual learning approaches introduce extra memory like additional model capacity (Li et al., 2019; Yoon et al., 2020) or a replay buffer (Riemer et al., 2018; Chaudhry et al., 2019a; Buzzega et al., 2020; Arani et al., 2022; Sarfraz et al., 2023). However, several works have focused on building memory-efficient continual learners using pruning-based constraints to exploit initial model capability more compactly. CLNPP (Golkar al., 2019) selects important neurons for given task using l regularization to induce sparsity and freezes them to maintain performance. And pruned neurons are reinitialized for future task training. Piggyback (Mallya et al., 2018) trains task-specific binary masks on the weights given a pre-trained model. However, it does not allow for knowledge transfer among tasks, so the performance highly depends on the quality of the backbone model. HAT (Serrà et al., 2018) proposes task musk-specific learnable attention vectors to identify significant weights per task. The masks are formulated to layerwise cumulative attention vectors during continual learning. LL-Tickets (Chen et al., 2021b) recently suggests sparse subnetworks called lifelong tickets that perform well on all tasks during continual learning. The method searches for more prominent tickets from current ones if the obtained tickets cannot sufficiently learn the new task while maintaining performance on past tasks. However, LL-Tickets require external data to maximize knowledge distillation with learned models for prior tasks, and the ticket expansion process involves retraining and pruning steps. As a strong architecture- al., 2022a) jointly learns the model weights and task-adaptive binary masks during continual learning. It prevents catastrophic forgetting of previous tasks by keeping the model weights selected, called winning tickets, intact at the end of each training. However, WSN is inappropriate for sequential complex video compilation since it reuses few adaptive but sparse learnable parameters. To overcome the weakness of WSN, our PFNR explores more appropriate forget-free weights for representing complex video in Fourier space (Li et al., 2020a;b; Kovachki et al., 2021; Tran et al., 2021) 1) using a newly proposed Fourier Subnetwork Operator (FSO).
baseline, WSN (Kang
3 PROGRESSIVE FOURIER NEURAL REPRESENTATION
This section presents our proposed continual neural implicit representation method, named Pro- gressive Fourier Neural Representation (PFNR). Given a supernet backbone, where we follow the NeRV (Chen et al., 2021a) architecture for video embedding and decoding, PFNR aims to expand
3

---

Published as a conference paper at ICLR 2024
000
S
Conw Upocale
1680000
Frare rddes (1. 8.
Paoedding
Upocalo
H
imag
Fourfer Subruera Operator
(FSO)
Sham
NeRV Dlook with F300
NeRV
fropenn parsmeters
roused paramaters
S2
el remained trainable paramaters
Figure 1: Progressive Fourier Neural Representation (PENR): PENR takes time and video (session) indices as input and uses sparse Stem + NeRV Blocks with Fourier Subneural Operator (FSO) to output the whole image through multi-heads HN where of is hidden representation. We denote frozen, reused, and trainable parameters in training at session 2. Note that each video representation is colored. inference, we only need indices of session 8 and frame and session mask (subnetwork).
its representation space continuously by sequentially encoding multiple videos within the Fourier space. As new videos arrive in the model, PFNR jointly updates the binary masks (including real and imaginary parts) with neural network weights, searching for the adaptive subnetwork to encode given videos. Once a video session is completed, we 'freeze' the weights of the chosen subnetwork. This approach ensures that the quality of previously learned representations and generated outputs remains unaffected by future training sessions, even if the new subnetwork structure shares some weights with videos encoded earlier. While the weights learned in earlier video sessions are frozen, we enable our PFNR to transfer prior knowledge to future video tasks (i.e., forward transfer). This makes the model adapt new videos effectively by leveraging the representation of past videos (Please see Figure I).
Problem Statement. Let video at 8th session V, {w; ) E x x3 be represented by a function with the trainable parameter 0, fo + R. #xxx3 during Video Continual Learning (VCL), where T, denotes the number of frames in a video at session s, and s E fl....,(51 |S]}. Given a session and frame index and I, respectively, the neural implicit representation aims to predict corresponding RGB image of E HR x3 by fitting an encoding function
neural network: of = fe ([s; t], H.)) where H₃ is Sth, head. For the sake of simplicity, we omit H, in the following equations. Let's consider real-world learning scenario in which S ] N more sessions arrive in the model sequentially. We denote that D, fe,,,, Us. (=1 is the dataset of session s, composed of T, pairs of raw embeddings Cg.2 [e,: e,] E R1x160 and corresponding frames vf. Here, assume that D, for session is only accessible when learning session due to the limited hardware memory and preyeccipres issues, and session identity is given in the training and testing stages. The primary training objective in this sequence of N video sessions is to minimize the following optimization problem:
T Σ (((
1 NT,
0" minimize
where the loss function L(v;) is composed of & loss and SSIM loss. The former minimizes the pixel-wise RGB gap with the original input frames evenly, and the latter maximizes the similarity between the two entire frames based on luminance, contrast, and structure, as follows:
T. ((V.) = T. Σ allof l1 + (1 - a) - (f
(2)
where of is the output generated by the model f. For all experiments, we set the hyperparameter a to 0.7, and adapt PixelShuffle (Shi et al., 2016) for session and time positional embedding.
Continual learners f frequently use over- parammmtettee deep neural networks to ensure enough capacity for learning future tasks. This approach often leads to the discovery of subnetworks that perform as well as or better than the original network. Given the neural network parameters 0, the binary attention mask m that describes the optimal subnetwork for session such that [mī] is less
4

---

Published as a conference paper at ICLR 2024
than the model capacity follows as:
T
m
minimize m,, T, 1=1
[(f(es,1: & m.). ug) J. subject to |m,1 S
(3)
where session loss == [(v;) and c < [0] (used as the selected proportion % of model parameters in the following section). A robust model adhering to this condition known as WSN (Kang al., 2022a). However, WSN falls short in handling sequential complex video compilation due to its reliance on a limited set of adaptable yet sparse parameters in convolutional operators. In the following section, introduce novel Fourier Subnetwork Operator (FSO) address this limitation.
3.1 FOURIER SUBNUERAL. OPERATOR (FSO)
Conventional continual learner (i.c., WSN) only uses few learnable parameters in convolutional operations represent complex sequential image streams. To capture more parameter-efficient forget-free NIRs, the NIR model requires fine discretization and video-specific sub-parameters. This motivation leads us to propose a novel subnetwork operator in Fourier space, which provides it with various bandwidths. Following the previous definition of Fourier convolutional operator (Li et al., 2020a), we adapt and redefine this definition to better fit the needs the NIR framework. We use the symbol F represent the Fourier transform of function f. which maps from embedding space of dimension de = 1 x 160 to a frame size denoted as de. The inverse this transformation is Fourier-integra Subneural Operator (FSO),
represented by F-11 In this context, introduce symbolized as K., which is tailored to enhance the capabilities our NIR system (see Appendix A.1):
of) (e..:) F FR (Fe.))(e..),
(4)
where öf is hidden representation; R& is the Fourier transform of periodic subnetwork function which is parameterized by its subnetwork's parameters of real (Oreal @ m!cai) and imaginary (Oimag @ m.may). We thus parameterize Rg separately as complex-valued tensors of real and imaginary ΦF E (freal gimag]. One key aspect of the FSO that its parameters grow with the depth of the layer and the input/output size. However, through careful layer-wise inspection and adjustments for sparsity. we can find a balance that allows the FSO to describe neural implicit representations efficiently. In the experimental section, will showcase the most efficient FSO structure and its performance. Figure shows one possible PFNR structure of a single FSO. We describe the optimization in the following section.
Algorithm Progressive Fourier Neural Representation (PENR) for VCL
input: {D.) model weights of FSO 0. = (0, $rso), score weights of FSO p. (p, prso). binary mask Mo {0l), 00rso and layer-wise capacity e%.
1: randomly initialize 0, and p..
2: for session = 1, |S| do
then
randomly re-initialize p.. 5: end if
6: for batch b: D. do
7: obtain mask of the top-c% scores p. at each layer
compute L (f(ee. @ m.), br). where input embedding, ex,t [e.;e.).
9:
0. - 0.
BC
3 (1 - M.-1))
D- trainable weight update
70.
P. + P. - T pp.
Dc
weight score update
11:
end for
O. = O. mr
13:
M. f- Mo-1 Vm.
>> accumulate binary mask
14: end for
output: (0.) (
3.2
SEQUENTIAL VIDEO REPRESENTATIONAL SUBNETWORKS
Let each weight 0, {0, Φrso} be associated with learnable parameter we call weight score P. (p, )Fso), which numerically determines the importance of the weight associated with it; that
5

---

Published as a conference paper at ICLR 2024
is, weight with a higher weight score is seen as more important. We find a sparse subnetwork 6₂ of the neural network and assign it as a solver of the current session s. We use subnetworks instead of the dense network solvers for two reasons: (1) Lottery Ticket Hypothesis (Frankle & Carbin, 2019) demonstrates the existence of a competitive subnetwork that is comparable with the dense network, and (2) the subnetwork requires less capacity than dense networks, and therefore it inherently reduces the size of the expansion of the solver.
Motivated by such benefits, we propose a novel PFNR, the joint-training method for sequential video representation compilation, as shown in Algorithm 1. The pseudo-code explains how to acquire subnetworks within a dense network. We find 0 0. @ m, by selecting the top-c% weights from the weight scores p., where e is the target layer-wise capacity ratio in %; m, is session-dependent binary mask. Formally, m is obtained by applying a indicator function 1, on p. where 11 (p.)) = if p. belongs top-c% scores and 0 otherwise. Therefore, the subnetworks (0.) for all video session S are obtained by 0 0. O m,. Straight-through estimator (Bengio et al., 2013; Hinton, 2012; Ramanujan et al., 2020) is used to update p., which ignores the derivative of the indicator function and passes on the incoming gradient as if the indicator function were an identity function.
EXPERIMENTS
Our method is validated on benchmark datasets for Video Task-incremental Learning (VTL) and compared against various continual learning baselines. In all experiments conducted for this paper, we utilize multi-head configuration for continual video representation learning. This means the session identifier, denoted as s, is provided during the training and inference phases. Our experimental setups align with NeRV (Chen et al., 2021a) and HNeRV (Chen et al., 2023).
Datasets. 1) UVG of 8 Video Sessions: We experiment on eight sequential videos to validate our PFNR. The eight videos consist of one from the scikit-video and seven from the UVG dataset. The category index and order in UVG8 are as follows: 1.bunny, 2.beauty, 3.bosphorus, 4. 5.jockey, 6.setgo, 7.shake, y.yacht.
2) UVG of 17 Video Sessions: We conducted an extended experiment on 17 video sessions by adding 9 more videos to the UVG of 8 video sessions. The category index and order in UVG17 are follows: 1. bubuny 2.city, 3.beauty, 4.focus, 5.bosphorus, 6.kids, 7.bee, 8.pan, 9.jockey, 10. I.sego, 12.race 13.shake, 14.river, 15.yccht, 16.sumbath, 17.twilight. Please refer to the supplementary material.
Architecture. We employ NeRV as our baseline architecture and follow its details for fair compar- ison. After the positional encoding, we apply 2 sparse MLP layers on the output of the positional encoding layer, followed by five sparse NeRV blocks with upscale factors of 5, 2, 2, 2, 2. These sparse NeRV blocks decode x280x frames from the 16x9 feature map obtained after the sparse MI.P layers. For the upscaling method in the sparse NeRV blocks, we also adopt PixelShuffle (Shi et al., 2016). Fourier Subncural Operator (FSO) is used at the NeRV2 or NeRV3 layer, as depicted in Table 7. The positional encoding for the video index and frame index t is as follows:
T(s, t) =[ sin(bb cos TT),
cos Ts), cbs)
(5)
sin(b" (t), (b nt), sin(t'-1 mt), cos(bo mt) },
where the hyperparameters are set to b- 1.25 and = s0 such that I(s, t) E 1x160 As differences from the previous NeRV model, the first layer of the MLP has its input size expanded from 80 to 160 to incorporate both frame and video indices, and distinct head layers after the NeRV block are utilized for each video. For the loss objective in Equation 2, a is set to 0.7. We evaluate the video quality, average video session quality, and backward transfer with two metrics: PSNR and MS-SSIM (Wang et al., 2003). We implement our model in Py PyTorc and train it in full precision (FP32). All experiments are run with NVIDIA RTX8000.
Baselines. To show the effectiveness, we compare our PFNR with strong CL baselines: Single-Task, Learning (STL), which trains on single tasks independently, EWC (Kirkpatrick et al., 2017), which is regularized baseline, iCaRL (Rebuffi et al., 2017), and ESMER (Sarfraz et al., 2023) which are current strong rehearsal- based baseline, WSN (Kang et al., 2022a) which is a current strong architecture-basse baseline, and Multi-Task Learning (MTL) which trains multiple video sessions simultancously, showing the upper-boure of WSN. Except for STL, all models are trained and evaluated on multi-head settings where video session and time (s, t) indices are provided.

---

Published as a conference paper at ICLR 2024
Table I:
PSNR results with Fourier Subnucral Operator (FSO) layer (f-NeRVs) (detailed in Table 7) on UVG8 Vidco Sessions with average PSNR and Backward Transfer (BWT). Note that denotes our reproduced results.
Method
Video Sessions
Avg. PSNR/ BWT
STL, NcRV Chen al. (2021a))
39,66
36.28
38.14
-2.03
36.58
29.22
37.27
31.45
36.33/
EWC Kirkgairick al. (2017)"
10.19
11.15
14.47
8.39
12.21
10.27
9.97
23.98
12.58 12.58/-17.59
iCaRL Rebuffi et al. (2017)'
30.84
26.30
27.28
34,48
20.90
17.28
30.33
24.64
26,51/ -3.90
ESMER Sarfaaz et (2023))
31.71
23.09
24.15
28.03
17.30
13.81
12.45
24.57
21,92/
WSN" c- 50.0
34.05
32.28
29.98
32.88
22.15
18.61
27.68
23.64
27.66//0.0
PFNK c= 50.0 96, f-NeRV2
34.46
33.91
32.17
36,43
25.26
20.74
30.18
25.45
29.82/ 20.02/0.0
.FN 9. f-NeRV3
36.45
35.15
35.10
38.57
28.07
23.06
32.83
27.70
32.12/0.0
MTL (uppee-bound)
34.22
32.79
32.31
38.33
25,30
22.44
33.73
27.05
30.78/-
Table 2: MS-SSIM results with Fourier Subnueral Operator (FSO) layer (f-NeRV*) (detailed in Table 7) on UVG8 Video Sessions with average MS-SSIM, Backward Transfer (BTW) of MS-SSIM. Note that * denotes reproduced results.
Method
Video Sessions
Avg. MS-SSIM/
STI., NeRV Chen et al. (202
0.99
0.95
0.98
0.99
0.97
0.96
0.98
0.95
0.97/-
EWC Kirkpatrick al. (2017)'
0.22
0.23
0.35
0.10
0.27
0.19
0.21
0.79
0.30/ 40.62
iCaRL Rebufli al. (2017)'
0.94
0.80
0.82
0.97
0.59
0.57
0.92
0.81
0.80/ -0.11
ESMER Sarfruz et al. (2023)*
0.88
0.65
0.68
0.90
0.42
0.32
0.19
0.81
0.51 / -0.33
WSN" c
0.98
0.91
0.97
0.74
0.62
0.88
PFNR c = =50.0 S. f-NeRV2
0.98
0.93
0.93
0.99
0.83
0.75
0.92
0.84
0.90/0.0
PENR 50.0 f-NeRV3
0.99
0.94
0.97
0.99
0.88
0.84
0.95
0.90
0.93/0.0
MTL (upper-bound)
0.98
0.91
0.93
0.99
0.84
0.82
0.95
0.89
0.91/-
Training. In all experiments, we follow the same experimental settings as NeRV (Chen et al., 2023) and HNeRV (Chen et al., 2023) for fair comparisons. We train WSN, PFNR, NeRV (STL), and MTL using Adam optimizer with a learning rate 5e-4. For the ablation study on UVG8 and UVG17, we use cosine annealing learning rate schedule (Loshchilov & Hutter, 2016), batch size of 1, training epochs of 150, and warmup epochs of 30 unless otherwise denoted.
VCL's performance metrics of PSNR & MS-SSIM. We evaluate all methods based on the following continual learning metrics:
1.
Average PSNR or MS-SSIM (i.e., Ave. PSNR) measures the average of the final performances on all vidco sessions: PSNR or MS-SSIM- N Ex= AN... where AN,, is the test PSNR or MS-SSIM for session after training the final video session
2. Backward Transfer (BWT) of PSNR or MS-SSIM measures the video representation forgetting during continual learning. Negative BWT means that learning new video sessions causes the video representation forgetting of past sessions: BWT
N-T EN=
AN..
As,s-
Table 3: PSNR results with Fourier Subnueral Operator (FSO) layer (f (f-NeRV*) (detailed Table 7) on UVGI7 Video Sessions with average PSNR and Backward Transfer (BWT) of PSNR. Note that denotes our reproduced results.
Video Statioe
Avg. PANK
16
13
14
15
16
STL NERK Chencral paitta""
06
41.09
16.25
41.13
16.14
11.53
31:24
16.38
36.85
29.33
31.11
37.37
$1.18
41.30
30
EWC Kinkswackk (2017y"
12.71
11.40
11:35
10.33
13.19
8.67
10:12
28.29
11.30-16.3 11613
(CURL Retslli ci (20.77"
2431
25.25
22.04
16.35
29.37
17.92
21.43
16.42
21.60
19.40
1540
26.40
USMIR ot catza"
3037
36.35
25.79
21.38
23.26
33.44
29.25
15:32
16.73
23.79
15:35
15:33
18.11
1932
24.59
3009
WSN c.
34.77
31.98
23.08
22.44
35.45
20.45
21:23
PINK, c. 2011 /-NatVv
33.01
35.34
13:97
33.17
11.34
31.03
16.01
25.53
34.83
21.50
10
30.40
12.21
PINK, c. 301 N-Natvs
33.4
39.34
34.21
37.79
3408
27.37
38.17
28.79
26.36
36.18
22.97
24.36
32.50
30.22
27.02
28.15
35.06
38.72/ ILB
MTL (opeeal
34.03
24.33
27.83
34.59
22.37
00
4.1 COMPARISONS WITH BASELINES
PSNR & MS-SSIM. To compare PFNR with conventional representative continual learning methods such as FWC, iCaRL., ESMER, and WSN, we prepare the reproduced results, as shown in Table 1, 2, 3, and 4. The architecture-based WSN outperformed the regularized method and replay method. The sparseness of WSN does not significantly affect sequential video representation results on two sequential benchmark datasets. Our PFNR outperforms all conventional baselines including WSN and MLT (upper-boum) of WSN) on the UVG8 and UVG17 benchmark datasets. Moreover, our performances of PFNR with f-NeRV3 are better than those of f f-NeRV2 since f-NeRV3 tends to represent local textures, stated in the following Section 4.2. Note that the number of parameters of MLTT is precisely the same those of WSN.
7

---

Published as a conference paper at ICLR 2024
Table 4: MS-SSIM results with Fourier Subnucral Operator (FSO) layer (f-NeRV=) (dctailed in Table 7) on UVG17 Video Sessions with average MS-SSIM and Backward Transfer (BWT) of MS -SIM. Note that denotes our reproduced results.
Video Semiens
16
BWT
STL, NeV Chee ctal. (2021a)"
0.99
0.99
0.95
0..8
0.98
0.96
0.99
0.98
0.97
0.55
0.96
0.96
0.95
0.95
0.96
0.99
0..9
0.97 /-
EWC Kökpanick ctal. (2017)
0.25
0.24
0.44
0.24
0.40
0.29
0.15
0.17
0.26
0.26
0.17
0.34
0.04
0.30
0.33
0.31
0.91
0.30 -0.55
Rebuli (2077)
0.53
ESMER stsl (2023)"
0.51
0.63
WSS c 50000
0.55
0.97
0.39
0193
0.88
0.77
0.97
0.77
0.73
0.1
0.60
0.74
0.85
0.11
0.76
0.72
1193
0.84
PFNR. cm 30.0 S. /-NeRV2
0.97
0.98
0.92
0.95
0.92
0.84
0.98
0.88
0.82
0.93
0.73
0:82
0.92
0.89
0.94
0.87
0.97
0.90/
PRNR.6
(uppes bound)
0.90
0.31
PFNR's Compression. We follow NeRV's video quantization and compression pipeline (Chen et al., 2021a), except for the model pruning step, evaluate performance drops and backward transfer in the video sequential learning, as shown in Figure 2. Once sequential training done, our PFNR doesn't t need any extra prune and finetune steps, unlike NeRV. This point is our key advantage of PFNR over NeRV. Figure 2 (a) shows the results of various sparsity and bit-quantization on the UVGI7 datasets: the 8bit PFNR's performances
comparable with 32bit ones without significant video quality drop. From our observations, the 8-bir subnetwork seems to be enough for video implicit representation. Figure (b) shows the rate-distortion curves. We compare PFNR with WSN and NeRV (STL). For fair comparison, we take steps of pruning, fine-tuning, quantizing, and encoding NeRV. Our PFNR outperforms all baselines.
40
60
WSN,
55
NcRV
0-30.0% #NeRV2
WSN
35
PFNR, c=30.0%, 1NeRV3
PFNR (ours)
25
20
16
32
0.2
0.3
0.4
0.6
0.7
0.8
DR
Bits pixel (BPP)
(a) Quantization and Compression of PFNR
(b) PSNR BPP
Figure 2: PSNR v.s. Bits-per-pixel (BPP) on the UVGI7 datasets
Performance and Capacity. Our PFNR outperforms WSN and MTL, as stated in Figure (a). This result might suggest that properly selected weights in Fourier space lead generalization more than others in VCL. Moreover, to show the behavior of PSNR, We prepare a progressive PSNR's s capacity and investigate how PFNR reuses weights over sequential video sessions, as shown in Figure (b). PFNR tends to progressively transfer weights used for prior session to weights for new ones, but the proposition of reused weights gets smaller as video sessions increase.
NeRV
110
fel.fcl..r weight
80-
WSN
100
A-NRV3 wonwwelight real
55:
PFNR [ours]
(aparityy
80
90
(-AiVV con..aeighttimma
NeRVS, conuweightt
40
35.
30
20-
20
15
9 10 11
12
33
34
15
16
10
2
3
7
8
10 11
12
13
15 16 17
Video Sessions
Video Sessions
(a) PPNR's performance with 30.0%
(b) PPNR capacity with 30.0%
Figure PFNR's Comparison PSNR with others and layer-wise accumulated capacities on UVG17 dataset. Note that, in (b), green represents the percentage of reused subnetwork's 's parameters of Stem, fNNeV33, and NeRV5 at the current session (s) obtained at the past (s-1) video sessions
4.2 PFNR's VIDEO REPRESENTATIONS
We prepare the results of video generation shown in Figure 5. We demonstrate that sparse solution (PFNR with c = 30.0%, f-NeRV3) generates video representations sequentially without significant performance drops. Compared with WSN, PFNR provides precise representations.
8

---

Published as a conference paper at ICLR 2024
To find out the results, we inspect the layer-wo wise representations shown in Figure 4, which provides essential observations that PFNR tends to capture local textures broadly at the NeRV3 layer while WSN focuses on local objects. This behavior of PFNR could lead to more generalized performances. Moreover, we conduct an ablation study to inspect the best sparsity of f-NeRV3 while holding the remaining parameters' sparsity (c=50.0 %),, shown in Figure 10. Please refer to the supplementary materials for comparisons with baselines.
NeRV3
NeRV4
NeRV5
NeRV6
Head
WSN, 5000.0 (28.95, PSNR)
PFNR, C=50.0%, /-NeRV2 (31.24, PSNR)
PFNR, 55000% 3-NeRV3 (34.05, PSNR)
Figure 4: PENR's Representations of NeRV Blocks with c 50.0% on the UVG17 dataset.
t=
t
WSN (31.00, PSNR)
PFNR, f-NeRV3 (34.21, PSNR)
Figure 5: PFNR's ' Video Generation (from t=0 to t=3) with = 30.0% on the UVG17 dataset.
CONCLUSION
Neural Implicit Representations (NIR) have gained significant attention recently due to their ability to represent complex and high-dimensional data. Unlike explicit representations, which require storing and manipulating individual data points, implicit representations capture information through learned mapping function without explicitly representing the data points themselves. While they often compress neural networks substantially to accelerate encoding/decoding speed, yet existing methods fail transfer learned representations to new videos. This work investigates the continuous expansion of implicit video representations as videos arrive sequentially over time, where the model can only access the videos from the current session. To tackle this problem, we propose a novel neural video representation, Progressive Fourier Neural Representation (PFNR), that finds an adaptive substructure from the supernet to the given video based on Lottery Ticket Hypothesis in a complex domain. At each training session, our PFNR transfers the learned knowledge of the previously obtained subnetworks to obtain the representation of the current video without modifying past subnetwork weights. Therefore, it can perfectly preserve the decoding ability (i.e., catastrophic forgetting) previous videos. We demonstrate the effectiveness of our proposed PFNR over baselines on the novel UVG8/17 and DAVIS50 video sequence benchmark datasets.

---

Published as a conference paper at ICLR 2024
REFERENCES
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual
learning method based on complementary learning system. arXiv preprins arXiv:2201. 2022.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, 2013.
Pictro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. Advances in neural information processing systems, 33: 15.15099-15900 2020.
Arslan Chaudhry, Marc' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In Proceedings of the International Conference on Learning Repre-
sentations (ICLR), 2019a. 3
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and M Ranzato. Continual learning with tiny episodic memories. arXiv preprint 1XXiv:1902.104800 2019b. 2
Arslan Chaudhry, Nacemullah Khan, Puneet K Dokania, and Philip HS Torr. Continual learning in low-rank orthogonal subspaces. In Advances in Neural Information Processing Systems (NeurIPS),
2020. 2
Geng Chen, Wendong Zhang, Han Lu, Siyu Gao, Yunbo Wang, Mingsheng Long, and Xiaokang Yang. Continual predictive learning from videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10728-10737, 2022a. 3
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34:21557-21568,
2021a. 3, 6, 7, 17, 18, 19, 20, 22, 23
Hao Chen, Matt Gwilliam, Bo He, Ser-Nam Lim, and Abhinav Shrivastava. Cnerv: Content-adaptive neural representation for visual data. arXiv preprint arXiv:2211.1042) 2022b. 1,3
Hao Chen, Matt Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. Hnerv: A hybrid neural representation for videos. arXiv preprint arXiv:2334.006633 2023. 6,7,
Tianlong Chen, Zhenyu Zhang. Sijia Liu, Shiyu Chang, and Zhangyang Wang. Long live the lottery: The existence of winning tickets in lifelong learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021b.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Junwoo Cho, Seungtae Nam, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Streamable neural fields. In European Conference on Computer Vision, pp. 595-612. Springer, 2022.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Proceedings of the International Conference on Learning Representations (ICLR), 2019. 2, 6
Siavash Golkar, Michael Kagan, and Kyunghyun Cho. Continual learning via neural pruning. arXiv preprint
2019. 3
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inssiree artificial intelligence. Neuron, (562):245-288, 2017. 1
Bo He, Xitong Yang, Hanyu Wang. Zuxuan Wu, Hao Chen, Shuaiyi Huang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. Towards scalable neural representation for diverse videos. arXiv preprint a30iv:2303. 20l.:233...1224.. 2023. 3
10

---

Published as a conference paper at ICLR 2024
Geoffrey Hinton. Neural networks for machine learning, 2012.
Sangwon Jung, Hongjoon Ahn, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeuriPS), 2020.
Haeyong Kang, Rusty John Lloyd Mina. Sultan Rizky Hikmawan Madjid, Jaehong Yoon, Mark Hasegawa-Johnson, Sung Ju Hwang, and Chang D Yoo. Forget-free continual learning with winning subnetworks. International Conference on Machine Learning, pp. 10734-10750. PMLR, 2022a. 6,
Haeyong Kang, Jaehong Yoon, Sultan Rizky Hikmawan Madjid, Sung Ju Hwang, and Chang D Yoo. On the soft-subnetwork for few-shot class incremental learning. arXiv preprint arXiv:2209.07559, 2022b.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. 2017. 2, 6, 7, 8, 16, 18, 19, 22, 23
Demis Hassabis,
Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Learning maps between function spaces. arXiv preprint arXiv:2 00..0088811111 2021. 3,
Xilai Li, Yingbo Zhou, Tianfu Wu. Richard Socher, and Caiming Xiong. Learn to grow: continual structure learning framework for overcoming catastrophic forgetting. In Proceedings of the International Conference on Machine Learning (ICML), 2019. 3
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expedite neural video representation with disentangled spatial-temporal context. Computer Vision-ECCC 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXV, pp. 267-284. Springer, 2022. 1,3
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895 2020a. 2, 3, 5, 13, 14, 15, 22
Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Neural operator: Graph kernel network for partial differential equations. arXiv preprint 2rXXh:2003.03885 2020b. 2, 3, 13, 22
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0390 2016.
Shishira R Maiya, Sharath Girish, Max Ehrlich, Hanyu Wang, Kwot Sin Lee, Patrick Poirson, Pengxi- ang Wu, Chen Wang, and Abhinav Shrivastava. Nirvana: Neural implicit representations of videos with adaptive networks and autoregressive patch-wise modeling, arXiv preprint arXiv:2212. 14593. 2022. 3
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), 2018. 2,3
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989. 2
Ishit Mehta, Michaël Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In Proceedings of the IEEÉ/CVF International Conference on Computer Vision, pp. 14214-14223,
2021. 1,3
11

---

Published as a conference paper at ICLR 2024
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106 106, 2021. 3
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. 2
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Sorkine-Hornung. F. Perazzi, J. Pont-Tuset B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine- Hornung. A benchmark dataset and evaluation methodology for video object segmentation. In Computer Vision and Puttern Recognition, 2016. 16
Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's 's hidden in a randomly weighted neural network? In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 6
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 2001-2010, 2017. 2, 6, 7, 8, 17, 18, 19, 22, 23
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning learn without forgetting by maximizing transfer and minimizing interference. arXiv preprint arXiv: 1810. 19900, 2018. 3
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint 1rXiv:1606.04671, 2016. 1,3
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2021. 2
Fahad Sarfraz. Elahe Arani, and Bahram Zonooz. Error sensitivity modulation based experience replay: Mitigating abrupt representation drift in continual learning. In The Eleventh International Conference Learning Representationss 2023. URL https: http://ppppweeeeeeeeepeeeeee//ep net forrmmmm Addzlbcii00 701923. 2, 3, 6, 7, 8, 17, 18, 19, 20, 23
/
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 33: 20154-20166, 2020. 3
Joan Serrà, Didac Suris. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the International Conference on Machine Learning (ICML), 2018. 2,
Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real- sial-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1874-1888 1883, 2016. 4,
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473. 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances Neural Information Processing Systems, 33:7537-7547, 2020,
Sebastian Thrun. A Lifelong Learning Perspective for Mobile Robot Control. Elsevier, 1995.
12

---

Published as a conference paper at ICLR 2024
Michalis K Titsias, Jonathan Schwarz, Alexander G de G Matthews, Razvan Pascanu, and Yee Whye Teh. Functional regularisation for continual learning with gaussian processes. In Proceedings of the International Conference Learning Representations (ICLR), 2020. 2
Alasdair Tran, Alexander Mathews, Lexing Xie, and Cheng Soon Ong. Factorized fourier neural operators, arXiv preprint arXiv:2111.13802 2021. 2, 3, 22
Andrés Villa, Juan León Alcázar, Motasem Alfarra, Kumail Alhamoud. Julio Hurtado, Fabian Caba Heilbron, Alvaro Soto, and Bernard Ghanem. Pivot: Prompting for video continual learning. arXiv preprint ar wtx.::2222.048222 2022. 3
Zhou Wang, Eero Simoncelli, and Alan C Bovik. Multiscale structural similarity for image quality assessment. In The Thrity- Seveeth Asilomar Conference on Signals, Systems & Computers, 2003, Ieee, 2003.
volume 2, pp. 1338-1400
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In Proceedings of the International Conference on Learning Representations
(ICLR), 2018. 2
Jachong Yoon, Sachoon Kim, Eunho Yang. and Sung Ju Hwang. Scalable and order-robust continual learning with additive parameter decomposition. In Proceedings of the International Conference Learning Representations (ICLR), 2020. 3
Jaehong Yoon, Divyam Madaan, Eunho Yang, and Sung Hwang. Online coreset selection for rehearsal-basee based continual learning. In Proceedings of the International Conference on Learning Representations (ICLR), 2022. URL https:/ / /pwprevveweww.eee/////// net forum?ic E9D SWNGNNV. 2
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995. PMLR, 2017.
A APPENDIX
A.1 FOURIER SUBNEURAL OPERATOR (FSO)
To elucidate the Fourier Subneural Operator (FSO) delineated in Equation 4, We'll delve into a comprehensive review of the Neural (Li et al., 2020b) Operator and Fourier Operator (Li et al., 2020a), focusing its discretization methodology.
A.1.1 DISCRETIZATION FOR NEURAL/FOURIER OPERATORS
Let's consider D, {-1,... ..) D n-point discretization of the domain D. Given this setting, we have observation dg|D, ngnx UjID, E gaxd. pertaining to finite collection of input-output pairs indexed by J. The goal of achieving discretization-inaarte operator implies that the operator is capable of generating response u(x) for any point x in the domain D, even for instances where r may not be an element of the discretized subset Dy. This characteristic ensures the neural operator maintains its predictive and functional integrity across the continuous domain D, notwithstanding the specific discretization points representing D.
with a neural
A.1.2 NEURAL OPERATOR
The neural operator, described by Li et al. (2020b) is formulated denoted by Up ト 11 H ... - Vy where Uy (for j = 1, T 1) represents a sequence of functions. Each function this sequence yields values the space -d As the process iterates, the transformation from state vy to the state U-+1 is defined by the interplay of two distinct
iterative architecture
13

---

Published as a conference paper at ICLR 2024
types of operations: non-local integral operator K. and a local, nonlinear activation function Specifically, the update from bt to 2/+1 in each iteration is articulated as the composition these two operations, mathematically represented follows:
(6) A θK + C(U(D; (,(D;; maps to bounded linear operators the function space (D; This mapping is parameterized by & E Og. Additionally. the function W R - Rd a linear transformation, and o + R is non-linear activation function whose action is defined coatned-commoentt-ee wise..
(x) == o(Wer (x) + (K(a; ())(())), V
In this formulation, functional operator
For increasing integration in defining complex, ficxible functional mappings, Li et al. (2020a) choose K((a; x) be kernel integral transformation parameterized by neural network, The kernel integral operator mapping in Equation is defined by
(K(a; ())))())= =- J, k(z, y, a(x), a(y); ()vr())d Vo E D
where kig (R((ddd)) + Rd,xd. xd. is neural network parameterized by & E Ox. Here, Kis plays the role of a kernel function, which we learn from data. Together Equation 6 and Equation 7 constitute generalization of neural networks to diment--ddmeniioall spaces.
By removing the dependence the function and enforcing shift-invariance property in the kernel function (s( y) = Kg (x - y), the operator simplifies into a convolution operator. This transformation aligns the kernel integral operator with the principles of fundamental solutions and leverages the intrinsic properties of convolution operations.
(K(a; ())))))) J ko(r (t) Ya D
This fact by parameterizing K₂ directly is exploited in Fourier space and used in the Fast Fourier Transform (FFT) to compute Equation efficiently.
A.1.3 FOURIER OPERATOR
Li et al. (2020a) suggest replacing the kernel integral operator in Equation 7, by a convolution operator (see Equation 8) defined in Fourier space. Let denote the Fourier transform a function f: F:D + dd and F-1 its inverse then
(F), = /. [,(T) 2ixcee da, (F-1))(0)= = 1 [1)) dk:
j = 1, dp where i = V is the imaginary unit. By letting (2, y: a(x), a(y)) kx in Equation and applying the convolution theorem, the convolutional operation Fourier space follows as:
(K(a; (z) -F-((((k))) (())))()). Ve Furthermore, the Fourier integral operator defined directly by parameterizing Ki in Fourier space as follows:
(K(()):)(z) = =F (Fr)))() Va E D
(9)
where Rs is the Fourier transform of a periodic function (i.e., sine and cosine functions in the time domain) ls D - Rddx parameterized by & E Ox.
For given frequency mode k E D, the Fourier transformation of ut denoted (k) E Cd. Similarly, (k) E Cd. Cd, representing complex-valuee matrix associated with each frequency mode kt Key aspects of this setup include:
(Periodicity and Fourier Series Expansion): Given the periodic nature of k. it can be represented by a Fourier series. This allows for the analysis and computation to be conducted in terms of discrete modes k E Zd.
(Truncation and Finite-dimensional Parameterization): To manage the complexity and ensure computational feasibility, the Fourier series is truncated at maximal number of modes kmozz- The truncation is quantified by the set Zymarz which includes all modes E 2d that satisfy the condition 1kyl < kmax.j for each dimension j 1.... d. The operator R. is parameterized as complex-valle valued tensor of shape (kmaz x d, x d.). comprising the collection truncated Fourier modes.
14

---

Published as a conference paper at ICLR 2024
(Conjugate Symmetry and Real-value of k): Due to the real-valued nature of k. conjugate symmetry is imposed on the Fourier coefficients. This is a fundamental property of the Fourier transform of real-valued functions, ensuring that the resulting inverse Fourier transform yields a real-valued function.
(Choice of Zkmer and Efficiency Considcrations): While the canonical choice for low- urefrreuency modes typically involves upper bound the ---morm of & zd, the set Zimaa is in this work chosen based on different criteria for efficient implementation.
A.1.4 FOURIER SUBNUERAL. OPERATOR (FSO)
(Discretization for VIL): One difference exists between prior physical modeling for Neural/Fourier Operators and neural implicit representation for Fourier Subneural Operator (FSO). Following the previous definition (Li et al.. 2020a), Vt is the temporal length. In the VIL setting, the function learns a time-specific continuous hidden output (an implicit representation, ;;)) given discrete session and time indices (session time, t).
(Fourier Subneural Opeartor (FSO): Conventional continual learner (i.e., WSN) only uses a few learnable parameters in convolutional operations to represent complex sequential image streams. To capture more parameter-efficient forget-free NIRs, the NIR model requires fine discretization and video-specific uub-parameters. This motivation leads us to propose novel subnetwork operator in Fourier space, which provides with various bandwidths. Following the previous definition of Fourier convolutional operator (Li et al., 2020a), we adapt and redefine this definition to better fit the needs of the NIR framework. We use the symbol F to represent the Fourier transform of function f. which maps from embedding space of dimension de 1 x 160 to a frame size denoted as de. The inverse of this transformation is represented by FF-1. In this context, we introduce our Fourier-integral Subneural Operator (FSO), symbolized as K., which is tailored to enhance the capabilities of our NIR system:
(K(()))) =F (est))
(10)
where of is a hidden representation, as shown in Figure I: Rs is the Fourier transform a periodic subnetwork function, Ks stated in Equation 8 which is parameterized by its subnetwork's s parameters of real (@real @ reaa) and imaginary (Ginag @ m miaag)) ). We thus parameterize R& separately as complex-valued tensors of real and imaginary #F E (Oreal (imagj, j. One key aspect of the FSO that its parameters grow with the depth of the layer and the input/output size since the operator Rgg, stated in Equation 9, is parameterized as a complex- vvauudd tensor of shape (kmar de x de). However, through careful wiier-wise inspection and adjustments for sparsity, we can find a balance that allows the FSO describe neural implicit representations efficiently. In the experimental section, we will showcase the most efficient FSO structure and its performance. Figure shows one possible PENR structure of single FSO.
A.2 DATASETS
1) UVG 8f Video Sessions: For "Big Buck Bunny" frames collected from the scikit-video, we use the frames provided with the NeRV official code. "Big Buck Bunny" comprises 132 frames of 720 x 1080 resolution. The frames for the other seven videos, collected from the UVG dataset, are extracted from YUV Y4M videos, and further information can be found in the implementation details. As shown in Table 5, the seven videos have 1920 x 1080 resolution, with the shaking video comprising 300 frames and the other 6 videos containing 600 frames each. These videos are captured at 120 frames per second (FPS), and the duration of the shaking video 2.5 seconds. while the duration of the other 6 videos is 5 seconds. For convenience, the video titles in the UVG of 8 Video Sessions are abbreviated, and their corresponding full titles the UVG dataset are as follows: 1.bunny: Big Buck Bunny, 2. 2.beauty: Beauty, 3. 3.bosphorus Bosphorus, 4.bee HoneyBee. 5.jockey Jockey, 6.setgo ReadySetGo, 7.shake ShakeNDry, 8. 8.yacht: YachtRide.
2) UVG of 17 Video Sessions: Compared to the UVG of 8 video sessions, the other nine videos are all collected from the UVG dataset. The frames for these videos are extracted from YUV RAW videos with resolution of 1920x 1080. Further information can be found in the implementation details. As shown in Table 6, the sunbath video consists of 300 frames at 50 FPS and 12 seconds, the lips video consists of 600 frames 120 FPS and 5 seconds, and the other seven videos comprised of 600 frames
15

---

Published as a conference paper at ICLR 2024
Table 5: the UVG8 Video Sessions.
Video Sessions
Categories
bunny
beauty
bosphorns 120
jockey
setgo
shake
yacht
FPS
120
120
120
120
120
120
Longth (sec)
5
5
5
2.5
5
Num. of Frames
132
600
600
600
600
600
300
600
Resolutions
1080
1920
1080
1920
1920
1080
1080
1920
1080
1920
1920
1080
at 50 FPS and 12 seconds. The full names of each video in the UVG dataset are follows: 2.city CityAlley, 4.focus FlowerFocus, 6.kids FlowerKids, 8.pan FlowerPan, 10.lips: Lips, 12.race: RaceNight, 14.river RiverBank, 16.sunbath. SunBath, 17. 17.nilhghh: Twilight.
Table 6: the UVG17 Video Sessions.
Video Seccions
Categories
bunny
city
beauty
Focus
busphorus 120
kids
bet
pun
juckey
FPS
50
120
20
50
120
50
120
Longth (s0c)
12
5
12
5
12
5
12
Num. of Frames
132
600
600
600
600
600
600
600
Reselutions
720 1080
1920
1920
1080
1080
1080
Video Sessions
16
Czeegorie
lips
seigo
race
sbake
river
yacht
sunbath
twilight
FPS
120
120
50
120
50
120
50
50
Longth (s0c)
5
12
2.5
12
6
12
Num. of Frames
600
600
600
300
600
600
500
600
Reselutions
1920
1060
1920
1920
1080
1920
1080
1920
3) DAVIS (Densely Annotated Video Segmentation) of 50 Video Sessions: We prepare a large-scale sequential video dataset, the Densely Annotation Video Segmentation dataset (DAVIS) (Perazzi et al., 2016). To validate our algorithm and investigate the limitations, we conducted the experiments on 50 video sequences with 3455 frames with a high-quality, reghurrsoo (1080p).
A.3 IMPREMENTATION DETAILS
1) 7 Videos in UVG of 8 Video Sessions: To utilize the same video frame with NeRV, we downloaded 7 videos from the UVG dataset and employed the following commands to extract frames from the YUV Y4M videos.
Download file : [title] 3840x2160 Sbit YUV Y4M
Command ffmpeg -i [amm] [path]/fe 05d.png
2) Videos UVG of 17 Video Sessions: To expand our usage of videos, we acquired an additional videos from the UVG dataset that are exclusively available as YUV RAW videos with a resolution of 3840x2160. We then extracted and resized the frames using the following command.
Download file: [title] 3840x2160 10bit YUV RAW
* Command ffmpeg -s 3840x2160 -pix fmt yuv420p10le -i [file name] -vf sca1e=1920:1080 pix fmt rgb24 [path//f% 05d.png
Digiturk provides the video contents of the UVG dataset. The dataset videos are available online at https: //tl:s//ulltee /tluttlttaiiiii deo. fi/ 4maim
EWC (Kirkpatrick al., 2017). We trained EWC on the two novel benchmark datasets regular- ized baseline. When training with new video, the EWC penalty was adopted as regularization term to alleviate catastrophic forgetting. The importance of the parameter was calculated through the diagonal component of the Fisher Information matrix, and the EWC penalty increased as the difference in the vital parameter increased as follows:
LE(vE) [.()) + > Σ Fr(O, - 02,))
(11)
where LE is the total loss for EWC learning, F represents the Fisher information matrix, À is a hyperparameter determine the importance of the previous video, i denotes each parameter, and 0z
16

---

Published as a conference paper at ICLR 2024
Table 7: PFNR's s architecture of Fourier Subneural Operator (FSO), f-NeRVs, Note that PE denotes positional encoding.
Ouiput Size (C x W x H)
Number of parameters
layer
Module
Upscale Factor
PE (w frame index)
80 1 x
PE (w vidco index)
80 1 x
1 STEM of Tel
512 x 1 x
81,920
STEM of fc2
112 x 16 x 9
8,257,536
2 NeRV2 block of conv
5x
112 x 80 x 45
2,825,200
f-NeRV2
1x
112 x 80 x 45
1,605,632
NeRV3 block of conv f-NeRV3
2x
96 x 160 x 90
387,456
3
1x
96 x 160 x 90
37,847,040
4
NeRV4 block of conv
2x
96 x 320 x 180
332,160
5
NeRV5 block of conv
2x
96 x 640 x 360
332,160
6
NeRV6 block conv
2x
96 x 1280 720
332,160
Multi-head layer
7 for a video session
x 1280 720
291
denotes the parameter after training with the previous video. We randomly sampled 10 frames per video to compute the Fisher diagonal and stored them in replay buffer. The hyperparameter À was experimentally to 2c6 to scale the EWC penalty.
iCaRL (Rebuffi et al., 2017). We trained iCaRL as a rehearsal-based baseline on the two novel benchmark datasets. To replay previous videos, we store a total of m 800 frames in the replay buffer as exemplar set, and as the training progresses, we save m/s frames per video. The exemplar management method is similar to that in iCaRL, where we compute the average feature map within the video and select m/s s video frames that approximate the average feature map. For knowledge distillation, we randomly sample frames from the exemplar set of previous videos at each learning step and performed training with the current video, as follows:
1 + L(7))
Lo(x;) L(v;) +
(12)
where of is frame sampled from example set of ith video, and is hyperparameter experimentally set to 0.5.
ESMER (Sarfraz et al., 2023). We trained ESMER on the two novel benchmark datasets as a current strong rehearsal-based baseline. Error-Sensitie Reservoir Sampling (ESMER) maintains episodic memory, which leverages the error history to pre-select low-loss samples as candidates for the buffer of 800 samples. At this time, we didn't use any noisy labels when training ESMER. We observed that the ESMER could not reduce forgetting in representations. It seems better suited for retaining information in image classification tasks rather than neural implicit representations. Lastly, compared with iCaRL., FSMER replies buffer at each iteration, leading to ineffective training cost and performance shown in Table 9.
Table 8: Statistics of FPS and PSNR on the UVGI7 Video Sessions
Method
FPS/ Resolution
Avg. PSNR
FPS Resolution
Avg. PSNR
STL,, NeRV Chen al. (2021a)"
120 1920 1080
35.97
35.56
iCaLL Rebulli al. (2017)*
120 x 1080
21.94
21.06
ESMER Sarfraz et (2023)"
120 1920 x 1080
21.43
50 1920 1080
19.25
WSN" PFNR. f-NeRV2
120 1920 1080
27.01
50 1920 1080
25.95
120 1920 x 1080
29.65
50/ 1920
1080
28.36
PFNR, f-NeRV3
120 1920 x 1080
31.53
50 1920 1080
31.47
MTL (upper-bound)
120 1920 x 1080
29.64
50/ 1920 x 1080
28.83
A.4 ARCHITECTURE
In this paper, we set NeRV as the baseline architecture, and we present a detailed overview of our architecture, as shown in Table Our architecture has two differences compared to the previous
17

---

Published as a conference paper at ICLR 2024
NeRV. Firstly, to incorporate the outputs from the positional encoder of both the video index and frame index, we expand the input size of the first layer in the MLP from 80 to 160. Secondly, to enable Multi-Task Learning (MTL). we employ distinct head layers (multi-heads) for each video after the NeRV block. Apart from these modifications, our architecture remains consistent with the previous NeRV architecture. We stack five NeRV blocks with upscaling factors of 5, 2, 2, 2, and 2, respectively. The lowest channel width for output feature maps in the NeRV block is also set to 96. Comparing our architecture the baseline NeRV, the number of parameters increases by 0.3433% to 22,567,560 for eight videos. Similarly, for seventeen videos, the number of parameters increases by 0.3642% to 12,570,179.
A.5 ADDITIONAL RESULTS
Dataset statistic of FPS and PSNR. As shown in Table 5 and Table 6, we have extracted sample frames according to each video's 's FPS and Length (sec) during training. There are two kinds of videos: 300 and 600 frames. We train each video for 150 epochs. Statistically, the average PSNR of 120 FPS was better than those of 50 FPS, as shown Table 8. When considering 30 PSNR is known as a sufficient resolution, 50 FPS-based sequence training could be adequate using the proposed PENR. Moreover, the number of frames does not seem to be a critical factor to PSNR when considering that the PSNR of video session 1 (bunny), with 132 (720 x 1080) frames, is greater than the 30 PSNR score.
PFNR's Structure. We investigate the most expressive, effective, and efficient structure with Fourier Subneural Operator (FSO, Equation 4) for progressive neural implicit representations. To do so, we prepare the layer-wise FSO maintain the output size of the baseline layer as shown in Table 7: the number of parameters of a spectral layer depends on input and output size, so as the layer increases, the parameters also increase. Table 9 shows the effectiveness of PFNR with spectral layers in terms of PSNR, BWT, and CAP. To acquire neural implicit representations, PFNR explores more diverse parameters than WSN regarding the same capacity.
Table 9: PSNR results with Fourier Subnueral Operator (FSO) layer (f-NeRV*) (NeRV block, Table 7) on UVG8 Video Sessions with average PSNR and Backward Transfer (BWT), and Capacity (CAP). Note that denotes our reproduced results.
Method
Video Sessions
Avg. PSNR/
CAP
BWT
STI, NeRV Chen et al. (2021a)
39.66 39.66
36.28
38.14
42.05
36.58
29.22
37.27
31.45
36,33/
800.00 %
EWC Kirkpatrick et al. (2017)'
10.19
11.15
11.17
8.59
12.21
10.27
9.97
23.98
12.58/-17.55
100,00 %
iCaRL Rehuffi al. 12017)"
30.84
26.30
27.28
20.90
17.28
100.00
ESMER Sarfraz ct al. (2023)'
31.71
23.09
24.15
28.03
17.30
13.81
12.45
24.57
21.92/ -9.99
100.00 %
WSN". 10.0 $
27.81
30.66
29.30
33.06
22.16
18.10
27.81
22.97
26.52
28.00 %
WSN' 30,0%
29.92
18.96
59.00
WSN". = -5000 %
34.05
32,28
29.98
32.88
22.15
18.61
27.68
23.64
27.66 /0.0
77.00 %
WSNN: 70.0 $
35.62
32.08
29.40 16.46
31.57
21.60
18.13
27.33
22.61
27.28 /0.0
91.00% %
PFNR 10.0%% f-NeRV2
30.30
21.10
21.76
28.10
33.83
PFNR e 30.0%, J-NeRV2
31.99
33.56
31.82
36.61
25.28
20.97
31 37.07
25.73
29.63 70.0
76.76 %%
PFNR == 50.0 %. N-NeRV2
34.46
33.91
32.17
36.
25.26
20.74
30.18
25.45
29. /0.0
102.64 9
PPNR,
f-NeRV2
36.04
33,46
31.05
32.57
23.40
19.41
28.31
24.31
28.57 /0.0
111.66 %.
MTL (upper-bouxd)
34,22
32,79
32.34
38.33
25.30
22.44
33.73
27.05
30.78/
100.00 %
Large-scale Sequence. We conducted the large-scale video sequence training to show its effectiveness and investigate the limitation of architecture parameters as shown in Table 10. The overall PFNR performances (STL, WSN, PFNR, and MTL) on the DAVIS50 dataset on the UVG8/17 dataset are lower than those on the UVG17 dataset. However, the performance trends of PFNR observed on the DAVIS50 dataset are consistent with the experimental results obtained from the UVG8/17 dataset. Regarding sequential and multi-task learning, the current performance falls short of achieving the target of 30 PSNR. This indicates a need for future work focused on designing model structures capable of continual learning on large-scale datasets.
Forget-free Transfer Matrix. We prepare the transfer matrix to prove our PFNR's forget-freeness and to show video correlation among other videos, as shown in Figure 6 on the UVG17 dataset; lower triangular estimated by each session subnetwork denotes that our PFNR is forget-free method and upper triangular calculated by current session subnetwork denotes the video similarity between source and target. The PFNR proves the effectiveness from the lower triangular of Figure 6 (a) and
18

---

Published as a conference paper at ICLR 2024
Table 10: PSNR results with Fourier Subnueral Operator (FSO) layer (f (f-NeRV*) (NeRV block, Table on DAVIS50 Video Sessions with average PSNR and Backward Transfer (BWT), and Capacity (CAP). Note that denotes our reproduced results.
Method
Video Sessions
Avg. PSNR/ BWT
20
40
STL. NeRV Chen et al. (202
28.92
31.10
34,96
29.35
28.87
31.41/
EWC Kirkpurrick et al. (2017)*
10.57
12.87
15.22
11.25
10.45
12.24 / -88.23
iCaRL. Rebuffi et al. (2017)"
15.22
18.55
17.59
13.47
15.89
16.54/ -8.48
ESMER Sarfraz
(2023))
13.54
15.46
16.78
12.48
13.22
14.78/ -15.95
WSN" 30.0 %
19.20
20.80
23.39
21.56
21.45
2,056/000
PFNR, 30.0 9. f-NeRV2
23.14
23.14
23.14 14
23. 14
23.08
24.22 /..
PFNR, c 30.0 &,, 3-NeVV
25.58
27.79
31.42
27.22
24.88
27.57 0.0
MTL (upper-bound) bound)
23.10
23.19
24.63
22.84
23.45
24.57
32.50
3045
7:2
3.54 9:70 19:2
23.94
1:35
7:33
8:75
5:37
5:70
10:3
1:31
7:03
3701.44
5.9
8:55
1:20
5.04
5:57
50.4
5.9
0.00
9:20
5.94
5:27
23.054
5.2
38
9.5
16:33
03
5.3
16:2
21
5:21
1235
2.7
19
442
9:90
1/0
443
6:15
19
11 10-4 /0112 w.2.--000000 mon so
1.17
11:1
20
1.67
2002 1as
7.4
5.65
1:24
2.03
162
9:15
20.2
2922
6.26
L154
24
515
23 5034
2022
6:37 475
515
31-2232220000 24660)
12.0
48
33 5034
24
24
23
22 04
246003
2655 06.100 32
55
535
55
535
(a) WSN, c 30.0%
(b) PFNR, c 30.0%. f-NeRV3
Figure 6: Transfer Matrixes of WSN v.s. PFNR on the UVG17 dataset measured by PSNR of source and target.
(b). Nothing special is observable from the upper triangular since they are not correlated, however, there might be some shared representations.
Ablation Study of FSO. We prepare several ablation studies to prove the effectiveness of FSO. First, we show the performances of only real part (ignore an imaginary part) in f-NeRV2/3 as shown in Table 11. The PSNR performances of only real part were lower than those of both real and imaginary parts in f-NeRV2/3. We infer that the imaginary part of the winning ticket improves the implicit neural representations. Second, we also investigate the effectiveness of only FSO without Conv. Layer in f-NeRV2/3, shown in Table 12. The PSNR performances were lower than FSO with Conv block. Therefore, the ensemble of FSO and Conv improves the implicit representations. Lastly, we investigate the effectiveness of sparse FSO in STL, as shown in Table 13. The sparse FSO boots the PSNR performances in STL. These ablation studies further strengthen the effectiveness of FSO for sequential neural implicit representations.
Table 11: PSNR results with Fourier Subnueral Operator (FSO) layer (f-NeRV*) (detailed in Table 7) on UYG8 Video Sessions with average PSNR and Backward Transfer (BWT). Note that w/o imag. ignores the imaginary part in J-NeV
Method
Video Sessions
Avg. BWT
PENR_ 50.0%, NoRR22
34,46
33.91
32.17
36.43
25.26
20.74
30.18
25.45
29.82/0.0
PENR 50.0 % NeRV2 w/o imag.
31,34
33.79
32.04
36.
25.11
20.59
30.17
25.27
29.71/0.0
PFNR 50.0 /-NoRV3
36.45
35.15
35.10
38.57
28.07
23.06
32.83
27.70
32.12/
PENR. c 50.0 % NeRkV3 wio Imag.
35.66
34.65
34,09
37.95
25.80
21.94
32.17
26.91
3..15/0.0
19

---

Published as a conference paper at ICLR 2024
Table 12: PSNR results with Fourier Subnueral Operator (FSO) layer (f-NeRV*) (detailed in Table 7) on UVG8 Video Sessions with average PSNR and Backward Transfer (BWT). Note that w/o com: ignores the conv. layer in f-NeRV*.
Video Sessions
Avg. PSNR/ BWT
PFNR, = 50.0 St. F-NeRV2
34,46
33,91
32.17
36.43
25.26
20.74
30.18
25.45
29.82/0.
PFNR, 50.0 2-eRV2 w/o
30.05
32.10
30.12
31.82
24.00
19.60
26.21
24.17
27.54/0.0
PFNR, %. 3-NeRV3
36.45
35.15
32.83
27.70
32.12/0.0
PFNR, = 50.0 %. 3-NeRV3 w/o com..
35.46
35.06
34.98
38.23
28.00
22.90 98
32.57
27.45
31.84/0.0
Table 13: PSNR results of STL with Fourier Subnueral Operator (FSO) layer (f-NeRV+) (detailed in Table 7) on UVG8 Video Sessions with average PSNR and Backward Transfer (BWT).
Method
Video Sessions
5
Avg. PSNR/
BWT
STL, NeRV Chen et al. (2021a))
39.66
36.28
38.14
42.03
36.58
29.22
37.27
31.45
36.33/
STL. NeRV. f-NeRV2
39.73
36.30
42.03
36.64
29.25
31.65
36.40/
STL,, NeRV, f-NeRV3
42.75
37.65
42.05
42.36
40.01
34.21
40.15
36.15
39.41/
Training time and Decoding FPS. We train and test two baselines (NeRV, ESMER) with f-NeRV2 using one GPU (TITAN V, 12G) with a single batch size to investigate the computational expenses and decoding FPS on the UVG8 dataset, as shown in Table 14. In STL, NeRV with f-NeRV2 costs more computational times than NeRV. In VCL, memory auffer-based methods, i.e., ESMER, cost more training time since they replay the memory buffer in sequential training. On the other hand, architecture-baseee methods, i.e., PFNR, provide parameter-efficient, faster, forget-free solutions in training while cost computation expenses in the decoding process. Considering these limitations and advantages, we would find a more parameter-efficient FSO algorithm future work.
Table 14: Training time and Decoding FPS with Fourier Subnueral Operator (FSO) layer (f-NeRV2) (detailed
in Table 7) on UVG8 Video Sessions
Method
Training time [hours]
Decoding FPS
STL, NeRV Chen et. (2021a)"
13.10
56.38
STL, NeRV. /-NeRV2
25.66
31.72
ESMER Sarfraz et al. (2023)"
101.71
56.93
PFNR, f-NeRV2
31.55
Video Generation, We prepared some results of video generation as shown in Figure 7. At the human recognition level, approximately 30 PSNR provides a little bit of blurred generated images as shown in WSN's PSNR((9) (29.26); if the PSNR score is greater than 30, we hardly distinguish the quality of sequential neural implicit representations (city session). Thus, our objective to maintain the sequential neural implicit representation of the 30 PSNR score. We have achieved this target PSNR score on the UVG17 dataset. We also show the quantized and encoded PFNR's video generation results Figure 8. We demonstrate that compressed sparse solution in FP8 (PENR with 30.0%, f-NeRV2) generates video representations sequentially without a significant quality drop. The results of the FP4 showed that the compressed model can not create image pixels in detail.
A.6 LAYER-WISE REPRESENTATIONS
To investigate the property of FSO, we prepare the layer-wise representations, as shown in Figure 9. The representations of PFNR (f-NeRV3) focus on textures rather than objects at the NeRV3 layer in the video session of bosphorus. In the video session of bee, PFNR (f-NeRV3) also tends to capture local textures broadly at the NeRV3 layer. This behavior of f-NeRV3 leads to better generalization at final prediction than others, such as WSN and PFNR (f-NeRV2). Moreover, we conducted an ablation study to inspect the best sparsity of f-NeRV3, as shown in Figure 10. Specifically, the performances of PFNR, c=50.0% depend on the sparsity of f-NeRV3, We observed that f-NeRV3 with c=50.0% was the best sparsity.
A.7 CURRENT LIMITATIONS AND FUTURE WORK
Since the parameters of FSO depend on the input/output feature map size, in this task, the deeper the FSO layer, the larger the parameters increase. Nevertheless, we found the most parameter-efficient
20

---

Published as a conference paper at ICLR 2024
1=0
t=l
t=2
WSN (34.37, PSNR)
PFNR, f- f-NeRV2 (35.84, PSNR)
PFNR, f-NeRV3 (39.24, PSNR)
WSN (29.26, PSNR)
PFNR, f-NeRV2 (31.24, PSNR)
PFNR, f-NeRV3 (34.05, PSNR)
Figure 7: PFNR's Video Generation (from t=0 to t=2) with c 30.0% on the UVG17 dataset.
FSO structure through layer-wise inspection and layer sparsity to describe the best neural implicit representations. future work, we will design more parameter-efficien FSO layer in continual tasks such as neural implicit representation and task/class incremental learnings.
A.8 BROADER IMPACTS
As the most popular media format nowadays, videos are generally viewed as frames of sequences. Unlike that, proposed PFNR is novel way to represent sequential videos function of video session and time, parameterized by the neural network firstly in Fourier space, which is more efficient and might be used in many video-related tasks, such sequential video compression, sequential
21

---

Published as a conference paper at ICLR 2024
2.city
3.beauty
7.bee
PENR (35.84, PSNR),
(32.97, PSNR),
(36.01, PSNR) FP32,
PENR (35.84, PSNR),
(32.97, PSNR),
(36.01, PSNR) in FP16
PENR (35.81, PSNR).
(32.95, PSNR),
(35.82, PSNR) in FP8.
PFNR (27.43, PSNR).
(24.65, PSNR),
(16.24, PSNR) in FP4.
PFNR's Qunatizationed and Compresssed Video Generation (t=0) with 30.0%, f- NeRV2 on the UVG17 dataset. Note that PFNR's prediction in FP32, 16, 8, and 4.
Figure 8:
Table 15: PSNR results with Fourier Subnueral Operator (FSO) layer (f (f-NeRV=) (detailed in Table 7) on UVG17 Video Sessions with average PSNR, Backward Transfer (BWT) PSNR, Model Capacity (CAP). Note denotes our reproduced results.
Metiel
Vidoo Sonkess
3
13
11
13
14
12
15
Anp PANR
CAP
IWC Kilipuraken elll (2011y"
11.15
12:31
11.0
15:55
13
6.34
10.31
0.55
10.33
1890
36.36
11.38-36
10310/1
ICRL Rabutt puiity"
35.25
22.39
22.74
2284
16.55
27.43
2410
21.40
19.43
15.00
26.46
21477
100.00 S
BSMER of E20Oy"
30.7
25.35
22.39
2133
23.76
13.54
28.25
15:23
16.1
21.78
13.35
1523
18.31
19.33
24.99
2011
22.42
20355-153 15.23
10..0%
WSM 10.0%
21.88
31:11
31.23
31,8
25.66
22.37
S1A2
22.04
21.35
51.71
12.53
30.00
200
23.44
22.35
20.30
25.35
2595 (LD
55.25
31.00
32.38
32.27
24.13
WENT 20ars
31.54
31.74
29.56
21.83
33.35
19:22
21.30
25.36
26.77
97.23
WSS 10
34.76
31.35
30.27
21.00
31.79
32.37
17.63
22.34
20,99
25.08
(L.
9901
25.34
FThaL 33.-5 1-MAR82
7584
13.07
35.17
34.53
35.52
25.40
(1.
102.00
rThaL 91.%%, 1-068812
31.13
31.21
35.50
34.30
24.75
24.73
35.65
25.15
25.18
165.33
FENAL N1.%% 1-0681122
15.30
32.09
12.15
29.67
23.15
34.00
24.12
23.47
33.34
112.33 9
XTL (ppeeede-
$2.99
34.55
31.45
3408
10.00
34.39
57.15
2383
23.90
54.99
30.71
22.37
55.71
25.00
23.89
36.00
33.16
25:427-
101.0%
video denoising, complex sequential physical modeling (Li et al., 2020a;b; Kovachki et al., 2021; Tran al., 2021), and on. Hopefully, this save bandwidth and fasten media streaming, enriching entertainment potential. Unfortunately, like many advances in deep learning for videos, this approach could be used for various purposes beyond our control.
A.9 ACKNOWLEDGEMENT
This work was partly supported by Institute for Information communications Technology Planning Evaluation (ITTP) grant funded by the Korea MOV)) (No. 2021-0-01381, Development
22

---

Published as a conference paper at ICLR 2024
Table 16: MS-SSIM results with Fourier Subnueral Operator (FSO) layer (f-NeRV*) (detailed in Table on UVG17 Video Sessions with average MS-SSIM and Backward Transfer (BWT) of MS SSSM. Note that denotes our reproduced results.
Video
STL, NeVV" Clan al (20014)
0.99
0.99
0.35
0..8
0.98
0.96
0.99
0.98
0.97
0.55
0.96
0.96
0.95
0.95
0.96
0.99
0..9
0.97
EWC Käkpunick etai (2017)
0.25
0.24
0.44
0.24
0.40
0.29
0.15
0.17
0.26
0.26
0.17
0.34
0.04
0.30
0.33
0.31
0.01
0.301 -0.5
Reoum (2017)
Sariaa (2023)"
0.51
0.62/-0377
WSN < 100
0.9
0.35
1192
0.87
0.55
0.74
0.09
0.1
0.57
0.72
0.85
0.80
0.74
0.09
1192
0.82
c 30.0 %
0.95
0.97
0.29
0.93
0.88
0.77
0.97
0.77
0.33
0.91
0.60
0.74
0.36
0.51
0.76
0.72
0.03
0.84 00.
W
WaK".cm 70,0
0.90
PINK, c S.
0.9
1.50
1194
01.00
0.81
0.59
0.86
0.79
0.55
0.69
0.79
0.91
0.87
0.54
11.90
0.88 1001
PFNR,c=2000 9./-eRV2
0.97
0.95
0.92
0.05
0.92
0.84
0.98
0.58
0.82
0.93
0.73
0:82
0.92
0.89
0.94
0.57
0.97
0.90 00.
0.95
10.
P /NNdVV
0.77
PINK, c S.
0.5
1.99
0.92
11.66
01.4
0.86
0.59
0.94
0.82
0.55
0.76
0.83
0.93
0.9
11.00
11.96
0.91/10 70.0
PENR,c=J000 /-NeV33
0.95
0.99
0.93
0.97
0.96
0.91
0.99
0.96
0.87
0.94
0.84
0:37
0.94
0.94
0.90
0.94
0.98
0.94
0.94/00
P 70.0% /-NNVa
MTL tuppes bound)
037
0,50
0.94
0.91
0.82
0.99
0.92
0.80
0.92
0.75
0.41
0.98
030
0.39
11.7
0.907
NeRV3
NeRV4
NeRV5
NeRV6
Head
WSN, 50.00%
PFNR, 50000% N
PFNR, -550.0% /-NeV3
Figure 9: PFNR's Representations of NeRV Blocks with c = 50.0% on the UVG17 dataset.
of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and partly supported by the National Research Foundation Korea (NRF) grant funded by the Korea (ST)) (No. 2022R A2C2012706)
23

---

Published as a conference paper at ICLR 2024
NeRV3
NeRV4
NeRVs
NeRV6
Head
PFNR, 5000%, sparsity of 3-.5.
PFNR, 55000% sparsity of 3-25RV3-2.5%.
PFNR, 50000% sparsity of /-50RV3=5.0%
PFNR, 55.0%, sparsity of 1=1.0%
PFNR, c5000% sparsity of f-NcRV3=25500%
PFNR, 55000% sparsity of 3-5eRV3=5000%
Figure 10: Various sparsity of f-NeRV3 ranging from 0.05 % (top row) to 50.0 % (bottom row) the UVG17 dataset.
A.10 PUBLIC CODES OF OUR PFNR
We provide the core parts of PFNR to understand better. Please refer to the attached file. We will provide all training and inference codes soon.
24

---

Published as a conference paper at ICLR 2024
Table 17: PENR (f-NeRV2) of Quantization Compression the UVG17 Video Sessions with average PSNR/MS- and Backward Transfer (BTW). Note that is the bit length used to represent parameter
value.
Mechod
Video Sondons
Arg. */
10
1I
12
13
14
15
16
17
10.0%
PSNNR
biled
17.73
24.61
24.79/-0.00
29.37
26.58
Ni-16
28.31
33.57
31.92
33.57
29.98
23.99
34.39
24.80
23.94
35.08
19.70
22.03
20.56
26.57
24.79
24.10
31.35
25.10/ 0.00
Na
28.31
13.57
M1.2
31.67
79
$4.39
24.30
21.94
15.00
70
22.03
28.56
26.57
24. 79
34. 10
51.35
28, 18.10/ 0.00
SSIM
biws
0.92
0.97
0.90
0.94
0.90
0:41
0.98
0.86
I.79
0.69
0.79
0.91
0.57
0.32
0.84
0.56
0.88/ 0.00
No-16
0.92
0197
0.90
0.9
11:00
0:31
0.58
0.56
11779
11933
0.69
0.79
0.91
0157
11:22
0.84
0.56
(1857 0.01
0.66
30.
PSIR
hikmd
24.87
24.19/-2.15
31.72
27.37
25.52
Nim16
$2.01
35.64
$2.07
$5.17
31.34
34.83
16.01
25.85
24.83
35.76
20.90
22.75
30:40
27.37
25.52
25.
$2.70
20.16/ 0.00
Nz=52
$2,01
35.84
32.97
35.17
31.24
24.82
36.01
25.85
24.83
35.76
20.30
22.79
30,40
27.37
25.52
25 40
32,70
20:367 0.00
M5-55IM
Nack
0.97
0.92
0.95
0:84
09%
0.88
0.75
0.82
0.92
0.88
0.84
0.87
0.97
(1907 0.00
hG-16
0.99
0.95
0.92
0.95
0.92
0:84
0:98
0.58
1.52
0.93
0.73
0.82
0.92
0.59
0.54
0.57
0.97
0.007 0.00
0.99
0334
SQLDRS
PSNRR
20.92/-1.01
24.78
29.76
27.04
25.17
Ki-16
34.49
37.13
33.21
35.50
30.57
24.72
34.36
24.79
24.73
35.65
20.33
22.65
20.76
27.05
25.18
25.15
$2.39
29.29/ 0.00
$4.49
37.13
53.21
35.50
30.62
34.77
14.36
24.79
24.75
35.63
51
22.65
20.76
27.05
25.18
25. 13
32.99
29:39
0.00
SSIM
News
0.98
01.99
0.92
0.95
0.92
0:33
0.5%
0.56
0.72
0.82
0.91
0.58
0.34
0.57
0.97
(189/ 0.00
No=10
0.98
1199
0.92
0.95
019
0333
0.5%
0.56
1181
1193
0.73
0.82
0.91
0158
1154
0.87
0.97
(1897 0.001
70.1
PSIR
Nikwd
7.05
22.55
22.04
19.157-3448
21.29
21.86
25.56
24.13
23.47
30.30
Niml6
02
36.50
32,00
$2.15
25.67
23.85
30.63
23.85
23.18
34.00
19.08
27.87
25.86
24,12
23.47
50.34
27.79 0.00
Nz=32
$6.02
36.50
32,09
32.15
28.67
23.35
30.63
22.86
23.18
34.50
19.00
21.30
27.87
25.86
24.12
23.47
30.34
27.79 0.00
SSIM
0.98
0.91
0.93
0.80
0.78
0.88
0.77
0.89
0.8
0.90
0.83
0.95
(185-) 0.00
No-16
0.59
0.95
0.91
0.90
0.37
0.75
096
0.77
1177
0.93
0.66
0.77
0.59
0.5
0.50
0.83
0.95
0.867 0.00
0.5M
0.55
0.00
25


Reviews:
[{"review_id": "LidXaaBhGG", "reviewer": "Reviewer_inAP", "summary": "This article addresses the existing problem in Neural Implicit Representation (NIR) that these models specialize in learning only one mapping between target data and fail to generalize when implemented over more data, thus limiting scalability. The authors propose a modification on top of the NeRV pipeline [Chen et al, 2021a]. In order to capture information while doing a fine discretization of model parameters, the authors transform the weights to the Fourier space, on the lines of the work done by [Li et al., 2020a]. Experimentation has been done on Video Task-incremental Learning and comparisons have been made against relevant baselines. The experiments closely follow NeRV [Chen et al, 2021a] and HNeRV [Chen et al., 2023] to make fair comparisons. Results are satisfactory and visualizations look good.\n\n\nHao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. Advances in Neural Information Processing Systems, 34:21557–21568, 2021a.\n\nZongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar. Fourier neural operator for parametric partial differential equations. arXiv preprint arXiv:2010.08895, 2020a\n\nHao Chen, Matt Gwilliam, Ser-Nam Lim, and Abhinav Shrivastava. Hnerv: A hybrid neural representation for videos. arXiv preprint arXiv:2304.02633, 2023.", "questions": "How does the proposed architecture compare with existing baselines in terms of computational requirement? Does the FSO module increase the computational requirements significantly from the baseline NeRV?\n\nHow many frames for each video was considered during training? Can the authors elaborate the video frame resolution vs frame rate trade-off i.e. what’s the maximum number of frames that can be considered at a time at a certain resolution or what’s the maximum resolution that can be processed at a certain number of frames?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The article benefits from generally good writing, appropriate use of mathematical language. \n\nBackground literature survey is highly appropriate and extensive.\n\nThe article builds on top of a NeRV pipeline, addressing relevant and interesting shortcomings. The rationale behind the use of Fourier Subneural Operators has been developed clearly.\n\nExtensive experimentation and provision of visual results in the paper and the supplementary file also adds to the strengths of the paper. Results are satisfactory.\n\nThe ability of the model to generate videos has also been shown in the supplementary document which only adds to the broader impacts of the research done by the authors.", "weaknesses": "The methodology section may appear somewhat difficult to some readers without the necessary background in the domain. This however is only a minor weakness that does not affect the final rating.\n\nIt is not very clear if the authors used the exact frame sizes of the videos as available, or if the authors did any spatial or temporal downsampling on the videos to fit the model training pipeline, as is usual with most papers pertaining to video data.\n\nIt appears that the model may be limited by computational expenses as the datasets used are that of short video clips.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o7O6BVPsaY", "reviewer": "Reviewer_uKbd", "summary": "This paper investigates the neural implicit representation in continual learning settings and proposes a novel method to encode the learned knowledge compactly. With this design, the model becomes able to accumulate neural representations for multiple videos with few decoding losses for previous videos. Experiments on UVG8/17 verifies its effectiveness.", "questions": "In Algorithm 1, the operation at line 7 is not differentiable, so how to calculate $\\partial L / \\partial \\rho$?\nI'm willing to change my score if all my issues can be solved", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed learning scenario is very practical in real applications, that is, encoding multiple videos continually via a neural implicit representation. With this setting, the generalization ability of NIR can be evaluated.\n\n2. I like the idea that incorporates the Lottery Ticket Hypothesis to find subnetworks for each video session. Frozing these subnetworks proves to be an effective way to encode existing knowledge about previous videos.\n\n3. The designed experiments verify its ability to adapt to new training sessions. With the proposed methods, the performance achieved significant improvements.", "weaknesses": "1. The writing is not so clear. It's hard to learn the connection of Sec 3.1 with other sections.\n\n2. The experiments miss one important point. All tables show the results of the newly added sessions. However, to verify the \"lossless decoding\" stated in the abstract, previous videos need also to be evaluated. Otherwise, the reported metrics only verify the quick adaptation ability of the proposed method.\n\n3. There are some neglected details.\n- At the 2nd line of Sec 3, the cited paper is E-NeRV instead of NeRV.\n- In the caption of Fig 1, the symbol $H_N$ is used without definition and no further usage.\n- At the line before Eq 3, should it be \"session s\" instead of \"session t\"?\n- I suggest moving Algorithm 1 to the same page as Sec 3.2, where it is referred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5WwUR57Dx", "reviewer": "Reviewer_MyBt", "summary": "The response provides a comprehensive analysis of a work focused on Neural Implicit Representation (NIR) for video data encoding. The work introduces a novel method, Progressive Fourier Neural Representation (PFNR), to improve the accumulation and transfer of neural implicit representations for complex video data across sequential encoding sessions. PFNR leverages a sparsified neural encoding in Fourier space, enabling better adaptation for future videos and lossless decoding for previous representations. The method shows impressive performance gains over continual learning baselines on UVG8/17 video sequence benchmarks.\n\nStrengths of the work include the novel and straightforward concept of combining Fourier representation with sparsification and the method's superior performance over baselines with the same capacity. However, weaknesses are noted in the manuscript’s clarity, particularly in interpreting tables and comparing performance to baselines, as well as in the diverse configuration for Fourier transform.\n\nSeveral questions are raised, seeking clarification on the commonality of NIR usage, details on Fourier transform configurations, baseline performances in Single Task Learning, comparisons to WSN, the meaning of outperforming an upper bound, and a typo in Table 3. These questions aim to probe deeper into the work’s methodology, performance, and presentation for a clearer understanding and evaluation.", "questions": "1. Is it common to use neural implicit representation (NIR)? As I know, many works adopt the term implicit neural representation (INR) rather than NIR.\n2. The configuration for Fourier transform?\n    - What is the temporal length of frames ($d_\\nu$ in the paper right?) Does the $d_\\nu$ affect the final performance?\n    - Some sparsification protocols for Fourier transform ignore an imaginary part in both Fourier and inverse Fourier transform. \n3. The performances of baselines in Single Task Learning (STL).\n    - What is the performance of the proposed method with Single Task Learning (STL)? Does the proposed method impact negatively due to the proposed components for continual learning?\n    - How about the performance only FSO without Conv block in NeRV block?\n4. Comparison to WSN\n    - What is the averaged PSNR and MS-SSIM performance of the proposed method and WSN on UVG8/17 with varying capacity? It is better to visualize in plot of the performance and capacity rather than table because the capacity of the model seems to be important for this setting. I conjecture that table 8 is the good starting point.\n5. Meaning of upper bound \n    - The proposed method, PFNR, outperforms the upper bound, MLT. What is the reason behind?\n6. Typo\n    - Table 3 shows the result of MS-SSIM while the caption has PSNR.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed idea, combining Fourier representation and its sparsification is a straightforward and novel concept.\n2. With the same capacity, the proposed method provides better performance over baselines.", "weaknesses": "1. The target task seems to be small compared to the generality of the method.\n2. The tables are a little hard to interpret.\n3. The current version of the manuscript is missing comparable performance to its baselines.\n4. The diverse configuration for Fourier transform.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nPmum0tBx5", "reviewer": "Reviewer_H5Ez", "summary": "This paper addresses a novel practical task scenario where a pretrained video neural implicit representation needs to continually learn new data while keeping the learned information. It proposes a novel Progressive Fourier Neural Representation method to tackle this, which continuously learns a compact subnetwork for each video in Fourier space. The proposed method is tested on several datasets with multiple metrics including PSNR and SSIM, and proves to outperform the baseline and other competiters.", "questions": "- In the abstract and introduction, it is illustrated that since the INRs learn to memorize videos \"regardless of data relevancy or similarity\", it is hard for them to be generalized to multiple complex data and thus continual learning is important for the models to learn new videos without forgetting previously learned videos. I agree that memorizing videos regardless of data relevancy or similarity limits INRs' efficiency and scalability, but shouldn't this characteristic help an INR to learn multiple unrelated videos compared to those that learn with data relevancy or similarity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem of encoding new videos into existing video INRs is important in practice, and the proposed method addresses it well according to the experiment results with higher PSNR and SSIM compared to other competitiers.\n\n- Besides the detailedly listed final metric results, it also shows abundant ablations on several hyperparameters and such as which layer to choose for the proposed model to learn etc., in both the main paper and the supplementary.\n\n- Many baselines are discussed in the related work section as well as compared technically in the experiments.", "weaknesses": "- Although the quantitative results are listed in details for every video in two settings, overall the proposed method is only tested on two datasets, and both UVG series. More diverse choices would make the results more solid, such as the DAVIS dataset series etc.\n\n- The compression performance (model size) is not well tested and displayed, especially in the tables. Figure 2 and its paragraph discussed some but is still not clear to link with the values in other tables.\n\n- The illustration of the proposed method is relatively limited, e.g. about the details in Fourier space, which might hide the novelty and complexity of the proposed method.\n\n- Not many visual comparisons are provided especially in the main paper, and Figure 7 is not specifically explained on their differences and advantages etc.\n\n- [Minor] Section 3.1 said that Figure 1 is one \"possible\" structure, while there isn't any other design mentioned in this paper. Maybe there can exist more variants but if not mentioned then Figure 1 is just exactly \"our proposed structure\" to be clearer.\n\n- [Minor] Figure 1 is overall good but the font and diagram size is a bit small compared to frame images.\n\n- [Minor] Sometimes the HNerv paper is noted as Nerv (but with the correct 2023 reference).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)