INR-V: A Continuous Representation Space for Video-based Generative Tasks
Authors: Bipasha Sen
Aditya Agarwal
Vinay P Namboodiri
C.V. Jawahar
Venue: Accepted by TMLR ()

Abstract: Generating videos is a complex task that is accomplished by generating a set of temporally coherent images frame-by-frame. This limits the expressivity of videos to only image-based operations on the individual video frames needing network designs to obtain temporally coherent trajectories in the underlying image space. We propose INR-V, a video representation network that learns a continuous space for video-based generative tasks. INR-V parameterizes videos using implicit neural representations (INRs), a multi-layered perceptron that predicts an RGB value for each input pixel location of the video. The INR is predicted using a meta-network which is a hypernetwork trained on neural representations of multiple video instances. Later, the meta-network can be sampled to generate diverse novel videos enabling many downstream video-based generative tasks. Interestingly, we find that conditional regularization and progressive weight initialization play a crucial role in obtaining INR-V. The representation space learned by INR-V is more expressive than an image space showcasing many interesting properties not possible with the existing works. For instance, INR-V can smoothly interpolate intermediate videos between known video instances (such as intermediate identities, expressions, and poses in face videos). It can also in-paint missing portions in videos to recover temporally coherent full videos. In this work, we evaluate the space learned by INR-V on diverse generative tasks such as video interpolation, novel video generation, video inversion, and video inpainting against the existing baselines. INR-V significantly outperforms the baselines on several of these demonstrated tasks, clearly showing the potential of the proposed representation space.

Published in Transactions on Machine Learning Research (10/2022)
INR-V: A Continuous Representation Space for Video-based Generative Tasks
Bipasha Sen' IIT Hyderabad
bigusha..en@hrrsearrhhii......
Aditya Agarwal' IIT Hyderabad
adikyp..gg@rresearchhiii..
Vinay P Namboodiri University of Bath
vpn22@both.com uk
C. V. Jawahar IIT Hyderabad
jawahar@siit. oc oc.in in
Reviewed on OpenReview: https: // openreview. net/ forum? id- al B1Ecc228
//
Abstract
Generating videos is a complex task that is accomplished by generating a set of temporally coherent images frame-by-frame. This limits the expressivity of videos to only image-based operations on the individual video frames needing network designs to obtain temporally coherent trajectories in the underlying image space. We propose INR-V, a video repre- sentation network that learns a continuous space for video-based generative tasks. INR-V parameterizes videos using implicit neural representations (INRs), multi-layered percep- tron that predicts an RGB value for each input pixel location of the video. The INR predicted using a meta-network which hypernctwork trained on neural representations of multiple video instances. Later, the mela-network can be sampled to generate diverse novel videos enabling many downstream video- based generative tasks. Interestingly, find that conditional regularization and progressive weight initialization play crucial role in obtaining INR-V. The representation space learned by INR-V is more expressive than an image space showcasing many interesting properties not possible with the existing works. For instance, INR-V can smoothly interpolate intermediate videos between known video instances (such as intermediate identities, expressions, and poses in face videos). It can also in-paint missing portions in videos recover temporally coherent full videos. In this work, evaluate the space learned by INR-V diverse generative tasks such video inter- polation, novel video generation, video inversion, and video inpainting against the existing baselines. INR-V significantly outperforms the baselines on several of these demonstrated tasks, clearly showcasing the potential of the proposed representation space.
1 Introduction
Learning to generate complex spatio-temporal vidcos from simple distributions is a challenging problem computer vision that has been recently addressed in various ways Tian al. (2021): Tulyakov al. (2017)
Clark et al. (2019); Skorokhodov et al. (2021) Ding et al. (2019) Yu et al. 2022 Yan et al. [2021) State-of- the-art (SOTA) works Skorokhodov et al. 2021 Tian et al. (2021): Yu et al. (2022) treat video generation as task of generating a sequence of temporally coherent frames. Although such networks have advanced
the SOTA to generate high-quality frames (such as carefully crafted eyes, nose, and mouth for talking-head
Equual contribution.

---

Published in Transactions on Machine Learning Research (10/2022)
Single Video Unrolled
Latent Interpolation in continuous video space
Real Video V,
Real Video 1/2
Figure 1: Demonstrating the continuity of the video space learned by INR-V by interpolating novel videos between two real videos V1 and V2. Note that content (identity, hair) and motion (pose, expressions) gradually transition as we traverse the trajectory in the latent space between V1 and Vy's latents.
videos), they come with a major limitation: They rely on an image space. This limits the application of the learned space to image-based operations such as animating images and editing on frames. Direct operations on videos, such as interpolating intermediate videos between two videos and generating future segment of a video, become difficult. This because such operations require learning the set of frame and motion constraints and ensuring that they are coherently learned.
We propose that videos can be represented a single unit in- stead of being broken into sequence of images. One can learn latent space where each latent point represents a complete video. However, with existing video generator architectures, such representations are difficult, Firstly, such a video gen- crator would be made of several 3D convolution operations. As the dimension and length of the video increase, such an architecture would become drastically computationally expen- sive (a GPU with limited memory can only fit video of lim- ited dimension). Secondly, videos are high-dimensional signals spanning both spatial and temporal directions. Representing such a highly expressive signal by a single latent point would require complicated generator architectures and a very high- dimensional latent space. Instead, videos can be paraneterized function of space and time using implicit neural represen- tations (INRs). Any point in video Vrws can be represented by function fo(h, w, f) > RGB where t denotes the 1th frame in the video and h, w denote the spatial location in the frame and RGB denotes the color at the pixel position (h, w, t}. Here, the dynamic dimension of videos (a few million pixels) is reduced to a constant number of weights 0 (a few thousand) required for the parameterization. A network can then be used to learn a prior over videos in this parameterized space. This can be obtained through a meta-network that learns function to map from latent space to a reduced parameter space that maps to a video. A complete video is thus represented as a single latent point.
6
fzz
Random Barnliig
Continuous
Video Space -.
Novel Video Instances
Figure 2: Overview of INR-V: INR-V learns continuous video space by first pa- rameterizing videos implicit neural repre- sentations denoted by Se,, where denotes a unique video instance V,, Next, a meta- network based on hypernetworks denoted by dg is used to learn a continuous represen- tation over the neural representations. da is conditioned by an underlying continuous video space where cach point denotes the condition for complete video.
We propose INR-V, V, a video generator network with a continuous video representation space based on learning an implicit neural representation for videos. It illustrated in Fig. [ INR-V is made of key clements that, when combined, makes it ideal for video representation: (1) Its INR is free of expensive convolutional layers (millions of parameters) such in the existing architectures Tian et al. (021) Skorokhodov et al. (0011)
2

---

Published in Transactions on Machine Learning Research (10/2022)
and relies on few layers of traditional mullti-layered perceptrons (MLPs), leading to very few parameters (a few thousand). (2) Having very few parameters, INR's weights can be populated using secondary meta- network called hypernetwork Ha et al. (2016) that learns a continuous function over the INRs by getting trained on multiple vidco instances. (3) is trained on a deterministic distance loss, such as Euclidean or Manhattan distance. This allows INR-V V to learn the exact requirements of a coherent video directly from the ground truth video instances.
Hypernctworks have seen wide applications in graphics Sitzmann et al 2020bb 2021 Chiang et al. (2021)
Sitzmann et al. (211); however, they have been seldom used for videos. Hypernetworks are notoriously unstable to train, especially on the parameterizations of highly expressive signals like vidcos. Thus, we propose a key prior regularization and a progressive weight initialization scheme to stabilize the hypernetwork training allowing it to scale quickly to more than 30,000 videos. As we show in the experimental section, INR-V demonstrates an expressive and continuous video space by getting trained on these datapoints. The Icarned prior enables several downstream tasks such as novel vidco generation, video inversion, future segment prediction, and video inpainting directly at the video level. As shown in Fig. T INR-V also showcases smooth interpolation of novel videos between two videos by traversing the path between their latent points, Interpolation morphs different identitics and motions and generates coherent videos. Interestingly, the properties demonstrated in this work are not enforced at training but are natural outcomes of the continuous video space. To summarize, our contributions in this work are as follows:
1. We propose considering videos as single unit and learning continuous latent space for videos where each latent point represents complete video.
2. We propose INR-V, a vidco representation technique that parameterizes vidcos using INRs, bringing down the dimension of a video from a dynamic few million to a constant few thousand. INR-V V uses a hypernetwork as a meta-network to learn a continuous space over these parameterizations.
3. We demonstrate the benefit of key regularization and progressive weight initialization scheme to stabilize the hypernetwork training. We scale the hypernetworks to more than 30,000 video points enabling it to learn a continuous meaningful latent space over the INRs.
Lastly, we retenstrate key properties of the learned video space, such as video interpolation, video inversion, and so on, by conducting several experiments and evaluations.
2 Related Work
Vidco Generation. Vidco generation aims to produce novel vidcos from scratch. It falls under the paradigm
of 'video synthesis' that encompasses several categories, including (1) Video prediction Luc el al. (2020):
Moing et al. (202T): ; Walker et al. (2021): that predicts the next set of frames given the current frames, (2)
Frame interpolation Park al. (0211); Niklaus & Liu (2020); Niklaus et al. (2017); Zhang et al. (2021): ): that interpolates frames between given frames of video. These tasks generate the unseen portion of the video based on the context of the seen portion. On the other hand, video generation produces videos without any expressive prior context, making the task more challenging. The complexity of the problem has led to a plethora of works in this area Tian et al. (2021); Tulyakov et al. (0117) Skorokhodov et al. (2021) Clark
et al. (2199) Ding et al. (2019) Yu et al. (0022) VideoGPT Yan et al. (2021) tackled this challenge by first reducing the raw videos of up to 128 x 128 dimension to quantized space. It then trained a transformer
architecture to model a prior over the quantized space. Our architecture is conceptually similar to VideoGPT, which used likelihood-basee generative model to learn video prior. However, VideoGPT operates quantized space that is discontinuous, making the prior less expressive. INR-V, the other hand, models a continuous vidco space. VidcoGPT also consists of 3D convolution layers making the model computationally expensive for larger videos. INR-V is a simple MLP, based on a continuous parameteriiation scheme of INRs, making it agnostic to the video dimension. This allows scaling to multiple resolutions (64 x 64 or 256 x 256) at inference without any architectural changes or finctuning. More recent works StyleGAN-V [Skorkkhodov et al. (2021), DIGAN Yu al. (2022), and Mo Tian et al. (2021) are a GAN-based setup that model videos as temporally coherent trajectory over image space. Use of continuous representation
3

---

Published in Transactions on Machine Learning Research (10/2022)
space for videos has been considered before in Fernando et al. (2015): Bilen et al. (0116) for the task of action classification. However, in this work, we focus on learning representation space for generative tasks.
Hypernetworks. Hypernetworks Ha et al. (0016) were introduced as a metafunction that initializes the weights for a different network called the primary network. Hypernetworks have been widely used for several purposes, starting from representation learning for continuous signals Park et al. (2019); [Sizzmann et al.
(2021
2020a
by: Mescheder et al. (2018); [Sizzmann et al. (2019), compression Nguyen al. (2022); Gao few-shot learning (Snddrra al. (2022); Lamb et al. (2021). continual learning von Oswald
et al.
20210,
et al.
2019 language modeling (Suarez (2017). We use hypernetworks to populate our primary video generation network, an MLP parameterizing different video instances.
Implicit Neural Representations. In this paradigm, a continuous signal is represented by a neural network. LNRs have had wide adaptations in 3D Computer Vision Park et al. (2019 : Genova et al. (2019)
Sitzmann et al. (2018): Mcscheder ct (2018); Sitzmann ct al. (2021):
): Mildenhall et al. (2020) and Computer
Graphics Guo et al.
[2021]; Yao al. (0022). Recently, y, LNR was adopted for images Skorokhodov el al.)
(2020) and videos Chen et al. (2021); Sitzmann et al. (2020b); Yu et al. (2022): Chen et al. (2022). INR-
GAN Skorokhodov et al. (2020) first showed the application of LNRs in generating high-quality images by
replacing the generator component of StyleGAN2 Karras et al. (019)) with an MLP-based INR. It then used a hypernetwork to populate the INR. Unlike INR-GAN, which is trained using a stochastic discriminator;
INR-V relies on a deterministic distance based loss to train the hypernetwork. SIREN Sitzmann et al. 2020b)
proposed periodic activation functions for INRs as a replacement for ReLU activation to parameterize many different data types like images, videos, sounds, and 3D shapes, with fine details. NeRV Chen et al. (2021)
designed implicit function as a continuous function of time and used convolution blocks each time step to parameterize discrete frames showcasing an improved frame quality over SIREN. Recently VideoINR Chen
et al. 2022) was proposed that used INRs for video superresolution. DIGAN Yu et al. (2022) incorporated INRs made of MLP layers for video generation. It consisted of two separate networks that generated spatial and temporal codes for generating videos in a frame-wise fashion. StyleGAN-V Skorokhodov et al. (2021) also incorporated INRs and relied on continuous non-periodic positional encodings for each timestep of a video. Like NeRV, StyleGAN-V used traditional convolution operations for frane-by-frand brrammme video generation. Both DIGAN and StyleGAN-V used GAN sctup to train the video generators. INR-V is based on MLPs with ReLU activation trained in fashion similar to Light Field Networks (LFNs) Sitzmann et al. 20221 LFNs proposed a novel neural scene representation for novel view synthesis and trained hypernetwork over multiple object instances using distance- based losses like Euclidean or Manhattan distance. Like LFNs and INR-GAN, INR-V parameterizes the entire signal (a video) using INRs and relics on a single hypernetwork to generate the INRs. However, unlike LFNs and INR-GAN, INR-V encodes a denser representation of a volumetric 3D signal e R&3 data making hypernetwork training more challenging.
3 INR-V: Implicit Neural Representation for Video Synthesis
Each video instance V, consists of pixels at locations (h, w) at fth frame. We have a particular parameter vector On that is used by network f to generate the value of the color RG for that pixel location (h, w, l). We need to learn a network d with parameters Ω that predicts the parameters O,, for particular video Vn. Here, d is hyper-network. The overall approach to train the network is illustrated in Fig. 3
3.1 Hypernetwork for Modeling Multiple Video Instances
As fe implicitly stores a single video signal, any new video would need its own implicit function. Let fe. denote the implicit function for given video (Va J.V where N is the total number of available videos in the training dataset D. Each of these implicit functions, fo, can be modeled using neural network trained each pixel value of the video Vn. Thus, implicit functions minimize the following objective:
1 T W H
I(O.) = T W H ΣΣΣ
(fo (h, w. BG
(1)
4

---

Published in Transactions on Machine Learning Research (10/2022)
Encoding
Video-CLIP Embeddings
9n
Real
Mn
Codebook C
Auto Decoding
FC
FC
FC
Code Instance
Grid
e
Distance Loss
h
FC
FC
fen
W
Hx W
Prediction
Hx WXXWTT T
Figure Architecture INR-V: Any video instance Vn is represented by ils correspouding implicit neural representation, an MLP, fo. fo., takes grid as input denoting the pixel positions of the video encoded using periodic positional encodings. It then generates the pixel values for all the positions. fe. is initialized by meta-network called hypernetworks denoted da composed of set of MLPs. da is conditioned by an instance code unique to every video instance V,, m, is trained by combining (1) auto-decoding framework to regress to a code on and (2) encoding-frame framework to regularize the space using CLIP embedding that generates V, semantic code Jn. At the time of inference, m, is randomly sampled from an underlying learned distribution T.
Generating a novel video V: translates to generating a novel implicit function So, that represents the video meaningfully. Let consider fo,, an unseen sample from an underlying distribution d. Each point in the distribution denotes an implicit function of meaningful video. To randomly sample faz, make use of meta-network to learn the distribution +b
We use hypernetwork da a meta-network to parameterize fo, such that dm = 0n for video instance Vn. Here m, is a a d-dimensional point in the latent space, say т, and serves as an instance code for Vn. Given enough number of samples N, de learns to map the latent codes sampled from T to their corresponding parameterized space $, as shown Fig. The parameters 6 are then used initialize f to generate Vn.
Let us consider 7 a mela-distributtin such that {m}o 7. At the time of inference, m, can be sampled 4, my enables dg to generate meaningful implicit
from As do has learned valid representation function fe, E d. Sampling from can be made straight forward by making sure is regularized during training. At the time of training, Ω and {m, are optimized together. 0 is non-lcarnable parameter and f is initialized as the output of dg- The following objective is optimized:
1111 N
W ΣΣΣ
L( m) NTWH
Σ
(fo, (h, w, RGB,2k) and 0, ()(mm))
(2)
3.2
Regularizing for Hypernetwork Conditioning
To generate novel video, a random latent my is sampled from the latent space т. dg is then conditioned on m, generating an implicit function fa. E 4. In a standard hypernetwork training Siizmaann et al. (2019) 2021 Park et al. (2019): Sitzmann al. (2020bb, m, is optimized in an auto-docoding framework as given
Equ. 2 However, given the complexity of the signal V (a 3D volumetric representation) that da has model, m p collapse to single point if not regularized at the time of training, bringing the expressiveness
5

---

Published in Transactions on Machine Learning Research (10/2022)
of de down to single implicit function. We regularize T by leveraging pretrained CLIP Radford et al. (2021) h designed for generating semantically meaningful embeddings for images. We design Video-CT.IP that encodes an entire video Vn to a vector In. As shown in Fig. 4 Video-CLIP first generates the image-level CLIP embeddings. These embeddings are then passed through a bi-directional GRU. The mean of the hidden stale outputs of the final layer produces In- As shown Fig. 3 the regularized instance code mn is now:
My (C))
where is the instance code of Vn, optimized in an auto-decoding fashion at the time of training, and e is a neural network. The pretrained CLIP embeddings are kept frozen during training and the learnable param- eters are the instance codes Cn that are regularized by Jn- CLIP regularization encourages the latent codes to be spaced sufficiently apart by leveraging predefined semantic encoding. This helps avoid mode collapse during the initial stages of training. Please find the ablation on CLIP regularization in Appendix A.1.1
3.3 Progressive Training
A video dense 3D volume mandating its neural representation to model every single point in the volume. Although implicit rep- resentations have a constant number of parameters made of only a few layers of MLPs in our case, learning meta-function using a hypernetwork over such dense representations is challenging. As a result, if not appropriately initialized, the snpernetworks can eas- ily collapse to a single representation despite CL.IP regularization. Moreover, a sub-optimal hypernetwork initialization could result in
Video-CLIP
Average
Bi-GRU
CLIP
CLIP
CLIP
CLIP
a significantly longer convergence period. To tackle this challenge, we adopt progressive initialization scheme. Firstly, the training is divided into multiple stages. Each stage, denoted by ( where KC is the total number of stages, is made of subset of the training dataset D. The number of samples N₁ in each stage is given as:
Ni - (
Ni-1 + E
1>1
Figure 4: Video-CLIP Encoding a (4) video Vn to a latent vector gn by using image-level CLIP encodings.
1=1
where C is a constant and 6 denotes the mumber of additional sam- ples for 1th stage. Each step consists of {V datapoints that is computed as:
(V)) = ( + (V)) Noite
(5)
where the order of set (V) is maintained across the training stages. At the start of the training, the model is trained on C < 10 examples. This allows the hypernetwork to quickly adapt to the handful of examples and initialize the weights. However, jumping from C to ~ 30,000 samples causes the network to collapse again. Thus, we adapt the network progressively to the given examples. Fach stage of the progressive training a full training of the model, with the weights in the current stage initialized with the weights Icarned from the previous stage. This includes reusing the instance codes C, learned at previous stage 1-1 in the current stage l. This step is crucial, as without this, the hypernetwork is pushed to re-learn all the instance codes. The new instance codes added in the current stage are initialized from Gaussian distribution.
4 Experiments
Experimental Sctup: We perform our experiments on (1) How2Sign-Faces Duarte et al. (2020), (2) SkyTinelapse Xiong et al. (2017), (3) Moving-MNIST Srivastava el al. (2015), and (4) Rainbow- Jelly Skorokhodov et al. (2021). Real video samples of each dataset are visualized in Appendix Fig. 19 How2Sign Duarte et al. (2020) a full-body sign-language dataset consisting of 11 signers. The signers have elaborate facial expressions, mouth, and head movements. We modify How2Sign to How2Sign- -accs by cropping the face region out of all the videos and randomly sample 10,000 talking head videos, each of
6

---

Published in Transactions on Machine Learning Research (10/2022)
Single- INR
INR-V
Dataset
3 4
PS t
SSIMso
Eso
PSNR50
SSIM50
PSNRFULL
SSIMFULL
How2Sign- Faces
4.83
29.72
0.925
8.29
25.69
0.850
25.84
0.869
SkyTTicclape 4.69
36.19
0.943
5.87
33.69
0.931
33.94
0.924
Moving-MNNI 3.57
37.26
0.978
6.06
29.81
0.949
29.54
0.975
RainbowJellly 4.17
35.93
0.918
5.02
33.34
0.937
33.57
0.938
Table 1: Quantitative metrics on reconstruction quality. Comparison set is made of 50 videos per training dataset. INRs trained individually for each video is denoted as Single-IRR. INR-V trains a single hypernetwork dg populate the comparison set,
the INRs of all the videos in the training dataset. PSNRao and SSIMse PSNRFULL and SSIMFULL are computed on the entire training set. is computed on videos with pixel range [0. 255]. INR-V performs comparably with Single-INR despite getting trained on huge datasets of more than 30,000 videos.
computed
Method
How2Sigu-Faces
Sky SkyTimelapse
Moving-MNIST
RainbowJelly
MoCoGAN-HD
396.53
321.44
296.95
1856.21
DIGAN
165.89
135.60
144.97
408.19 19
StyloGAN-V 94.64
85.05
109.85
1227.70
INR-V 161.68
153.42
103.24
260.72
+ Denoising
87.22
47.28
Table 2:
FVD16 metrics computed on random videos generated by the respective models.
at least 25 frames, of dimension 128 x 128. Sicelieeee Xiong et al. (2017) consists of scenic videos of sky changes. It made of 1803 videos, each at least 25 frames. The videos are first center-cropped to 360 x 360 from an original dimension of 360 x 620 and then resized to 128 128 for training. Moving-MNISTT Srivastaval (Sriatssivvssata et al. (2015) is a video dataset of moving MNIST digits containing a total of 10,000 datapoints. Each video is 20 frames long. Rainbow.Jelly is single underwater video capturing colorful jellyfishes. The video is first extracted into frames which are then divided into videos of 25 frames cach, making total of 34,526 videos. Similar to 3kyTimelapse, the videos are first center cropped to 360 x 360 and then resized to 128 x 128.
All experiments are performed on 2 NVIDIA-GTX 2080-ti GPUs with 12 GB memory each. All models, except INR-V, V, are trained at a resolution of 128 x 128. To make training computationally efficient, INR-V is trained on a lower resolution of 100 x 100 videos. Based on INRs, INR-V can infer directly at multiple resolutions (please refer section 5.2 For evaluations and comparisons, INR-V is inferred 128 128 like the other models. The training setup and model architecture are explained in Appendix A.2
4.1 Comparing INR-V with Single-INR
INR-V uses hypernetworks to learn a distribution over the INRs of videos. A single hypernetwork dg can initialize the INRs for multiple videos {V.) based on their respective instance codes m,- Thus, measuring if da generates the INR functions fo accurately crucial. We evaluate this using a set of 50 randomly sampled videos from the training dataset. Each video first trained fit a single INR function fo, using Eqn. []] denoted as Single-INR. Next, the LNRs of these 50 videos are populated using a pretrained hypernetwork da trained the entire dataset. We mcasure the reconstruction quality with PSNR (Peak Signal to Noisc Ratio), SSIM (Structural SIMilarity), and the error as:
1 50 Σ
HWT (V. -V,)²)))
(6)
where V. denotes the video generated using the implicit function fe. Single-INR was optimized for 750 steps using Eqn. taking ~ 5.56 minutes for each video (~ 4.63 hours for 50 videos). Table. presents quantitative
TTe codebase, dataset, and pretrained models can be found

---

Published in Transactions on Machine Learning Research (10/2022)
4.4 A
4
Mrt
0. 4
4.4
7
7
7
7
7
7
4
3
2
2
4
7/4
%
--
/
/
A
t
--
%
a
1
17
?
2 l
7
?
?
1
4 6
9
6
9
6
9
Figure Examples of random videos generated on, from top to bottom, How2Sign Faces
Duarte et al. (2020)
Sky'I T'imelapse Xiong et al. (2017), Rainbow.Jelly, and Moving-MNIST SSrvaatavaa et al. (2015). For Moving MNIST, every 2nd frame of 20 frames long videos and for other datasets, every 3rd frame of frames long generated are shown.
MoVing-MNIST and How2Sign-Faces are passed through VQVAE2 denoising network described in Section 1.2
metrics on the videos reconstructed using Single-INR and INR-V. PSNRFULL computes the PSNR on the entire training dataset, PSNR50 computes the metric on the selected 50 videos for comparison. As can be soen, although hypernetwork da is trained on huge datascts, it performs comparably with Single-INR. For Rainbow.elll, it even outperforms Single-INR in SSIM metric and performs at par on Sky Timelalsee This indicates that dg has learned to accurately generate INRs for complex mrati-ttemporal signals. Thus, INR-V can be used as a compression technique to compress 1000s of videos with minimal loss perceptual quality.
4.2 Comparing INR-V with SOTA video generation networks
Overview:
Fig. and Table present qualitative and quantitative comparisons respectively between
MocoGAN-HD Tian al. 22021 DIGAN Yu et. al. (2022). StyleGAN-V (Skorokhodov et. al. (2021), and

---

Published in Transactions on Machine Learning Research (10/2022)
Oure StylGGAN- DIGAN McCoGANN-
Figure 6: Each cell displays 12th frame of 25 frames long generated videos. The videos demonstrate interpolation between two given videos (in red boxes) by traversing along a trajectory in the latent space connecting the latent points of the given videos. Here, MoCoGAN+ and StyleGAN+ denote HDD and StyleGAN-V. White boxes indicate a sudden transition in content (e.g. identity) or motion (e.g. pose).
INR-V. All models were trained from scratch. As we train the models on smaller datasets of 10,000 datapoints, MoCoGAN-HD is trained on StyleGAN2-ADA Karras et al. (2020) mmage-generator backend. For each model, the best-performing checkpoint. selected for comparison.
Evaluation: As can be seen in Fig. INR-V generates novel videos with coherent content and motion. MoCoGAN-HD fails to maintain the identity in single video instance. For quantitative evaluation, use the Frechet Video Distance (FVD) metric as implemented by StyleGAN-V. FVD1s is computed on 2048 videos of 16 frames sampled at a resolution of 128 128. As can be seen in Table 2 INR-V outperforms the existing networks on Moving-MNNIN MNIST and RainbowJelly and performs comparably on the remaining datasets.
Enhancing INR-V's Visual Quality Enhancing image and video quality has been an area of extensive research Yang et al. (2021); Chu al. (2020); Liang et al. (2022); Chadha al. (2020) with many break- throughs. We propose that video generation be partitioned into two stages (1) generating coherent content and motion (2) enhancing the visual quality, Note that, in the current work, our effort has been (1) to propose novel continuous representation space for videos. We demonstrate (2) by developing a simple demoising network using standard VQVAE2 Razavi et al. (2019). We train VQVAE2 as a frame-by-frame denoising autoencoder making one minor change: Instead reconstructing the given low-quality input, we use the high-quality frame for computing the error. The low-quality inputs are the intermediate video instances reconstructed by INR-V during training. We train denoising VQVAE2 on How2Sign-Facee and Moving-MNTST Appendix Fig. T8 demonstrates the results of the denoising network on blurry instances generated by INR-V, As can be seen from the quantitative metrics in Table. 2 using an additional denoising network improves the network's performance by ~ 2x.
5 Applications of the continuous video space learned by INR-V
INR-V learns continuous latent representation for videos allowing complex spalio-temppoal video signals to be represented using single latent point. In this section, we showcase the advantage of such latent space through several demonstrated properties and comparisons. We also benchmark several tasks based on the inversion of 256 videos on Hlw2SiinnFacce Faces using full and incomplete video context.
5.1 Video Interpolation
Given two vidcos Vi and V2, a continuous vidco space should be able to make a gradual transition between the two videos such that every point along the trajectory between the two (1) produces a meaningful video and (2) shares content and motion properties from Vi and V2. We demonstrate this property in Fig. and Fig. B with
9

---

Published in Transactions on Machine Learning Research (10/2022)
64 64
frames
25 framms
128 128
256 > 256 frames
100 100 50 fromes
Figure 7: INR-V direct inference on multiple resolutions and frame length. INR-V trained on only 25 frames long 100 x 100 videos. Novel videos of multiple resolutions (64 64, 128 x 128, 256 256) and video length (50) are directly generated on the trained model without any architectural change finctuning.
The images are not upto scale, please refer Appendix Fig. 25 for scaled representation.
Spherical Linear Interpolation (Slerp Each cell in Fig. 6 demonstrate the 12 frame of the 25 frames long videos. As can be seen, INR-V observed a gradual change in motion (pose, mouth movements, expressions, cloud shift) and content (identity, visibility of sun). The interpolated vidcos are spatio-temporally coherent (best seen in the supplementary video). Appendix Fig. 23 and Fig. 24 demonstrate the spatio-temporal transition on How2Sign-Faces and SkyTimelapse. As represent an entire video in a single point in the continuous video space, interpolation is natural operation that can be performed with INR-V.
Comparisons: Existing models have different motion and content codes; thus, to interpolate videos, intermediate content codes were interpolated between two videos by Slerp interpolation. INR-V does not have separate motion and content vectors; thus, videos can be interpolated directly using given video's latent points. As shown in Fig. 6 INR-V has gradual transition in motion and content. For How2Sigu-Faces, StyleGAN-V abruptly changes motion (cell 5-7), and DIGAN abruptly switches identity (cell 1-2, cell 5-6). This effect is highlighted in white boxes. This is expected as both of these architectures operate in the image space, and thus a gradual spatio-temporal transition is harder to achiove. We performed a user study on 30 users to qualitatively evaluate the interpolation quality of INR-V against the SOTA models and report the metrics in Table. 3 INR-V interpolation was randomly shown against either of the other three models. The users provided their preference on which interpolation looked smoother in terms of transition in content and motion. INR-V was preferred at least 85% more than all the SOTA networks. This demonstrates the continuous nature of the video space learned by INR-V.
MoCoGAN-HD DIGAN
StylGANNV
100.00
89.43
95.24
Table 3: Video interpolation study: % of times INR-V interpolation was preferred over existing models.
5.2 Multi-Resolution and Multi-Length Inference
In Fig. 7 we show INR-V trained on videos of only 100 x 100 resolution with 25 frames per video, generating novel videos of multiple resolutions and lengths, maintaining the content and motion quality of the output. An underlying property of LNRs is a contimuous representation of the signal given as fo w, + RGB. This enables the model to understand a continuous property of the signal making it agnostic to the dimension. We show quantitative metrics on INR-V inferred at multiple resolutions and compare INR-V V with existing SOTA superresolution techniques Chen et al. (2022) in Appendix A.5
5.3 Video Inversion and its applications
Inversion has been widely adopted in many applications prominently for images. StyleGAN2 Karras et al. 2019 is extensively used for image inversion enabling many downstream image editing tasks such as changing the emotion, age, or gender of a given face. In video inversion, we aim to invert a given video back into the latent space of pretrained video generation network. Existing methods perform frame-by-fran inversion to individually invert the context code for each frame and the motion code for the video. In INR-V, we only
10

---

Published in Transactions on Machine Learning Research (10/2022)
need to invert to single latent code that can be achieved through simple optimization objective:
1 t=l ==
argmin TW ΣEΣ
(fo, (h, t) RGB,)² where 02 (m
(7)
where my is the latent point for a video instance V.. Fig S shows the qualitative demonstration of INR-V inversion trained on How2Sign-Paces for a video outside of the training dataset D.
Video Complction: Key categories of 'vidco syn- thesis' include future frames prediction (future pre- diction), completing the video between frames (frame interpolation), and predicting the missing part of the video (video inpainting). In INR-V, a video V2, rep- resented by a single latent code m, can be gener- ated without any additional knowledge. Thus, all the above operations can be performed using modified
Ground Truth
Video Inversion
Video Inpainting + Prediction
Context
optimization operation based on Equ 7 on the seen part of the video given as:
argmin S (fo, (he, Wos ts) R
mi 5 and 0g = (m())))
(8)
Frome Prediction
where S' is the number of context points, has ws and t, are the context points of V₂ seen at the time of op- timization. With the optimizod m2, the full vidco can simply be generated back with INR-V. Fig. demon- strates the results for the various operation on video outside of 'D with 2.5 minutes of optimization single 12 GB NVIDIA GTX 2080ti GPU. As can be seen, the network is able to regress to latent corre- sponding to the given identity while preserving finer details like spectacles, mouth shape, pose, etc. In the
Context
Prediction
Frome Interpalaction
a
Context
Prediction
Contect
Contaxt
Spore Inpainting
Prediction. Context Prod croon ctoan
case of 'Video Inpainting', the network understands the person's pose. For 'Frame Prediction', although the pose does not match the ground truth, the over- all video is coherent. In 'Frame Interpolation', the model is able to generate coherent context between two frames, including the pose, expressions, identity, and mouth movements. In 'Sparse Inpainting', we
SuperReollltt oofon-- +
randomly set 25% of all the video pixels as the con-
text points for optimization. Even with very sparse context, INR-V able to regress to the correct spec- ifications including the finer content. details.
Figure 8: Video Inversion and it's applications. INR-V can be directly used for several tasks by simply inverting video to its latent point based on the given context. We demonstrate some qualitative results.
Vidco Superrcsolution through inversion: Video Superresolution is the task of enhancing the resolution of a given video. Recent works such as Chu et al. (2020): Liang et al. (022); Sajjadi et al. (0088): Chadha et al. (2020); Wang et al. (2019); Chen et al. (2022) have made significant progress in video super- resolution, showcasing 4x enhancement. INR-V can directly superresolve seen video instances as showcased in Appendix Table. For unseen instances, combining the capability of video inversion and multi-resolution vidco generation, INR-V can superresolve a given vidco V, of lower resolution (say 32 32) simply following: (1) Invert V. at the smaller resolution to obtain m., (2) Render V. from my directly at higher resolution (say 256 x 256). In Fig. g) we demonstrate the qualitative results on a video outside the training dataset. The video was optimized at 32 x 32 for ~ 2.5 minutes. The inverted video was then superresolved at a scale factor of 8x to 256 x 256. Additional details are present in Appendix A.5
11

---

Published in Transactions on Machine Learning Research (10/2022)
Quantitative Evalua- tion: To quantify the performance of INR-V, we prepare a comparison set by randomly sam- pling 256 videos outside of the training set. We compare against DIGAN on the tasks of Video Inversion, Video Inpaint- ing, Frame Prediction, Frame Interpolation, and Sparse Interpolation and against StyleGAN-V on the task of Video Inversion. Since DIGAN is based on INRs, it can invert in- complete frames, however, StyleGAN-V expects a full frame for backpropagation. Thus we do not compare with StyleGAN-V on the other tasks. For the task of Superresolution, Upsampling and VideoINR at scale factor of 4x from 32 x 32 to 128 x 128.
Task
Method
GT-ID
TL-ID
IG-ID1 f
Context-Ii L1
PSNR 1
SSIM f
Cost
DIGAN
0.652
0.953
0.9599
45.08
19.59
0.653
4.25
Inv.
Style
0.804
0.985
0.998
42.16
19.65
0.665
3.25
INR-V
0.770
0.950
0.950
5.25
21.21
0.773
2.75
DIGAN
0.628
0.960
0.969
45,80
4.25
Inp.
INR-V
0.758
0.948
0.939
4.83
2.75
DIGAN
0,603
0.940
0.928
40,26
-.25
Pre.
INR-V
0.703
0.946
0.932
4.72
2.75
DIGAN
0.653
0.925
0.921
48.66
~ 4.25
Int.
INR-V V
0.702
0.928
0.905
7.46
~ 2.75
DIGAN
0.718
0.961
0.967
46,24
19.74
0.671
~ 4.25
Spr.
INR-V
0.768
0.968
0.974
5.29
22.35
0.774
2.75
Bicubie
0.808
0.923
0.903
28.36
0.906
Sup. 4x
VideoINR
0.939
0.982
0.974
32.86
0.957
INR-V
0.734
0.911
0.903
4.92
21.94
0.742
2.75
Table 4: Comparison of INR-V on various video inversion tasks: Video Inversion (Inv.), Video Inpainting (Inp.), Frame Prediction (Pre.), Frame Interpolation (Tnt.), Sparse Interpolation (Spr.), and Superresolution (Sup.). Comparison set made of 256 videos outside of the training dataset. Metries used for evaluation is explained
Sec. 5.3 Cost denotes the time to optimize a single video instance minutes.
compare against Bicubic
We evaluate on the following metrics: (1) PSNR, (2) SSIM, (3) Temporally Locally (TL-ID) and Temporally Globally (TG-ID) Identity Preservation, (4) Context-L1, and (5) Ground Truth Identity (GT-ID) Match. TL-ID and TG-ID were proposed in Tzaban et al. (022). They evaluate a video's 's identity consistency at a local and global level. For both metrics, a score of would indicate that the method successfully maintains the identity consistency of the original video. Context-LL computes the L1 error on the inverted videos at the given context points. An error of 0 would indicate that the inversion is perfect. GT-ID measures the match in identity between the ground truth and the inverted video. DeepPace face features are extracted for both the videos, and the cosine similarity is computed between the extracted features. Since there is no single correct prediction for tasks like FFtture Frame Prediction', 'Frame Interpolation', and 'Video Inpainting', we do not evaluate these tasks on PSNR and SSIM.
As can be seen, INR-V outperforms all the existing networks in most of the metrics on video inversion and the proposed inversion tasks, except 'Superresolution', indicating the advantage and robustness of the proposed space. For the task of Superresolution, INR-V performs comparably with Bicubic and VideoINR. However, unlike these works that directly superresolve video, INR-V first inverts the low resolution video to generate a high resolution vidco. Such a mechanism opens several possibilities, such as inverting a low resolution incomplete video (missing frames due to corruption) to a high resolution video with full context.
6 Conclusion
We present INR-V, continuous vidco representation network. Unlike existing architectures that extend superior image generation networks for generating videos one frame at time, we use implicit neural rep- resentations to parameterize videos as complete signals allowing meta-network to encode it to a single latent point. Given enough examples, the meta-network learns a continuous video space as demonstrated through video interpolation and inversion tasks. INR-V generates diverse coherent videos outperforming many existing video generation networks. INR-V opens the door to a multitude of video-based tasks and removes the dependency on an image generator. To showcase this, we propose several downstream tasks and observe that INR-V V outperforms the existing works on a majority of these tasks. This demonstrates the advantages and potential of continuous video space and we hope to encourage research in this direction.
http://gghhabccmm//rrrrgll///ddp
12

---

Published in Transactions on Machine Learning Research (10/2022)
References
Hakan Bilen, Basura Fernando, Efstratios Gavves, Andrea. Vedaldi, and Stephen Gould. Dynamic image networks for action recognition. In 2016 IEEE Conference on Computer Vision and Pottern Recognition (CVPR), pp. 1034-3042 2016. doi: 10.1109/ 10.1109/CVPR. /CVPC.22......311.
Aman Chadha, John Britto, and M. Mani Roja. iSeeBetter: Spatio-temporal video super-resolution using recurrent generative back-projection networks. Computational Visual Media, 6(3):307-317, jul 2020. doi:
10.10077/441095-000-017--- URL
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser-Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for vidcos, 2021. URL http::///aaxiv.orr/abb//////////
13903
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous spaco-time super- resolution, 2022. URL https://arxxv..rrt hwwpg//bb//200.... 2200604647
Pei-Ze Chiang, Meng-Shiun Tsai, Hung-Yu Tseng, Wei-sheng Lai, and Wei-Chen Chiu. Stylizing 3d scene via implicit representation and hypernetwork, 2021. URL attps://arxiis wwrgg/bss/2ue 2205/ 13016
Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coherence via self-supervision for GAN-based video generation. ACM Tronsactions on Graphics, 39(,), aug 2020, doi: URL (0.1145/3386569333222 URL /cttss///aadl.oor//1.14453333333
Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial video generation on complex datasets, 2019.
Zihan Ding, Xiao-Yang Liu, Miao Yin, and Linghe Kong. Tgan: Deep tensor generative adversarial nets for large image generation, 2019. URL https://arxii: http://axxvv..or//aaaa 09953
Amanda Duarte, Shruti Palaskar, Lucas Ventura, Deepti Ghadiyaram, Kenneth DeHaan, Florian Metze, Jordi Torres, and Xavier Giro-i Nieto. How2sign: A large-scale multinodal dataset for continuous american sign language, 2020. URL https://arxiv. org/ abs 2ttp:/200....11100
Basura Fernando, Efstratios Gavves, Jose Oramas M, Amir Ghodrati, and Tinne Tuytelaars. Modeling video evolution for action recognition, 06 2015. doi: 111.10//CVPR.2215.229111
Shangqian Gao, Feihu Huang, and Heng Huang. Model compression via hyper-structure network, 2021. URL httpp:://openrevie teee/foorum
Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna, William T. Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions, 2019. URL https://arxiv org/ abs/ 1904.
06447
Yudong Guo, Keyu Chen, Sen Liang, Yong-Jin Liu, Hujun Bao, and Juyong Zhang. Ad-nerf: Audio driven neural radiance fields for talking head synthesis, 2021. URL httpp://aaravv.rr//aaa/////////
David Ha, Andrew Dai, and Quoc V. Le. Hypernetworks, 2016. URL httpe://arrivvooorg/b///000
Tero Karras, Samuli Taine, Milka Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan, 2019. URL
Tero Karras, Mika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data, 2020. URL https://arxxvv.rrg orgg/bs////aaeeeo/osssses//////e 06676
Angus Lamb, Evgeny Saveliev, Yingzhen Li, Sebastian Tschiatschek, Camilla Longden, Simon Woodhead, José Miguel Hernández-Lobate Richard E. Turner, Pashmina Cameron, and Cheng Zhang. Contextual works for novel feature adaptation, 2021. URL httts:///axxi...e. http://aaavv..oaaa/////aa
05860
Jingyun Liang, Jiezhang Cao, Yuchen Fan, Kai Zhang, Rakesh Ranjan, Yawei Li, Radu Timofte, and Luc Van Gool. Vrt: A video restoration transformer, 2022. URL http::://aaxii...r///a////////// 12288
13

---

Published in Transactions on Machine Learning Research (10/2022)
Pauline Luc, Aidan Clark, Sander Dieleman, Diego de Las Casas, Yotam Doron, Albin Cassirer, and Karen Simonyan. Transformattion-based adversarial video prediction on large-scale data, 2020. URL https:
harxiv.orggabs/2003..o 04035
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy
networks: Learning 3d reconstruction in function space, 2018. URL https:// http:///aaxxi..o//aa/a////////// wrw.//////bbb/1.. 03828
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis, 2020. URL https://arxiv arg/abs/2003.000
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: Context-aware controllable video synthesis, 2021. URL htttp:://axxiv.org/aab//////////
08037
Phuoc Nguyen, Truyen Tran, Ky Le, Sunil Gupta, Santu Rana, Dang Nguyen, Trong Nguyen, Shannon Ryan, and Svetha Venkatesh. Fast conditional network compression using bayesian hypernetworks, 2022. URL
wttp://wwwrrrgaaa/22...... 06404
Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation, 2020. URL https: //apsii//arxvv org/abe htt///bb/20000555333
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution, 2017.
URL https://arxiv.ot org/ www.aa//.........
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Decpsdf:
Learning continuous signed distance functions for shape representation, 2019. URL https://arxiv..r org/
abs/ abs/1901.05103
Junheum Park, Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame interpolation, 2021. URL
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL
wrgp.bs////...ooo. 00020
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2, 2019. URL
00446
Mchdi S. M. Sajjadi, Raviteja Vemulapalli, and Matthew Brown. Framo-rocurrent video super-rosolution, 2018. URL httpttp://aaiv.rrr////////000000
Marcin Sendera, Marcin Przewiizlikowski, Konrad Karanowski, Maciej Zięba, Jacek Tabor, and Przemyslaw Spurek. Hypershot: Few-shot learning by kernel hypernetworks, 2022. URL htt.hhtpps.. //trx//a/vvvvv//// org/ abs/ 2203. 11378
Vincent Sitzmann, Justus Thics, Felix Heide, Matthias Nießnor, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings, 2018. URL https://arxiv http://aaxxivvorr/a/////11/ 01024
org/abs/18122
Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Contin- uous 3d-sstructree-wware neural scene representations. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL pdf.
Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta- learning signed distance functions, 2020a. URL htttpp:://axx....rr/aaa/////////
14

---

Published in Transactions on Machine Learning Research (10/2022)
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions, 2020b. URL https://arxiv org/ abs/ 29066096661
Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Joshua B. Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with singlo-evaluation rendering, 2021. URL
org/abs/2106. 02634
Ivan Skorokhodov, Savva Ignatyev, and Mohamed Elhosciny. Adversarial generation of continuous images, 2020. URL
22200
Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny, Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2, 2021. URL ptttps://axxiv.org/bbb////// 14683
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhutdinov. Unsupervised learning of vidco representa- tions using Istms, 2015. URL http:://arrivv.rrr/aaa////////
Joseph Suarez. Language modeling with recurrent highway hypernctworks. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL httpss//ppocceedigs..eeet hew.:/eeeeddignn..nerrrs........ www.gg.....urrs.... cc/
pddf
Yu Tian, Jian Ren, Menglei Chai, Kyle Olszewski, Xi Peng, Dimitris N. Mctaxas, and Sergey Tulyakov. A good image generator is what you need for high-resolution video synthesis, 2021. URL hntpss//arxii. org/ ht5/abb/211044. 15069
Sergey Tulyakov, Ming-Yu Yu Liu, Xiaodong Yang, and Jan Kaulz, Mocogan: Decomposing motion and content for video generation, 2017. URL
04993
Rotem Tzaban, Ron Mokady, Rinon Gal, Amit H. Bermano, and Daniel Cohen-Or. Stitch it in time: Gan-based facial editing of real videos, 2022. URL https://arxiv wttp://rrr./ab//...././/
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks, 2019. URL httts:://///tt org. httg/ab//1996..0000
Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae, 2021. URL https: //arxiv.org/abs/ 00950
Xintao Wang, Kelvin C. K. Chan, Ke Yu, Chao Dong, and Chen Change Loy. Edvr: Video restoration with 02761
enhanced deformable convolutional networks, 2019. URL
Wei Xiong, Wenhan Luo, Lin Ma, Wei Liu, and Jiebo Luo. Learning generate time-lapse videos using multi-stage dynamic generative adversarial networks, 2017. URL
1709
Wilson Yan, Yunzhi Zhang, Picter Abbeol, and Aravind Srinivas. Vidcogpt: Video generation using vq-vae and transformers, 2021. URL httph::/ttaaatttee http://axxiv..o//ab//a//// httpg/ab/2104..100000
Tao Yang, Peiran Ren, Xuansong Xie, and Lei Zhang. Gan prior embedded network for blind face restoration in the wild, 2021. URL https://aaxxv..rg/abb////// 06070
Shunyu Yao, RuiZhe Zhong, Yichao Yan, Guangtao Zhai, and Xiaokang Yang. Dfa-nerf: Porsonalized talking head generation via disentangled face attributes neural rendering, 2022. URL dettss//arxiv..om orgabbs// abs/ 2201.00791
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung- Woo Ha, and Jinwoo Shin. Generating videos with dynamics-awar aware implicit generative adversarial networks, 2022. URL
org/
Youjian Zhang, Chaoyue Wang, and Dacheng Tao. Video frame interpolation without temporal priors, 2021.
URL attps://ee //tr:////vvv.////a www.:org/ggoo.... arg/abs/2112..... 01161
15

---

Published in Transactions on Machine Learning Research (10/2022)
A
Appendix
A.1 Ablation
A.1.1
Effect of Regularization
Ground fruhh
CUP
CUP
[a] Reconstruction of an Intermediate Step
(b) Random Samples
Figure 9: Qualitative results of CLIP regularization. Intermediate results shown after 30 hours of training 2 NVIDIA GTX 2080ti GPUs. (a) Video reconstruction quality. CLIP regularization enables the meta- network to model the INRs with finer details. (b) Videos generated by random sampling. CLIP regularization improves the quality of the sampled videos and encourages variation in the implicit representations.
2500
900
HodeCrlaper
No Regularization
No Regularization
Progressive Training
800
Clip Regularization
2000
Regularization
Gaussian Regularization
Clip Reg. Progressive
1500
500
400
1000
300
500
150
200
50
100
150
200
Time (in hours)
Time hours)
Figure 10: Convergence rate for different regularization schemes on Rainbow.Jlll (left) and Sky TkyTellapse (right) datasets. Rainbow:Jlll consists of ~ 34K datapoints and SkyTimelapse consists of ~ 2K datapoints. As can be observed, TkyTimmeppse being a relatively smaller dataset performs equally well on all the regu- larization schemes. RainbowJelly performs worst without any regularization facing mode-collapse for about the first 75 hours. Progressive training and CL.P regularization help INR-V converge the fastest.
In this section, we compare the training time and the performance of INR-V (1) with with/without CLIP regu- larization and (2) with/without progressive training. Fig. 9 presents the qualitative results on INR-V after 30 hours of training on NVIDIA GTX 2080ti GPUs. Fig. plots the rate of convergence on RainbowJJelly (left) and Sky Timeellsp (right) datasets on the same training setup. We also show Gaussian regularization with INR-V by adding the following additional loss term the overall loss term in Eqn.
8DKL (N,, 1))
(9)
LN and ò is hyperpa- where y and a denote the mean and standard deviation over the latent codes [m, rameter. In our experiments, 1.0.
As can be observed from Fig. M roconstruction quality is much worse without CLIP. This expocted Vidco- CLIP (see Fig. assigns semantic meaning to the initialized codes for each video instance. As we observe the
16

---

Published in Transactions on Machine Learning Research (10/2022)
novel video instances generated using this model (Fig. right), we already see a motion emerging with expres- sive faces. As can be observed from Fig. 10 on Rain- bowJelly dataset (made of 34526 instances), INR-V takes more than 250 hours (~ days) converge without any regularization scheme or progressive training. With progressive training, the convergence time is drastically reduced to less than 180 hours (~ 7.5 days). The best performance is achieved when progressive training is done along with CLIP regularization where the convergence occurs in less than 120 hours (~ 5 days) on NVIDIA GTX 2080ti GPUs. On SkyTimelapse dataset (made 1803 instances), INR-V converges equally on either of the regularization schemes. With Gaussian prior, we ob- served slight advantage in terms of convergence time. Fig. I plots additional comparison on RainbowJelly with progressive training on three different regularization methods: Gaussian, CLIP, and no regularization.
0
Progressive Training
Clip Reg. Progressive
2000
Gaussian Regularizatio
1500
1000
500
20
40
60
80
100
120
140
160
Time hours]
Figure 11: Convergence rate for different regulariza- tion schemes on RainbowJelly dataset made of ~ 34K instances. All the models are trained progressively, CLIP Regularization leads to the fastest convergence.
The graphs indicate that CLIP regularization is more suitable for a larger dataset like Rainbow.Jelly, however for smaller dataset like SkyTimelapse (Fig 10 right), Gaussian regularization is forms equally well on either of the training schemees, given enough time to train. This indicates that the genera- tion capabilitios is inherent to the proposed architecture, whereas the different training schemes help in stabilizing the training and thus, lead to a faster convergence. Ad- ditional insights are provided in Appendix A.4
effective. INR-V per-
400
250
A.1.2 Effect of the size of the codebook
5000
10000
15000
Codebook Size
20000
25000
30000
35000
Fig 12 presents a comparison between the FVDs INR- V when trained on varying number of vidco instances on the Rainbow.elll dataset. FVD is computed against the entire dataset made of 34, 526 samples As can be seen from the graph, the performance of INR-V deteriorates as the number of video instances reduce. However, the effect of the added instances is marginal as the codebook size increases; the FVD improving by only 12% when go- ing from 10K 34K (24K additional instances) video instances. However, the FVD) improves by 20% when the codebook size incrcases from 500 to 1000 (500 additional instances) video instances.
Figure 12: Performance of INR-V when trained on varying number of video instances (codebook size). FVD computed against the full dataset.
Intiaaization
2500
Intialization
Initialization
Intializetion
2000
1500
1000
A.1.3 Progressive Training with different Initializations
500
20
60
100
120
To stabilize the hypernetwork training, train our net- work progressively as explained in Sec. 3.3 In this see- tion, we compare the performance of INR-V on training with different video intializations. We randomly assign videos for cach stage of the training and the random as- signments differ across the different initializations. As
hours)
Figure 13: INR-V on progressive training with different initializations: the order of video selec- tion for each stage differ across the init initiaiizt.oo.
shown in Fig. 3.3 INR-V takes about the same time to converge for all of them.
17

---

Published in Transactions on Machine Learning Research (10/2022)
A.2 INR-V Implementation Details
The implicit neural representation fo is an MLP with three 256-dimensional hidden layers. Each hidden layer is passed through ReLU activations. The hypernetwork da is set of MLPs. Each MLP predicts the weights for a single hidden layer and the output layer of fg. Each MLP has three 256-dimensional hidden layers. CLIP embeddings are 512 512-dimensional vectors, Video-CLIP encodes the CT.IP embeddings of each frame through three 512 dimensional, GRU layers. As shown in Fig. 4 video-level embedding 9n- C. is a 512-dimensional context voctor that is regressed in an auto-decoding fashion during training. & is made of 3-hidden layers that lakes a 1024-dinensional vector as inpul. (concalenation of 9n. and c,) and produces my 128-dimensional instance code of Vn, as the input for dg. The input to So is a periodic positional encoding of [[h] [ W (1) as implemented in Sitzmann et al.) (2021). Adam optimizer is used with learning rate of 1e 4 during training and 1e - 2 during inversion tasks, No scheduler is used. Progressive training is done at power of 10 where gth stage is made of min(10', N) examples. i = O...K I. such that 10 < N + 1, where N is the total number of training samples. Each stage except the last stage is trained until the reconstruction error reaches a threshold of le -3 3.
Video-CLIP produces 512-dimensional
A.3 Comparison of Computational Complexity of INR-V against 3D Convolutional models
We compare the computational complex- ity of INR-V against standard implemen- tations of 3D convolution-bes cration modcls of varying spatial and tem- poral dimensions. We call these models as "3DConv'. 3DConvs do not generate any meaningful output and are used solely to compare the computation costs against INR-V. They comprise of several trans- pose 3D convolution-based upsampling lay- ers and take a fixed 128-dimensional latent vector as input. For comparison against INR-V, we gradually vary their spatial di- mension from 128 x 128 to 1024 1024 by keeping the temporal dimension fixed to 25 frames. To generate video of resolution 128 x 128, three 3D convolution- upsampling layers are used. An additional upsampling layer is added for every subsequent jump in the spatial dimension. Next, we vary the temporal dimension for 25, 50, 75, and 100 frames by keeping the spatial extent and the number of layers fixed to 256 256 and four, respectively, and adjusting the kernel size corresponding to the temporal dimension. The stride, kernel size, and padding are appropriately adjusted for all the models. The batch size for comparison is fixed to 1 for both 3DConv and INR-V. As can be seen in Table. 5 the number of operations (MAC) increases drastically prohibitively expensive to generate videos of higher spatial dimensions. For example, generating a single video of 25 frames of dimension 1024 x 1024 results in out of memory (OOM) on a single NVIDIA GTX 2080 ti GPU with a memory of 12 GB. In summary, MAC remains hundreds of orders of magnitude lower for INR-V compared to its 3D convolution-basel counterparts as it mainly comprises incxpensive MLPs.
128 128
256 x 256
512 x 512
1024 x 1024
video gen-
INR-V V
1.68
6.71
26.84
107.36
3DConv
252.71
691.76
3860.77
OOM
25 frames
50 frames
75 frames
100 frames
INR-V V
6.71
13.42
20.13
26.84
3DCouv
691.76
2036.05
3719.98
6673.27
Table 5: Comparison of computational complexity of INR-V against 3D convolutional- video generative models for dif- ferent spatial and temporal dimensions. The computational complexity is the total number of multiply-add Giga opera- tions denoted by GMAC (Multiply-Add Cumulation).
the spatial dimension increases for the 3DConvs. It. becomes
A.4 Insights on the learned latent space
Unlike the existing video generation networks that are conformed to predefined latent space (Gaussian or Uniform), INR-V learns space that best fits given distribution. The experiments (Sec. 5.1 and Sec. 5.3) and our observations indicate that the learned space is continuous, supports inversion, and smooth video interpolations. Thus, such a space learns a structure in the dataset. For instance, we observe a smooth transition across different poscs, expressions, mouth movements, and identity on How2Sign-Facce (best viewed in the Supplementary Video). Therefore, novel instances are observed as we traverse the path between seen latent points A and B. We use this property to generate novel videos by sampling latent points
18

---

Published in Transactions on Machine Learning Research (10/2022)
0.01
No Regulorization
0.00
-0.01
-0.01
0.00
0.01
0,02
-6
-2
7.5
10
CONNERR
5.0
CLIP Regularization
25
0.0
-2.5
-5.0
-10
-7.5
-8
-4
-2
-10
10
Epoch 1
Final Epoch
Figure 14: t-SNE visualization of the latent codes (m,) learned by INR-V with and without CLIP regular- ization on RainbowJelly. In Epoch 1, the latent codes are bounded within a very small region when INR-V is trained without any regularization (top-left). With CLIP regularization, the latents are more spread out (bottom-left). In the final epochs (top bottom right), both latent representations have spread out farther. INR-V without regularization forms a denser space; and with regularization converges faster. Both spaces continuous, form meaningful interpolations, and support inversion.
Seen Video A
Noval interpolated Videos......
Seen Video
Figure 15: We demonstrate the ability of INR-V learn the underlying structure of dataset. To do so, create toy video dataset called BouncingBall consisting of a blue ball bouncing horizontally at different heights. Given 50 instances of such videos, INR-V V learns the structure and we demonstrate the learned structure through interpolation. In this example, we show the 16 frame of each video with the novel interpolated videos in gray boxes. The videos in red boxes are seen during training, We can observe a correct interpolation with smooth transition motion: the height and the horizontal position of the ball gradually shifts from bottom to top and left to right respectively.
19

---

Published in Transactions on Machine Learning Research (10/2022)
through Slerp interpolation. In this section, we aim to validate the space learned by INR-V and answer the following question: what does INR-V learn? We analyse this in two ways: (1) by visualizing the latent codes learned by LNR-V through t-SNE visualization and (2) by training INR-V on a structured toy dataset to see if it learns the underlying structure.
A.4.1 t-SNE visualization
Fig. 14 plots t-SNFO on the latent codes learned by ANR-V on the Rain- bowJelly dataset. We see a clear pattern of 'interpolation' occurring in the learned latent space in both cases (with and without CLIP regular- ization). No progressive training is used in this visualization. At the end of the first epoch, the latent codes are tightly bounded when trained without CLIP regularization. As the model is trained with CLIP, the latent vectors are more spread out possibly in semantically meaningful manner. In both the cases, by the final epoch, observe patterns of interpolation evolve.
Video Mestsnoe. A
Video Insstance 8
Horizontal Motion
A.4.2 Learning Bouncing Ball
Figure 16: BouncingBall dataset with an infused structure. Each video instance is 100 x 100 and has a ball bouncing horizontally at specific height. Red lines are addod to show the height of the ball and is not a part of the videos.
In this section, we want to analyze if INR-V can learn the structure of dataset. To do so, generate a toy dataset, BouncingBall, with artificially infused structure. The dalaset is made of 50 video instances where each video instance consists of a blue ball bouncing horizontally at different heights as shown in Fig. [6. The videos are of dimension 100 x 100 and are 25 seconds long each.
[+1
1-14
1-
As shown in Fig. 15 INR-V is able learn the infused structure when trained on BouncingBall. We demonstrate interpolated videos by sam- pling intermediate points through Slerp interpolation. The intermedi- ate videos (grey boxes) demonstrate a smooth interpolation, and have heights and horizontal displacements gradually varying from Video A to B (red boxes). An example of an intermediate video shown in Fig.
Movel Instarice
Recometnctee
Instance
A.5 Inferring at Multiple Resolutions and Multiple Lengths
An underlying property of INRs is a continuous representation of the signal. This allows inferring INR-V on multiple spatial and tempo- ral resolutions directly without changing the model's architecture or additional finetuning. To generate a video of an arbitrary dimension Ā W f, those many number of equally spaced points are sampled between [-1, 1]. In Table G we report FVD16 scores on random vidcos generated on INR-V with varying spatial dimensions on 25 frames. To infer at multiple resolutions, INR-V pretrained on 100 x 100 dimensional videos of 25 frames was used. Note that the FVD scores do not degrade even at higher spatial resolutions. Additionally, we compare INR-V with existing SOTA superresolution techniques Chen et al. 2022) in Table 7 on 2048 videos randomll sampled from the Rainbow.elll dataset. As can be seen, INR-V performs comparably with methods on the task of superresolution. Moreover, INR-V can be superresolved to any arbitrary resolution (3.6x) and aspect ratio. Please note that we do not solve the task of superresolution but rather show superresolution as potential application of our work.
Figure 17: Videos generated by INR-V on BouncingBall. Novel video is generated at an unseen height. Red lines are for demon- stration and not generated.
A.6 Discussion
Limitations. Although INR-V has learned a powerful video space demonstrating several intriguing prop- erties, the videos generated by the model sometimes blurry. This is prominent when moving away to unseen points in the video space far from the seen instances. Fig, 18 demonstrates the enhancement on one
http::///dmmmaaeengithbb.
20

---

Published in Transactions on Machine Learning Research (10/2022)
such blurry sample. This is done by training standard VQVAE2 network in denoising fashion (please refer to Sec. 4.2). However, the entire process is broken into generating a relatively lower quality output and relying on a second network to improve its quality. A single end-to-end network capable of retaining the demonstrated powerful properties while generating high-quality videos is a potential future work.
Another limitation of INR-V is generating infinitely long videos. Although coupling the content and time into single latent has clear advantages, it removes the network's ability to leverage the temporal dimen- sion separately and find infinitely long temporally co- herent paths in the image space. This can be tackled by training INR-V V to encode video segments of mul- tiple lengths in a single space (1 to 50 or more frames long video segments). temporally and semantically coherent trajectory bet these video segments can then be learned. Such generation technique would directly leverage video segments and potentially re- move repetitions in the long videos. We believe that leveraging a video space for generating infinitely long videos at multiple resolutions presents an interesting and exciting direction for future research.
128 x 128 256 x 256 360 x 360
INR-V (Ous)
260.72
232.43
251.14
Table 6: FVD1e (L) metrics on random video generation at multi-resolution on INR-V. Training was done on only 100 x 100 dimensional videos of 25 frames. Inference was taken directly on multiple resolutions without any finetuning or architectural changes.
2x
3x 300
3.6x
201
2010
300
360 x 360
PSNR 1
SSIM
PSNR f
SSIM f
PSNR
SSIM
Bleubic
31.53
0.884
32.13
0.915
32.31
0.920
VideoINR INR-V
31.59
0.863
33.01
0.913
0.892
29.17
0.891
29.05
0.896
Table 7: Quantitative metrics video superresolu- tion using INR-V and SOTA superresolution networks on video instance seen at the time of training. INR-V V was trained at 100 x 100 video resolution. INR-V performs comparably with the SOTA superresolution nelworks.
Lastly, we observed that INR-V does not learn meaningful representation space when trained on datasets like UCF-101 that have extreme diversity and limited structure in motion. A similar issue was observed when training the basclinc models on such datasets. However, INR-V can be trained on a single action class of UCF-101 (such as Jumppopee to learn a meaningful representation space even with significant camera motion and in-video subject movement. A single action class limits the visual and motion diversity in the dataset.
Broader Impact. The polential negative impact of our work is similar to existing image-based and video- based GANs: creating ppootorellssticdeepaaaes and us- ing them for malicious purposes. Our simple training strategy makes easier to train a model which pro- duces realistic-looking videos. However, this is partly addressed for the following reasons: (1) Even though our network produces diverse novel videos, the percep- tual quality of our generated videos falls short of the ex- isting state-of-the-art image-based generators that pro- duce high-resolution images. (2) The availability of high- quality video datasets limits the intended malicious use of this codebase. Despite these limitations, we believe that the potential of our work far outweighs its limitations. A continuous vidco representation space offers tremendous applications in areas requiring video prediction, interpolation, and conditional video generation. E.g. pedestrian trajectory prediction is an important area of research for self-driving cars. Pedestrian trajec- tory prediction through future frame generation can serve to reduce accidents in fully-antonomous vehicles in the future. Similarly, conditional video generation can be used for synthesizing novel sign language videos that can be integrated into schools and universities to encourage and enable hard-of-hearing students.
0
0
NR-V
0
8
8
8
6
0
Denoxing
0
8
8
8
Denoising VQVAE2 reconstructions to enhance the visual quality of relatively blurry videos generated by INR-V. Please refer Sec. 4.2
Figure 18:
A.7 Additional Qualitative Results
We encourage our readers to view the supplementary video results of INR-V. Fig. 19 presents the real video instances in the training set. Fig. 20 and Fig. 21 presents qualitative results on the reconstruction of video instances from different training datasets. Fig 22 presents random videos generated by INR-V on different datasets. Fig 23 and Fig. 24 present. spatio-temporal view of video interpolations on How2Sign-Faces and
21

---

Published in Transactions on Machine Learning Research (10/2022)
H2S
MNIST
2
0 2
A
8
9
9
9
7
A
7
7
Jelly
Sky
Figure 19: Examples of real videos instances of How2Sign- Faces (H2S) Duarte et al. 2020 Moving-MNNN (MNIST) Srivastava et al. (0155), Rainbow.Jelly (Jelly), and SkyTimelapse (Sky) Xiong (2017) datasets.
SkyTimelapse respectively. Fig. 25 presents the random generation of INR-V on multiple resolutions starting from 32 x 32 to 256 x 256 jumping a scale factor of 8x. The visualization is up to scale, and one can see the scale jump. INR-V V can also be inferred at multiple frame rates. The supplementary videos include inferences Fig. 32 present the qualitative results and comparisons on the proposed inversion tasks. Fig. 30 presents an example of multi-modal future segment prediction. Additional results on several inversion tasks can also be found in the supplementary video.
at frames. Fig. 26
22

---

Published in Transactions on Machine Learning Research (10/2022)
How2Sign Faces
Ressnnl Ground Trulh
Ground Trush
recomslnue iton
Ground Trulh
Recors wns fio
RainbowJelly
Ground Fuil
Recorsituutioo
Reczont Cround Trush
Resoreliaaliin Ground Fuuh
Figure 20: Examples of video instances in the training set reconstructed by INR-V.
23

---

Published in Transactions on Machine Learning Research (10/2022)
Moving-MNNN
Recoure ltu lion Ground
33
3
33
3
3
3
3
3
3
3
3
3
-3
3
3
3
3
3
3
3
3
3
Ressoneinuction Gounnd frulh
47
4
4
7
7
7
7
7
7
4
4
4
4
4
7
7
7
7
4
7
4
7
4
4
4
7
-
7
7
4
7
4
Recoonalrusion Groand Tuih
6
/
/
/
/
6
d
0
0
0
0
0
O
6
1
1
1
0
0
0
0
0
0
0
SkyTimelapse
Reeconsinuclon Ground
Reconsinuctiinn Ground Trulss Reconelraatiio Ground
Figure 21: Examples of video instances in the training set reconstructed by INR-V.
24

---

Published in Transactions on Machine Learning Research (10/2022)
How2Sign-Focee Faces
SkyTimelapse
RainbowJelly
1
1
1
1
3
5
5,
57
5
5
3
Moving Moving-MNISI -NNIST
37
37
3
37
37
37
37
37
37
06
0
96
06
26
06
06
06
26
06
Figure 22: Examples of random videos generated by INR-V. V.
25

---

Published in Transactions on Machine Learning Research (10/2022)
time
space
Figure 23: Examples of video interpolation in INR-V on How2Sign-Faces. Two latent points are sampled from the training dataset. Intermediate videos are then generated by sampling intermediate latent points using Slerp interpolation technique. We urge the readers to view the supplementary video for best experience.
26

---

Published in Transactions on Machine Learning Research (10/2022)
time
space
Figure 24: Examples of video interpolation in INR-V on SkyTimclapse. Two latent points are sampled from the training dataset. Intermediate videos are then generated by sampling intermediate latent points using Slerp interpolation technique. We urge the readers view the supplementary video for best experience.
27

---

Published in Transactions on Machine Learning Research (10/2022)
32
64
128
256
32
64
128
256
Figure 25: Examples of random videos generated by INR-V at multiple resolutions of 32 x 32, 64 x 64, 128 x 128, and 256 x 256 on How2Sign- Faces (top) and RainbowJelly (bottom). The videos are 25 frames long cach. The videos are upto scale. INR-V was trained on videos of only 100 x 100 resolution. Please refer the supplementary videos for better experience and additional results on 50 frames long video generation.
28

---

Published in Transactions on Machine Learning Research (10/2022)
GT
DIGAN
INR-V StyleGAN-V
Figure 26: Comparison of video inversion. Red boxes highlight the differences and matches between the
ground truth (GT) and the various methods. To note, INR-V is able to preserve the finer mouth movements.
29

---

Published in Transactions on Machine Learning Research (10/2022)
Context
GT
INR-V DIGAN
Figure 27: Comparison of half-context inversion in inpainting setting. At the time of optimization, the model only sees the top half of the video. It then generates the full video back. There can be multiple correct predictions, we showcasc one such prediction.
30

---

Published in Transactions on Machine Learning Research (10/2022)
Context
GT
DIGAN
INR-V
Figure 28: Comparison of hall-context inversion in sparse context setting. At the time of optimization, the model only sees 25% of the full video. INR-V preserve the identity including finer content details like carrings. It also preserves motion like pose and mouth movements.
31

---

Published in Transactions on Machine Learning Research (10/2022)
Context
Missing Portion
GT
DIGAN
INR-V
Figure 29: Comparison of half-context inversion in a future frame prediction setting. At the time of opti- mization, the model only sees the first 4 frames of the video. There can be multiple correct predictions given the identity is preserved across the video. We show one such example.
32

---

Published in Transactions on Machine Learning Research (10/2022)
Context
Missing Portion
GT
Original setting
1 frame
Inversion
2 frames
Inversion
Original Setting + Noise
Figure 30: Multimodal Future Frame Prediction: Given the first few frames of a video, we want to predict multiple different outcomes. This can be achieved using INR-V by conditionally optimizing the video's latent vector on half-context. In this example, we condition the latent by varying the number of frames used for inversion or by adding Gaussian noise to the optimized latent. As can be seen, the inversion preserves the seen context; and the future predictions have varying mouth movements, pose, and eye gazes.
33

---

Published in Transactions on Machine Learning Research (10/2022)
Context
Missing Portion
GT
DIGAN
INR- V
Figure 31: Comparison of half-context inversion in frame interpolation setting. At the time of optimization, the model only sees the first and last frames of the video. As can be scen, the first and the last frame generated by INR-V match the context (pose, identily, mouth movements), whereas the intermediate frames are very different from the ground truth.
34

---

Published in Transactions on Machine Learning Research (10/2022)
GT
Bicubic
INR-V VideoINR
Comparison of video superresolution. A video of 32 32 is given as input to INR-V for opti- mization. Once the video is optimized, INR-V regenerates the video at higher resolution of 128 x 128. VidcoINR and Bicubic directly see the 32 x 32 video and superresolves it to 128 x 128. Here, INR-V is not influenced by the glaze on the spectacles and superresolves to a higher dimension closer to the ground truth.
Figure 32:
35


Reviews:
[{"review_id": "Bspbck17nc", "reviewer": "Reviewer_G6uQ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Positives\n\nI am very sympathetic to all objectives and desiderata put forward by the paper. Learning a representation space that is tailored to spatiotemporal data and video is certainly missing in the literature. A simplistic view of merely extending existing image-based representation learning methods is not sufficient, as also demonstrated in the submission.\n\nI believe that this work follows the late trend of using the so called ‘foundational models’ (such as CLIP) for secondary tasks. The results show that indeed CLIP can play a critical role to learning of good video representations.\n\nThe visualizations and results look quite nice at times, and this shows that the method is onto something.\n\nNegatives\n\n1. Despite being sympathetic to the work and its objectives, unfortunately it is not clear to me what is the message that the submission is striving for.\n\nIs the message ‘this is a nice new representation space for videos’? If yes, I would expect deeper analysis with respect to downstream tasks that are supposed to be benefit from a representation space (action classification, event classification, …), together with appropriate comparisons against relevant supervised/unsupervised/self-supervised baselines. What is more, I am doubting whether one can make such a claim when the method cannot work without using literally and explicitly and external method (CLIP) with its trained weights etc. If anything, CLIP is image-based, so at least the claims regarding video specific representations should be toned down.\n\nIs the message ‘Using CLIP (or similar foundational models) is critical for future works on the video representation space’? I assume not, since this is not emphasized in the contributions. However, without CLIP I doubt whether the suggested method would still work well enough, as also demonstrated in ablation A.1.\n\n2. The paper does not have clear contributions from a technical point of view, and the overall method is mostly a combination of existing and well known techniques which are expected to work well together, one way or the other. The method can basically described as\n    1. We represent videos as functions whose parameters are learned to be predicted by hypernetworks.\n    2. To avoid collapse of the latent space to the observed points, the ‘latent’ codes are a combination of CLIP codes and autoencoding codes. To avoid collapse, also, bootstrapping must be used.\n    3. Because the generations do not look always that crisp, we also use an additional denoising step borrowed from VQVAE.\n3. Going deeper into the technical aspects, I am not even sure whether the method is correct. At several locations in section 3.1, it is mentioned that a video is represented as a theta-parameterized function, sampled from tau. Also, it is mentioned that the representations are constrained on tau.\n    1. To sample from any distribution, one must have a probability density function either explicitly (like with VAEs) or with implicitly (like GANs). Like that, one can use basic samplers (like for gaussians in VAE), or advanced samplers that rely on basic samplers (like the decoder of a GAN or of a Normalizing Flow). How are samples sampled in this paper, especially given that CLIP is not generative as far as I know? Are samples obtained ad hoc via a uniform sampling? If yes, why does this work and how can one be sure that indeed the points in learned latent space are indeed distributed uniformly, that there are no ‘empty spaces’, etc? This is even more worrisome given that the method claims to be able to generate novel samples without explaining how this is possible in the training (only an L2 loss is described, which alone simply guarantees optimization/fitting).\n    2. It is quite unclear to me what is ‘constraining a space’? Is the space shaped according to a particular topology? Or is it meant that only part of the CLIP space is really considered, so placing an ad hoc constraint. The term regularization or constraint has usually a specific (and different) meaning.\n    3. The progressive training of section 3.2 seems ad hoc and thus sensitive to the way it is defined or implemented. For instance, how are the 10 examples selected that kick start the training, and how sensitive is the method to them? And what about the additional examples included at each round? Furthermore, I suppose the mini-batching takes places within the set V? Otherwise how does the mini batches fit to memory?\n4. While the results are not bad, they are not that convincing either. In terms of metrics, the method appears to be doing well natively on Rainbow Jelly, and about the same on Moving MNIST (table 1). With the extra denoising step the model does better than baselines, however, it is not clear if this denoising step would also benefit the baselines (in that sense, perhaps it makes sense to more tightly integrate the denoising into the model).\n5. More generally, it is no clear if all comparisons are all that fair, given that the proposed method relies on an external component (CLIP) that could easily be plugged to the other baselines. For instance, if CLIP was combined with the GAN baselines, would the sudden irregularities in figure 6 still be there?\n6. There are several passages that are quite unclear because terminology is assumed to be known, which however will not necessarily be the case.\n    1. In 5.1 please explain what is Slerp interpolation.\n    2. There is significant notation clutter in 3.1 with new notations used only once or twice.\n    3. In the beginning of 5.2 it is really unclear to me what the method is supposed to do, and how can it work with different lengths and scales.\n    4. Perhaps it makes sense to explain briefly what it is meant by ‘video inversion’ in the beginning of 5.3. It becomes later on clear in the text, however, not all readers might be familiar with the term. \n    5. Table 4 is inconsistent compared to previous tables. There is no bold face, nor arrows indicating when the number is better.\n    6. I am not sure if one can use the word ‘superresolve’, although I guess it is technically correct.\n    7. Also, the section in ‘Quantitative evaluation’ in 5.3 is very unstructured and it is hard to understand what it means/follow.\n7. A last point, which however I do not consider that important, is that continuous representations (in the sense of a representation from the entirety of a video) were previously considered as well, in the context of action recognition. An example is Fernando et al., Modeling Video Evolution for Action Recognition, CVPR 2015, or Action Recognition with Dynamic Images, which learn to represent a video as the parameters of a linear SVR or an approximation thereof (in a deep network setting).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submitted work presents a method for learning a continuous representation space specifically for videos. The space is continuous in that the latent codes represent whole videos and not frame-by-frame instances. Being a continuous representation space, the model can more easily generate videos that are coherent in both motion and content without sudden and unrealistic changes. For this, crucial part plays the use of pretrained CLIP encoders, without which the method appears to suffer. Applications of this method include video generation, inpainting, future prediction, and so on (or what the method succinctly calls ‘video inversion’). Other contributions claimed by the paper are having a neural representation for video that is in the order of thousands instead of millions, as well as a suggested training methodology to stabilize the training of the notoriously unstable hyper networks.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See list of weaknesses.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "kGBiUyIiei", "reviewer": "Reviewer_vVxg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths:*\nINV-R is interesting, and authors demonstrate  the model flexibility in the empirical evaluation. INR-V can be adapted to solve various tasks, from video generation to future frame prediction and super-resolutions\n\n*Weaknesses:*\nWhile the paper proposes an interesting and novel model for video generation, there are some limitations that need to be fixed before acceptance\n- Clarity:  \nThe paper clarity could be improved. In particular\n1) What is the code ‘c_n’ and the auto-encoding loss in section 3.2? Why does it help prevent collapse of m_z\n2) Why does the representation m_z collapse as this would hurt the reconstruction performance?\n3) Are  the hyper-network and INR trained jointly or in different steps?\n4) How do you generate new videos in section 4.2 as you don’t train an explicit prior model on the latent space?\n\n- Computational costs of INR-V:\nINR-V claims that it can generate large video, while 3D convolution operations would have drastic computational cost. However, implicit functions also have a high computational as you need to call the function for each voxel in your video volume. It would be informative to compare the computational complexity of the two approaches.\n\n- Motivation for the use of CLIP:\nHow would INR-V perform without using a pre-trained CLIP network. The use of the CLIP seems orthogonal to the main claims. It also provide an advantage to INR-V as CLIP is trained on more data and require more computation.\n\n- INR-V without upsampling: \nWhat is the advantage of INV-R over other video generation approaches as it perform similarly without the upsampling stage.\n\n- Lack of complexity in the evaluation datasets:\nAll the datasets considered for evaluation have limited visual complexity and almost no camera motion. How would INV-R perform on more complex dataset such as UCF-101 and Kinetics?\n\n\nAdditional question:\n- Video prediction is a multimodal prediction tasks as several outcomes are often plausible for a given input.  How is INV-R able to generate different futures for the same set of input frames?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes INR-V, a generative model of videos which is based on implicit neural representation. INR-V parametrizes a neural implicit function using a hyper-network. The hyper-network takes a video latent representation and outputs the parameters value of the implicit function. The implicit function is then used to generate the video RGB values.\n\nThe authors propose two tricks to stabilize the training of the hyper-network, latent regularization and progressive training (i.e. gradually increasing the size of the training set).\n\nAuthors demonstrate the ability of INR-V on various task including, video reconstruction, video generation, interpolation, and various input completion tasks \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Improve the clarity of the paper (see previous section).\n- Add video generation experiments on UCF-101 and Kinetics.\n- Compare computation complexity of INV and convolution networks for video generation.\n- Shows that INV-R is working without a CLIP network or better motivate the use of CLIP in the approach.", "broader_impact_concerns": "No specific concerns given the broader impact discussion in the paper", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rWn3NDnWw9", "reviewer": "Reviewer_Ec91", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The video representation as a NN mapping from location to RGB value is interesting. However, the current approach is only designed for video generation. It isn't immediately clear how or if this would be used for other tasks, like video classification. It would be interesting, and further strengthen the paper, to have a way to use this for those tasks. Otherwise the title seems a bit too broad.\n\n\nFor the key elements:\n(1) MLPs lead to fewer parameters than convolutions. I'm not sure this is true? A similar architecture with conv layers could be used with the same number of params? It isn't the choice of MLP vs. convs that leads to this reduction, but rather the configuration (channels, etc) of the MLP. \n\n\nThe experiments compare to other approaches, but this approach relies on a pretrained Video-CLIP model, which uses far more data than the previous works. This makes it a bit unclear how much benefit is coming from that additional data vs. the ideas proposed in this paper.\n\nRelated, there aren't any studies on the components of the proposed approach. E.g., effects of video clip, size of codebook, regularization choices, etc. It would strengthen the paper to have more thorough ablation experiments of the components.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main idea of this paper is to represent a video with a NN that can map from a (h,w,t) point to a RGB pixel value for that point. This NN has very few weights, which are dynamically predicted/created by another NN for each video. This is a different video representation from existing methods, and is interesting to study. The main difference from previous INR style works is that the weights are predicted by another network.\n\nThe approach is evaluated on video generation tasks, on the How2Sign Faces, SkyTimelapses, Moving-MNIST and RainbowJelly datasets.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Adding the ablation experiments would help the most.\n\nMinor comments:\nThe abstract is a bit confusing. After reading it, I was unsure of what the paper was proposing. Specifically,\n\"Images are considered a complete signal, whereas videos are usually broken down into a\nset of temporally coherent images. Consequently, an image space is leveraged for various\nvideo-based tasks, \"\n\nIs a bit confusing. It'd help to strengthen the idea of the paper to better explain this idea, e.g., why treating a video as a set of frames is bad and what the alternative is.\n\nThe INR and meta-network isn't really introduced, and there's no mention of the evaluation tasks (e.g., classification, detection, generation, etc). It'd be good to revise the abstract to give a more clear high-level overview and a mention what tasks this is used for.\n\nThe YouTube link in footnote 1 is broken.", "broader_impact_concerns": "The limitations and broader impacts seem sufficient.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers are happy with the improvements made by the authors during the reviewing period and are at least leaning towards acceptance. One of  the reviewers thinks it is still unclear in the paper "if this type of approach could be applied to more complex video datasets", so I would suggest addressing that comment in a minor revision. Otherwise I believe the paper is in a good state and a welcome addition to TMLR.


Decision: Accept (poster)