NeRV-Diffusion: Diffuse Implicit Neural Representation for Video Synthesis
Authors: Yixuan Ren
Hanyu Wang
Bo He
Hao Chen
Abhinav Shrivastava
Venue: ICLR 2026 Poster (2026)
Keywords: Video diffusion model
Implicit neural representation
Video tokenization
Primary area: generative models
TL;DR: NeRV-Diffusion synthesizes new videos via generating implicit neural representation weights from Gaussian noise.

Abstract: We present NeRV-Diffusion, an implicit latent video diffusion model that synthesizes videos via generating neural network weights.
The generated weights can be arranged as the parameters of a convolutional network, forming an implicit neural representation (INR) and decoding into videos with frame indices as the input. Our framework consists of two stages: First, a hypernetwork-based tokenizer that encodes raw videos from pixel space to neural parameter space, and the bottleneck latent serves as INR weights to decode; Second, an implicit diffusion transformer that denoises on the latent INR weights. In contrast to traditional video tokenizers that compress videos into frame-wise feature maps, NeRV-Diffusion generates a video as a compact dedicated neural network. This continuous holistic video representation obviates temporal cross-frame attentions while preserving flexible temporal interpolability. The INR decoder and weight latent feature sublinear complexity overhead regarding video resolution and length increase with additional upsampling layers. To enable Gaussian-distributed neural weights with high expressiveness, we reuse the bottleneck latent across all INR layers, as well as reform its weight modulation, upsampling connection and input coordinates. We also introduce SNR-adaptive loss weighting and scheduled sampling for effective training of the implicit diffusion model. NeRV-Diffusion reaches superior video synthesis quality over previous INR-based models and comparable performance to most recent state-of-the-art non-implicit models on real-world video benchmarks including UCF-101 and Kinetics-600. It also achieves outstanding decoding and generation efficiency when scaling up to high-resolution and long videos.

Published as a conference paper at ICLR 2026
NERV-DIFFUSION: DIFFUSE IMPLICIT NEURAL REPRESENTATIONS FOR VIDEO SYNTHESIS
Yixuan Ren, Hanyu Wang, Bo He, Hao Chen," Abhinav Shrivastava' University of Maryland, College Park {yxren, hywang66 bohe @uddd edu, fchenh, abhinav)ecs. umd. edu Project page: httts::/ner--d wtw.
ABSTRACT
We present NeRV-Diffusion, an implicit latent video diffusion model that synthe- sizes videos via generating neural network weights. The generated weights can be arranged as the parameters of convolutional network, forming an implicit neural representation (INR) and decoding into videos with frame indices as the input. Our framework consists of two stages: First, a hypernetwork-basse tokenizer that encodes raw videos from pixel space to neural parameter space, and the bottleneck latent serves as INR weights to decode: Second, an implicit diffusion transformer that denoises the latent INR weights. In contrast to traditional video tokeniz- ers that compress videos into frame-wise feature maps, NeRV-Diffusion generates video compact dedicated neural network. This holistic video representa- tion obviates temporal cross-frame attentions while preserving flexible temporal interpolability. The INR decoder and weight latent feature sublinear complexity overhead regarding video resolution and length increase with additional upsam- pling layers. To enable gruussian-distibuue ness, we reuse the bottleneck latent across all INR layers, as well as reform its weight modulation, upsampling connection and input coordinates. We also intro- duce SNR SNR-adaptive loss weighting and scheduled sampling for effective training of the implicit diffusion model. NeRV-Diffusion reaches superior video synthe- sis quality over previous INR-based models and comparable performance most recent state-of-the-arr non-implicit models on real-world video benchmarks in- cluding UCF-101 and Kinetics-600 It also achieves outstanding decoding and generation efficiency when scaling up to high-resolution and long videos.
neural weights with high expressive-
INTRODUCTION
Video latent diffusion models (LDMs) have achieved impressive generative capability. However, their tokenizers usually inherit from image models and encode videos as individual frame-wise fea- ture maps, ignoring the natural coherence across frames and resulting in redundant representations. Cross-frame attentions (Wang et al. 2023 Guo et al. 2023) are thus introduced to constrain tem- poral consistency, bloating the model size and leading to massive computation footprint. Moreover, traditional tokenizers have fixed dowmsampling factors, and the latent size will increase quadrati- cally when data resolution doubles. ID tokenizations Yu et al. 20024b;; Wang et al. (0025a) have been explored for holistic latent, while discrete tokens compromise the spatiotemporal granularity.
Implicit neural representations (INRs) neural networks that fit single data points. An INR takes unified coordinates as the input and outputs pixels as stored in its model weights. It has shown significant advantages on compression (Sitzmann et al. 2020 Dupont al. 202T). fast decoding (Chen et al., 2021a), and easy transformation (Mildenhall et al. 2021 Kerbl et al.) (023) by repre- senting data as an integral format of function. The continuity and differentiability
INRs facilitate advanced single-data generative tasks, such as super-resolution. restoration, style transfer and edit- ing, via smooth interpolations within the data space. Its compact representation also contributes to reducing memory overhead, making them highly suitable in eesource-constrainng environments.
"Co-corresponding authors.

---

Published as a conference paper at ICLR 2026
Tokenization
Frame Indices
500
Awesscterier
DIGAN
NeRV Encoder
Neural Weights
NeRV
GT Video
250
NcRV-Difsiim-S Onmiiakeiia
MAGVIT-AR
diffusing
150
XAGVIEV2-AR
Implicit DIT
demoising
Recon. / Gen.
50
NcRV-EMftaeee
LARP
20
60
30
100
NeRv Decoder]
Figure 1: Left: Overview of our NeRV-Diffusion framework. In the tokenization stage, NeRV Encoder projects RGB videos to neural weight latent, which serves as kernels of a NeRV and decodes for reconstruction. In the generation stage, implicit diffusion transformer is trained to denoise on NeRV weights. During inference, the implicit DiT generates NeRV weights from random noise, and the NeRV decoder renders them into videos. Right: NeRV-Diffusion outperforms previous INR-based as well as most recent non-implicit video generation models at all scales with more compact model sizes. The generative performance evaluated in gFVD on UCF.
To harness the strengths of both latent generative models and implicit neural representations, establish an implicit latent diffusion model, NeRV-Diffusion, for video synthesis by generating INR weights. where video is represented In the tokenization stage, a hypernetwork based encoder compresses RGB videos into parametric latent tokens. The tokens instantiate an INR to decode for reconstruction with unified frame indices input. In the generation stage, diffusion transformer denoises in the encoded implicit latent space, mapping random noise to INR weight tokens. Figure (left) overviews the framework.
holistic set of INR weights. It consists of two stages:
However, it is not trivial to acquire Gaussian- neural network weights for smooth dif- fusion that are meanwhile able to represent diverse realistic data with high fidelity. We adopt convolutional video INR, NeRV (Chen et al. (2021a), and build transformer INR encoder based on FastNeRV (Chen al., 2024a). They are originally designed toward video compression perfor- mance only and their produced INR weights are not generatable. To ensure the bottleneck latent tokens fitful for both faithful reconstruction and smooth diffusive generation, we have made several critical architectural modifications. The detailed architectures are illustrated in Figure
Specifically, we reuse the encoded weight tokens with multiple linear affine layers such that each NeRV layer is fulfilled by all tokens independently. We also redesign the weight parameteriza- tion approach, proposing to directly set the latent tokens to be the convolution kernels, instead of modulating the shared base weights with scale and shift. These upgrades fundamentally enlarge the expressiveness and smoothness of the implicit space while maintaining its compactness. We leverage vanilla diffusion transformer (Peebles & Xie 2023) (DiT) to denoise weight tokens that imply no spatial or temporal structures. We also handle the error accumulation with aadatiiaeive loss weighting and scheduled sampling for optimal denoising in the implicit latent space.
NeRV-Diffusion leverages video INRs as instance-speciff decoders, offering faithful reconstruction, compact model and fast decoding compared the large. shared decoders in traditional LDMs. encodes and generates video frames holistically as integral INR weights, implies the keyframe- residue representation by reusing the same set of parameters decode all frames. It thus maintains temporal association and interpolation without cross-fram attention. Furthermore, its weight latent shape proportional the INR decoder size, and sublinear the data resolution upscaling. It only needs to append an additional upsampling block to the decoder to double the output resolution.
In summary, our contributions are as follows:
We propose a novel implicit video autoencoder that compresses videos into neural weight tokens of normal distribution, constituting generation-specializedd video INRs.
2

---

Published as a conference paper at ICLR 2026
We propose implicit diffusion model that denoises in neural weight space, achieving dynamic and diverse video synthesis via generating INR parameters.
NeRv-Diffusion surpasses prior implicit and non-implicit generative models on multiple real-world video benchmarks, and conveys efficient scaling up and smooth interpolation.
2
RELATED WORK
2.1 IMPLICIT NEURAL REPRESENTATIONS
Implicit neural representations (INRs) neural networks that fit single data points. An INR takes in coordinates and outputs corresponding pixel values of the stored data. has presented
capacity and flexibility in various modalities, including images (Sitzmann al. 2020 Dupont al.,
202T]. 3D shapes (Park et al.] 2019] Mildenhall et al., (202)) and videos (Chen et al. 2021a 2022
They are primarily developed for image compression (Strümpler et al. [022 Dupont et al.) 2022) and editing (Fan et al.)
2022; Yang et al. 2023). video compression (Li ct al. 2022 Kwan et al. 2024 Zhao al. 2023 Zhang al.] 2021 Lee et al. 2023) and editing (Ouyang et al. 2024), and novel view rendering (Kerbl et al. 2023] Barron et al. [023] Cao & Johnson] 2023) and 3D scene
editing (Yuan et al. [022] Liu et al. 2024). Although some editing applications have been explored,
they create the INRs after manipulating the data in pixel space.
standard INR is trained via memorizing the pixel data, which time-consuming in a backprop- agation manner. [henn & Wang (2022); Kim et al. (2023a); Chen et al. (2024a) suggest using transformer-basee hypernetworks to create INR weights given RGB data in a feed-forward fash- ion at scale. However, these methods are optimized solely toward reconstruction performance and incorporate no distribution regularization on the produced INR weights, leaving the implicit gener- ative task that synthesizes novel data points from random noise under-addressed.
2.2 IMPLICIT NEURAL REPRESENTATION GENERATION
INR generation is a challenging task. Traditional generative models learn mapping random noise to pixels latent features, while implicit generative models aim associate neural parameters with
Gaussian distribution. Several efforts have been made toward implicit generation. [Skorokhodoy
al. (022] builds a GAN for image INRs (Sitzmann et 2020), and Yu al. (2022) extends it to
videos by involving the temporal axis.
Erkoç et al. (2023; Chen al. (023); Müller
(2023) ; Shue al. 2023) study generating 3D NeRF parameters via diffusion models. (Chen et al. 2024b) applies latent diffusion models on image INRs (Chen et al., 2021b) for image synthesis, while their INR weights are derived by a complex decoder from the denoising latent space. Recently. Wang et al. (2024b) 2025b) propose to leverage the hypernetwork-INg
architecture to conduct flow matching on image or 3D pixel data. Lee et al. (2025) also developed masked image autoencoder for inpainting with similar structure. Despite these efforts, no video diffusion model that generates INR weights has yet been explored, casting this challenging task vidcos embed more dynamic information and diffusion models have a more strict demand on its denoising space.
2.3 LATENT VIDEO DIFFUSION MODELS
Latent video diffusion models (Wang al. 2023 Blattmann al. 2023b Guo et al., 2023) have achieved significant success in video generative modeling. However, traditional video tokenizers of- ten encode video frames as individual feature maps, calling cross-frame attentions in the denoising network to constrain temporal consistency. Kim et al. (2023)); ( et al. (2023) start to explore video autoencoders with motion awareness and temporal compression, splitting the complexity between the tokenization and generation stages. Recent 1D tokenization (Yu et al. [024b) Wang et al. 2025a Zha et al., (0255) encodes visual data into holistic tokens that project no spatial or temporal align- ment with pixels, while they remain focused on images or auto-regressive generation only. In this work, we look to synthesize videos by generating INR weights via diffusion, obviating frame-wiss wise representations by using the whole INR model to decode all frames given time indices. Moreover, symmetric autoencoders rely on a large-scale decoder to render synthesized latent to diverse RGB data with high fidelity, consuming non-negligible computational resources and time for end users
3

---

Published as a conference paper at ICLR 2026
Video Patches
Weight Queries
Sehhede Caraliee somplin ling
add noise
NeRV Encoder
Implicit DiT
Neural Weight Latent
Multi-Head Affine
A
A
Conv
Conv
Conv
Video
Frames
Grid
NeRV Decoder
Figure Detailed architectures of NeRV-Diffusin. Top left: Patchified videos and learnable weight queries concatenated and input into NeRV encoder, outputting latent weight tokens; Middle left: Weight tokens are reused and converted by multi-heal affines to instantiate each NeRV decoder layer; Bottom: NeRV decoder renders spatiotemporal positional embeddings into RGB videos, using the instance-specific kernels and global shared kernels. Block details and side connec- tions omitted: Top right: Weight tokens added noise and an implicit diffusion transformer is trained to denoise in this implicit weight space.
to visualize. We explore the space of asymmetric hypernetwork-IN IINR autoencoders, where the INR acts an efficient instace-sspecife decoder only needs to represent single data point.
3 NERV DIFFUSION
is a two-stage generative framework. In the tokenization stage, implicit autoen- coder ($3.T) is trained to compress a video from pixels to latent neural weight tokens, and the tokens function the parameters of INR ($3.2) and self-decode to reconstruct the video. In the gen- eration stage, an implicit diffusion transformer ($3.3) is trained to generate the weight tokens from random noise. Figures (left) and illusttrtee our full pipeline of both stages.
3.1 NERV-VAL
In the first stage, we aim to tokenize video into a latent space that represents the video through the parameters of INR. This is achieved by training an implicit autoencoder, where the encoder a hypernetwork that produces INR parameters = E(x) given pixel input x. The decoder is implemented an INR (-).) which decodes to pixel values given corresponding coordinates. We build the backbone of our INR encoder & upon ViT-based FastNeRV (Chen et al. 2024a). where we make several critical modifications align the learned latent space with generative tasks.
The RGB video is first segmented into patches and converted to transformer input embeddings. Since the output weight tokens have no spatiotemporal correspondence to the input patches, instead of mapping them directly we introduce dedicated query tokens and concatenate them with the data patches following (Peebles et al. 2022). Only the output tokens corresponding to the queries are retained. They are batch normalized along the token embedding dimension.
KL Bottleneck. Two additional fully connected (FC) layers appended after the NeRV coder's output to create an information bottleneck of compact latent dimension. KL divergence loss applied to align their distribution toward standard Gaussian distribution (/0 1).

---

Published as a conference paper at ICLR 2026
Multi-head Affine Mapping. FastNeRV use its encoded latent to modulate the parameters of subset of the INR layers, which limits the capacity of the latent tokens especially when KL con- straint is applied for generation tasks. Inspired by the multiple affine layers in Karras et al. 2019), we expand the post-bottleneck FC layer into multi-head affine mappings, and the single set of weight tokens reused to modulate all NeRV layers independently. Specifically, for each NeRV layer, dedicated affine head maps all the weight tokens into modulation parameters. This strategy signif- icantly expands the expressiveness of the weight tokens, as compact latent space will reduce the complexity of the diffusion process in the generation stage.
Channel-wise INR Parameterization. FastNeRV repeats the weight tokens and multiply them to the instance-agnostic INR base weights via dot product as the modulation. Skorokhodov et al. (2021]: Yu et al. (2022) perform low-rank vector cross product to amplify the modulation matrix dimension from condensed weight latent. Inspired by Lin et al. (202T) that prunes pretrained GAN generator by subsetting its kernels, we propose to directly set affined instance-specific weight tokens to be the convolutional kernels shared among all training data and are learnable during training. All kernel values are normalized along all dimensions except the output channels, following the demodulation in Karras et al. (2019). In this way, the generated weight tokens are directly involved in decoding with maximal degrees of freedom. This also enables smooth parameter interpolation between our INR decoders.
certain group of INR channels. Other parameters 0, are
Convolutional Discriminator. To generate realistic videos we incorporate adversarial training [Goodfellow et al.] 2020). We choose convolutional discriminator (Karras et al.] 2019) over transformer-based one, as we observe that the latter introduces flickering artifacts across frames.
Training Objectives. We train NeRV-VAE with the reconstruction objective. With an additional perceptual loss (Zhang et al. 2018) LLPISS and the adversarial loss LGAN, it is optimized via
LVAE(E, 0.) | a|l + LLPIPs @) + LGAN(E, Ø) + DKL (W 1),
(1)
3.2 GENERATIVE NERV DECODER
The encoded weight tokens are formed into video INR Do(-) that decodes to reconstruct the video. NeRV (Chen et al. 2021a) is a convolutional video INR that takes time index as the input query and yields a whole frame at each forwarding. We construct our implicit decoder based on while introducing several upgrades enhance its capacity for generative purposes.
Spatiotemporal Embedding Input. Time-query video INRs upsamples from RTxDXIXI to R xxxxx#xxX where spatial dimension is input. With this structure, we observe distinct move- ment in the reconstruction, however the spatial content lacks clarity. To balance between the ap- pearance and motion quality, we expand the input time embedding 3D spatiotemporal, while time remains the sole query axis. Specifically, we sample a 3D positional embedding and reshape it to RTx33 xaDxxxx This spatiotemporal input supplements geometric prior and avoids the leading FC layer in vanilla NeRV that were designed for transforming ID time embedding input. We observe that full convolutions fit optimally for generative quality with our multi-head affine modulation.
Scaling Blocks. Benefited from the reused weight modulation with multi-head affine map- pings, we are able largely scale up our NeRV decoder without extra weight tokens. We expand the upsampling layers to blocks, each performing one-level (2x) upsampling with additional con- volutions that don't it change the shape. Compared the assorted upsampling scales in limited layers in vanilla NeRV, this periodic upsampling structure evenly distributes the information from low to high resolutions, and cooperates well with our multi-head affine modulation. We also double the hidden dimensions of the layers in the last block following Karras et al. (2020) so that more native high-resolution information can be processed with sufficient capacity.
Upsampling Algorithm. While vanilla NeRV has tested that pixelshuffle results in the best recon- struction performance with similar amount of parameters, we again compare different upsampling algorithms for NeRV decoder. We find that transposed convolutions achieve negligiibe bet- ter generation quality to pixelshuffle with merely a quarter of parameters and computations. There- fore choose transposed convolutions for all the upsampling layers in NeRV decoder.
5

---

Published as a conference paper at ICLR 2026
GT
Reconstructed
Figure 3: Video reconstruction of our NeRV-VAE on UCF (left) and K600 (right).
Table 1: Generative performance comparison UCF and K600 different resolutions and lengths.
(a) UCF 16 frames at 128 resolution.
(c) UCF 16 frames at 256 resolution.
Method
gFVD
Method
gFVDJ
Iop-Implicit Models
VIDM (Mei & Patel 2023
263
Latte (Ma et al.
2024)
202
TATS Ge et al. 2022
332
Ookenzzer (Wang et al.
2024a)
191
MAGVIT-AR Yu al.
2023a
265
AR-Diffusion (Sun al. 2025
186
VideoFusion (Luo et al. 2023)
173
HPDM-M (Skorokhodov et al
2024)
143
MAG Yu ct al. 2024a)
109
LARP-L (Wang al
2025a
102
NeRV-Diffusion-L. (Ours)
140
Implicit Models
(d) UCF 128 frames at 128 resolution.
DIGAN Yu et al.
2022
465
NeRV-Diffusion-S (Ours)
184
Method
gFVD
NeRY-DiffusionBB (Ours)
133
Non-Implicit Models
NeRV-Diffusion-L (Ours)
97
Latte (Ma et al.
2024
1157
(b) K600 16 frames at 128 resolution.
PVDM Yu et 2023b)
505
VIDM Mer & Patel (0233
426
Method
gFVD
CoordTok (Jang et 2025
369
OmniTokenizer (Wang et al. 2024a) (2562)
Implicit Models
LARP-L (Wang et al. 2025a
17
DIGAN Yu et al 2022
1103
NeRV-Diiftusion-L (Ours)
22
NeRV-Diffuston (Ours)
366
Side Connections. With the increased depth of NeRV decoder by upscaled blocks, we further append side connections to effectively collate all intermediate resolution information with minimal computation overhead. We investigate the residual and skip connections in Karras et al.) (2020). If the side connection needs additional layers, they are also modulated by the same set of our weight tokens thanks to our multi-head affine mappings and no extra trainable parameter introduced. Residual connection fuses latent features different scales before decoding to RGB and is experi- mented to yield clearer appearance and stabler motion.
3.3 IMPLICIT DIFFUSION
With visual data tokenized from pixel space NeRV weight space by the implicit autoencoder described above, we perform diffusion process on these weight tokens by 0 = ag0p + are and train denoising network toward
LIDM Ea..-
not trivial to model denoising process on neural weights. Previous diffusion models are de- signed for pixel data their latent feature maps. Since NeRV weight tokens have spatiotemporal structure, transformers are more suitable than U-Nets to process them, and temporal attention is un- necessary in our denoising network like those in traditional video diffusion models (Ma al. 2024

---

Published as a conference paper at ICLR 2026
Figure 4: Class-conditioned video generation on UCF (rows 1-3) and video frame prediction on K600. (rows 4-5). Frames in front of the orange line are input conditions.
G.pt (Peebles et al. 2022) uses transformers to evolve neural network weights in a meta-learning fashion but not noisy data. DiT (Peebles & Xie 2023) tailors transformers for image diffusion and Zha et al. 22255) also use it to process ID image tokens in diffusion. We explored these back- bone options and DiT reaches the optimal performance with straightforward architecture. Besides, we curate the training scheme as below to fill the gap when adapting DiT to the implicit space.
Min-SNR-y Loss Weighting. We observe that our implicit diffusion model converges slower on early denoising timesteps than late ones, i.e. it is tougher to learn to parse more noisy input. To address this issue and speed up its convergence, we adopt Min-SNR-y loss weighting (Hang et al. 2023) and apply the coefficient w, min(SNR(t), 7} on the denoising loss, where SNR(t) of reflects the signal-noise ratio at timestep t, and constant controls the minimum of wt. This loss weighting strategy prevent the diffusion training to focus too much on the low noise levels and descends evenly toward the denoising directions at all timesteps.
Scheduled Sampling. To further enhance the implicit denoising chain and tackle the exposure bias issue, we introduce scheduled sampling (Bengio et al. 2015) into our training scheme. It is initially proposed for auto-regressive models, and has been applied on diffusion models (Ning et al. 2023 Ren et al.) 2024) to fill the training-inferenee gap brought by Teacher Forcing. During training, after the first forward round at step f, we randomly use the model predictions 0,-1 (o,0,, t) the new input and execute another forward pass, and calculate the total losses. It aligns the the training and inference modes, ensures low input disparity and minimizes error accumulation during sampling.
4 EXPERIMENTS
4.1 SETUPS
Datasets. We demonstrate NeRV-Diffusion on two video benchmarks: video generation on UCF. 101 (Soomro al. 2012) (UCF) and frame prediction Kinetics-600 (Kay et al. 2017) (K600).

---

Published as a conference paper at ICLR 2026
Table Decoding (top) and generation (bottom) efficiency comparisons at 128 and 256 resolutions. All results are tested on single NVIDIA A6000 GPU in bfloat16 at batch size and averaged over 100 runs. All generators have enabled CFG and iterate for their default sampling timesteps.
Decoder
#Params
#Tokens
I Latency
VRAM,
128
256
128
256
128
256
128
256
Latte Ma
2024
49M
0.288s
5.2G
CMD
Yu ct al.
2024c
24M
24M
272
1040
0.129s
1.030s
1.2G
4.1G
Open-MAGVITI (Luo et al.
2024
226M
1024
0.127s
45G
SD-VAE (Rombach et 2022
49M
49M
4096
16384
0.048s
0.260s
1.2G
4.3G
SVD-VAE (Blattmann et al. 2023a
98M
98M
1096
16384
0.094s
0.411s
1.3G
1.5G
NeRV-VAE-L (Ours)
55M
64.5M
128
160
0.032s
0.133s
0.99G
2.6G
Generator
#Params
# #Tokens
Steps
Latency
VRAM
128
256
128
256
128
256
Latte Ma et al. 2024
674M
512
250
37s
4.4G
LARP-L Wang al. 2025a
343M
1024
1024
20s
1.6G
OmniTokemizer Wang al. 2024a
650M
1280
5120
139s
4.5G
NeRV-Diffusion-L (Ours)
467M
160
250
6.8s
8.2s
1.8G
2.1G
We conduct experiments 16 128 frames of 128 and 256 resolution. Note that for 128-frame experiments we subsample the raw videos with frame interval of 8 to train both stages, and only produce 128 in diffusion sampling with temporal interpolation of the input frame indices ($4.6.1 We use the train split of K600, and all videos from UCF, following prior work Yuu et al. 2023a).
Implementations. We realize our NeRV encoder with the backbone of Vision Transformer (Doso- vitskiy et al. (02T) (ViT). We scale up our NeRV decoder to three configurations of progressive sizes: -Small (3.5M), -Base (14M) and -Large (55M). We use 128 weight tokens with 128 chan- nels for all configurations. We ablate our key design options in $4.5 Detailed model and training configurations are provided in Appendix
Metrics. We measure Fréchet Video Distance (FVD) (Unterthiner et al. 2018) to evaluate the
generation quality of NeRV-Diffusion. We calculate FVD on 2,048 sampled videos following prior work (Yu al. 2022 2023a Wang al. 2025a) for fair comparison. We adopt additional metrics
for reconstruction and generation in Appendix D.2
4.2 VIDEO RECONSTRUCTION
Visualized reconstruction output of our implicit tokenizer displayed in Figure NeRV-Diffusion achieves comparable performance to other con-implici methods, with much more compact model and latent sizes. It also worth noting that NeRV-Diflusion features small reconstruction-geneaati gap compared to other models, indicating our effective design of implicit video representations for generation purposes, and thus efficient usage of our latent space.
4.3 VIDEO GENERATION
Class- Consiiondit Video Generation. We conduct class- conditonnd video generation on UCF and present our visual results in Figure We quantitatively compare NeRV-Diffusion with other models in Table la NeRV-Diffusion outperforms previous INR-based generative methods well as most recent non-implicit models of various mechanisms, including GAN, diffusion and auto- regressive architectures. It is able to synthesize dynamic videos with diversity in both appearances and motions, ranging from detailed objects to complex scenes. More visualizations and qualitative comparisons displayed in Appendix D.I
Video Frame Prediction. Following Hong et al. 2022). we train our implicit diffusion model
given the initial 5 frames predict the rest 11 frames. We construct sequence of the 5 given
8

---

Published as a conference paper at ICLR 2026
Table Ablation studies the key design options in our NeRV-VAE, tested with NeRV-Diffusion-S and the best implicit denoiser configuration on UCF.
Spatial PE
gFVDJ
Modulation
gFVD
Reuse
gFVDL
h. 1
283
Repeat
741
No reuse
570
w 4
269
FMM
636
Direct reuse
562
h
8
254
Channel
570
Multi-head affines
283
w 16
277
(c)
Token Shape
gFVDJ
Upsampling
gFVD4
Side Connection
gFVDJ
32
128
219
PixelShuffle
254
Vanilla
248
64
128
193
Transposed Conv
248
Residual
219
128
128
184
Bilinear
287
Skips
235
256 x 128
206
(f)
Table 4: Ablation studies of the key design options in implicit diffusion model, tested with NeRV-Diffusion-S and the best NeRV-VAE configuration on UCF.
Configurations
gFVDJ
Model
gFVD4
Vanilla DiT
295
G.pt (Peebles al. 2022)
550
w/ Min-SNR-
238
DiT (Peebles & Xie 2023)
295
w/ Scheduled Sampling
261
Latte (Ma et al. 2024
342
w/ Both
184
(a)
(b)
frames and 11 duplicate 5th frames and encode them video clip to the NeRV weight space. We
double the input channels of the DiT input embedder to fuse the clean condition and noise. The quantitative results are listed in Table Tb and the visualizations are displayed in Figure 4 NeRV- Diffusion faithfully propagates the spatial content and movement flows to future frames.
Scaling Up Video Resolution and Length We scale up NeRV-Diffusion to 256 resolution and 128 frames on UCF. We report the generation performance comparison in Tables IC and ld both cases our NeRV encoder remains the same size. NeRV-Diffusion outperforms recent SOTAs with high compactness and easy extensibility. Notably, NeRV-Diffusion features sublinear complexity overhead when scaling up video resolutions and lengths, which address in the next section.
4.4 DECODING AND GENERATION EFFICIENCY
Conventional VAEs encode videos into frame-wise feature maps with fixed spatial and temporal downsampling factors, and their latent size thus increases quadratically w.r.t. RGB resolutions and length. In contrast, DeRV-Diffusion appends an additional layer/block to the end to perform an extra upsampling double the output resolution. Our generative NeRV also features smooth native time interpolation and can be trained with large frame intervals (6446611) Therefore, our model parameters increase sublinearly, and the neural weight latent size also only needs to increase accordingly. Specifically,
NcRV decoder size increases by 17%. and increase the neural latent token numbe by 25% for simple alignment of channel-wise parameterization.
We compare the inference speed and peak GPU memory of NeRV-Decoder and NeRV-Diffusion with other video decoders and generators, at 16 frames and both 128² and 256 resolutions. Results listed in Table 2 Our models cost far less latency and VRAM footprint in both decoding and generation stages. It demonstrates the superior efficiency of our implicit framework, especially by obviating temporal attentions and reusing the same parameters decode all frames with redundancy.

---

Published as a conference paper at ICLR 2026
4.5 ABLATION STUDIES
We conduct ablation studies assess the key components we propose in an validate the optimal design options for generative objectives. The quantitative results are tested with NeRV-Diffusion-> configuration on UCF and are listed in Sables/anddy
Table Sa indicates that in our NeRV-VAE, our channel-wise parameterization outperforms due to its maximal transparency to decode directly using the encoded weight tokens. In Table 33b] our multi- head affines significantly boost the capacity of NeRV by mapping the whole bottleneck weight tokens to different NeRV layers for reused modulation. Table 3c demonstrates that the spatiotem- poral input embedding of shape h = w = 8 expands the input space with peak expressiveness, while smaller sizes lead to truncated space and bigger sizes result in fewer upsampling layers. We compare different upsampling operations in Table 3d] and find that transposed convolution surpasses pixelshuffle by much fewer parameters and computations without inflated channels. We further ex- plore side connection types in Table Bel. and observe that residual connections fuse raw features at diverse scales without visible artifacts brought by skip connections when summing up multi- resolution RGB output. Finally scale up implicit latent space by increasing the number of tokens, as we meanwhile observe that the token dimension only makes slight impact on the output quality. 128 tokens reaches the peak performance and more tokens will lead to an over complex latent space for diffusion although the reconstruction error continues dropping.
For our implicit denoising network, consider three backbone candidates in Table 4a G.pt was designed for neural weight evolution, but not adapted to diffusion tasks. Latte was designed for video generation and incorporate with temporal attentions, which are beneficial to our implicit generation as NeRV weight tokens lack spatiotemporal structure. Table 4b) showcases that both Min-SNR-cy loss weighting and scheduled sampling scheme effectively minimize the gap between implicit diffusion training and inference, by emphasizing the denoising model more on high-noise predictions and imperfect input.
4.6
PROPERTIES OF GENERATIVE NERV
4.6.1 TIME INTERPOLATION AND EXTRAPOLATION
Benefited from the continuous frame index positional embedding, our generative NeRV features flexible time interpolation and extrapolation capability. In Figures A1 and A2 we interpolate the input time embeddings by factor of 8x to sample 128ffrmme videos with smooth and distinct mo- tions. This property indicates that our generative NeRV efficiently encodes high-density information and understand the residual intrinsic of frame sequences. It enables compact representation long videos and efficient training with fewer frames and large frame intervals.
4.6.2 GENERATIVE NERV WEIGHT INTERPOLATION
Our generative NeRV also features smooth interpolation between two distinct videos by interpo- lating their instance parameters. Given two generative NeRVs' parameters 0 and Oz, we perform linear interpolation λ01 + (1 - X)02 between them. The visual outcomes are exhibited in Figure A3 Our model produces progressively interpretable results compared to DIGAN. We attribute this parametric continuity to not only the Gaussian distribution constraint of our weight latent, but also simple yet effective linear bottleneck mapping and channel-wise parameterization.
CONCLUSION
We propose NeRV-Diffusion, two-staged video synthesis model via NeRV weight generation. Our NeRV-VAE projects videos into a Gaussian weight latent space for tokenization, where our im- plicit diffusion model denoises generate neural weights that render into videos. NeRV-Diffusion outperforms both INR-based and most recent non- non-implicit video generative models on multiple real- world video benchmarks, demonstrating promising scaling law. It also features smooth temporal and parametric interpolation properties. The outstanding performance of NeRV--iffusinn highlights the potential of new holistic video synthesis paradigm with efficient representations.
10

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This work was partially supported by NSF CAREER Award (#2238769) to AS. The authors ac- knowledge UMD's ' supercomputing resourees made available for conducting this research. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwith- standing any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies endorse- ments, either expressed or implied, of NSF or the U.S. Government.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19697-19705, 2023.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15122 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervieed vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Hao Su. Single- stage diffusion nerf: A unified approach to 3d generation and reconstruction. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 2416-2425, 2023.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neu- ral representations for videos. Advances in Neural Information Processing Systems, 34:21557- 21568, 2021a.
Hao Chen, Saining Xie, Ser-Nam Lim, and Abhinav Shrivastava. Fast encoding and decoding for implicit video representation. In ECCV, 2024a.
Yinbo Chen and Xiaolong Wang. Transformers meta-learners for implicit neural representations. In European Conference on Computer Vision, pp. 170-187. Springer, 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628-8638, 2021b.
Yinbo Chen, Oliver Wang, Richard Zhang, Eli Shechtman, Xiaolong Wang, and Michael Gharbi. Image neural field diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8007-8017, 2024b.

---

Published as a conference paper at ICLR 2026
Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. Videoinr: Learning video implicit neural representation for continuous space-time super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Viston and Puttern Recognition, pp. 2047-2057, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview
Emilien Dupont, Adam Goliñski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. Coin: Com- pression with implicit neural representations. arXiv preprini arXiv:2103.03/23 2021.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Goliñski, Yee Whye Teh, and Arnaud
Doucet. Coin++: Neural compression across modalities. arXiv preprint arXiv:2201.12904 2022.
Ziya Erkoç, Fangchang Ma, Qi Shan, Matthias Nießner, and Angela Dai. Hyperdiffusion: Generat- ing implicit neural fields with weight-space diffusion. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 14300-14310, 2023.
Zhiwen Fan, Yifan Jiang, Peihao Wang, Xinyu Gong, Dejia Xu, and Zhangyang Wang. Unified implicit neural stylization. In European Conference on Computer Vision, pp. 636-654. Springer, 2022.
songwei Ge, Thomas Hayes, Harry Yang. Xi Yin, Guan Pang, David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and sensitvve transformer. In European Conference on Computer Vision, pp. 102-118. Springer, 2022.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144 2020.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang. Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.04775, 2023.
Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Lix-video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00/03 2024.
Tiankai Hang. Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 7441-7451, 2023.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for text-to-video generation via transformers. arXiv preprint arXiv:2205, 15868, 2022.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Huiwon Jang, Sihyun Yu. Jinwoo Shin, Pieter Abbeel, and Younggyo Sco. Efficient long video tokenization via coordinate-bart patch reconstruction. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 22853-22863, 2025.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401 4410, 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 10-8119, 2020.
12

---

Published as a conference paper at ICLR 2026
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint 100.::07000009500 2017.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Trans. Graph., 42(4):133-1, 139-1,
Chiheon Kim, Doyup Lee, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattern composers. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 11808-118177 2023a.
Gyeongman Kim, Hajin Shim, Hyunsu Kim, Yunjey Choi, Junho Kim, and Eunho Yang. Diffu- sion video autoencoders: Toward temporally consistent face video editing via disentangled video encoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 6091-6100, 2023b. 20.3b.
Ho Man Kwan, Ge Gao, Fan Zhang, Andrew Gower, and David Bull. Hinerv: Video compres- sion with hierarchical encoding- based neural representation. Advances in Neural Information Processing Systems, 36, 2024.
Joo Chan Lee, Daniel Rho, Jong Hwan Ko, and Eunbyung Park. Ffnerv: Flow Noo-guided frame-wise neural representations for videos. In Proceedings of the 31st ACM International Conference on Multimedia, pp. 7859-7870, 2023.
Sua Lee, Joonhun Lee, and Myungjoo Kang. Minr: Implicit neural representations with masked image modelling. arXiv preprint urXiv:2507222404, 2025.
Zizhang Li, Mengmeng Wang, Huaijin Pi, Kechun Xu, Jianbiao Mei, and Yong Liu. E-nerv: Expe- dite neural video representation with disentangled spatial-temporal context. In European Confer- ence on Computer Vision, pp. 267-284. Springer, 2022.
Ji Lin, Richard Zhang, Frieder Ganz, Song Han, and Jun-Yan Zhu. Anycost gans for interactive image synthesis and editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11986-14996, 2021.
Xiangyue Liu, Han Xue, Kunming Luo, Ping Tan, and Li Yi. Genn2n: Generative nerf2nerf transla- tion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5105-5114, 2024.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100) 2017.
Zhengxiong Luo,
Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang. Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Vidcofusion: Decomposed diffusion models for high-quality video generation. arXiv preprint arXiv:2303. 083200 2023.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge. Yujiu Yang. Limin Wang, and Ying Shan. Open-magvit2: An open-source project toward democratizing auto-regressive visual generation. arXiv preprint arXiv:2409. 044 10, 2024.
Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03044 2024.
Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 9117-9125, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Norman Müller, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrff: Rendering-guidee 3d radiance field diffusion. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4328-4338, 2023.
13

---

Published as a conference paper at ICLR 2026
Mang Ning, Mingxiao Li, Jianlin Su, Albert Ali Salah, and Itir Onal Ertugrul. Elucidating the exposure bias in diffusion models. arXiv preprint arXiv:2308. 15321, 2023.
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8089-8099, 2024.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the JEEE/CVF conference on computer vision and paltern recognition, pp. 165-174, 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint 2209::220..1892 2022.
Zhiyao Ren, Yibing Zhan, Liang Ding, Gaoang Wang, Chaoyue Wang, Zhongyi Fan, and Dacheng Tao. Multi-step denoising scheduled sampling: Towards alleviating exposure bias for diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4667- 4675, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recogmition, pp. 10684-10695, 2022.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20875-20886, 2023.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020.
Ivan Skorokhodov. Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10753-10764, 2021.
Ivan Skorokhodov, Willi Menapace, Aliaksandr Siarohin, and Sergey Tulyakov. Hierarchical patch diffuuson models for high- regh-resolution video generation. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 7569-7579, 2024.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:1212.000 2012.
Yannick Strümpler, Janis Postels, Ren Yang, Lue Van Gool, and Federico Tombari. Implicit neural representations for image compression. In European Conference on Computer Vision, pp. 74-91. Springer, 2022.
Mingzhen Sun, Weining Wang, Gen Li, Jiawei Liu, Jiahui Sun, Wanquan Feng, Shanshan Lao, Yi Zhou, Qian He, and Jing Liu. Ar-diffusion; Asynchronous video generation with rute-regees diffusion. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 7364- 7373, 2025.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489 4499-4497,400 2015.
14

---

Published as a conference paper at ICLR 2026
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXhv:1812.011177 2018.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-We Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprini urXiv:2503.205 20314, 2025.
Hanyu Wang. Saksham Suri, Yixuan Ren, Hao Chen, and Abhinav Shrivastava. LARP: Tokenizing videos with a learned autoregressive generative prior. In The Thirteenth International Confer- ence on Learning Representations, 2025a. URL https:
https::/opeereviieeeeeeee /ttp////ppp/eeeeeeeee/ee/e net Forum?id
Wr 3U3Ex22
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope text-to-video technical report. arXiv preprint arXiv:2308.066: 06571, 2023.
Junke Wang, Yi Jiang, Zehuan Yuan, Bingyue Peng, Zuxuan Wu, and Yu-Gang Jiang. Omnito- kenizer: A joint image-video tokenizer for visual generation. Advances in Neural Information Processing Systems, 37:28281-282955 2024a.
Shuai Wang, Ziteng Gao. Chenhui Zhu, Weilin Huang, and Limin Wang. Pixnerd: Pixel neural field diffusion. arXiv preprini arXiv:2507.23266. 2025b.
Yuyang Wang, Anurag Ranjan, Josh Susskind, and Miguel Angel Bautista. Inrflow: Flow matching for inrs in ambient space. arXiv preprint arXiv:2412.0377 03391,, 2024b.
Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, and Zheng-Jun Zha. Improved video vae for latent video diffusion model. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18124-18133, 2025.
Shuzhou Yang, Moxuan Ding, Yanmin Wu, Zihan Li, and Jian Zhang. Implicit neural representation for cooperative low-light image enhancement. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 12918-12927, 2023.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- -iing Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10459-10469, 2023a.
Lijun Yu, Jose Lezama, Nitesh Bharadwaj Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, Boqing Gong. Ming-Haar Hsuan Yang. Irfan Essa, David A Ross, and Lu Jiang. Language model beats diffusion tokenizer is key to visual generation. The Twelfth International Conference on Learning Representations, 2024a. hRtps:/::
net/ wwt.:/ewwweeetteteeeeeeeeeeeeeee wow.mmidd-ggrraaaeae
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 1721128940-128666 2024b.
Sihyun Yu, Jihoon Tack, Sangwoo Mo, Hyunsu Kim, Junho Kim, Jung-Woo Ha, and Jinwoo Shin. Generating videos with dynamics-aware implicit generative adversarial networks. arXiv preprint 1rX1v:2202.1057,, 2022.
Sihyun Yu, Kihyuk Sohn, Subin Kim, and Jinwoo Shin. Video probabilistic diffusion models in projected latent space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18456-18466, 2023b.
Sihyun Yu. Weili Nie, De-An Huang, Boyi Li, Jinwoo Shin, and Anima Anandkumar. Efficient video diffusion models via content-frant motion- decomposition. In International Conference on Learning Representations, 2024c.
Yu-Jie Yuan, Yang- Tian Sun, Yu-Kun Lai. Yuewen Ma, Rongfei Jia, and Lin Gao. Nerf-editing: ge- ometry editing of neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18353-18388 18364, 2022.
15

---

Published as a conference paper at ICLR 2026
Kaiwen Zha, Lijun Yu, Alireza Fathi, David A Ross, Cordelia Schmid, Dina Katabi, and Xiuye Gu. Language- seeguugg-guided image tokenization for generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 15713-15722, 2025.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 386-595, 2018.
Yunfan Zhang, Ties Van Rozendaal, Johann Brehmer, Markus Nagel, and Taco Cohen. Implicit neural video compression. arXiv preprint arXiv:2112.11312, 2021.
Qi Zhao, M Salman Asif, and Zhan Ma. Dnerv: Modeling inherent dynamics via difference neural representation for videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
16

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 ADDITIONAL. MODEL AND TRAINING DETAILS
We use a medium configuration of ViT with 18 blocks, 14 heads and 896 hidden dimensions for our NeRV encoder. We the scale of KL divergence loss x 10-5. It patchifies the RGB videos into 8 x 8 X 1 patches along height, width and time dimensions. We use sinusoidal positional embedding for our generative NeRV's input time index, instead of the exponential embedding in vanilla NeRV. For residual connections we use bilinear to upsample the earlier feature maps before merging them into the main branch.
Our discriminator is adapted from a 3D StyleGAN with blocks, 64 unit hidden dimensions and channel multiplier of for each block. Its learning rate set one fifth of the NeRV-VAB's and it is updated every five iterations to stabilize the training. The scale of the GAN loss added to our MeRV-VABS is
Our implicit diffusion transformer adopts DiT-L, configuration with 24 layers, 16 heads and 1024 hidden dimensions. Its patch size follows the token shape as output by our NeRV encoder. Our implicit DiT is optimized for predicting the noise at each timestep, and thus also adjust the Min- SNR-y loss weighting accordingly. We employ CFG for class-conditioned sampling and the optimal guidance scale is 2.
Both our VAE and implicit DiT train with L2 reconstruction loss. We use Adam (Loshchilov] 2017) optimizer with linear warmup learning rate schedule and cosine decay. Both learning rates are set to 10-1. We train NeRV-VAB for 1M iterations and implicit DiT for 500K iterations small batch size of 32. We train each of them for week NVIDIA A6000 GPUs.
A.2 NERV DECODER ARCHITECTURE
We list the layer-wise architecture of our NeRV model in Table Al "Feature Map" refers to the output feature map of each layer. "Modulation Weight" refers to the instance-specific weight latent be assigned each NeRV layers. T is the number of frames.
We set the dimensions of the time, height and width positional embeddings all to 16. We start from sampling a spatiotemporal positional embedding shape [8, 8, T, 48). transposed to queries along the time axis [T. 48, 8, 8), and then spatial convolutions are applied on it.
We use kernel size tions that don't 't change the feature map shape. We set the base hidden dimensions D = 128, 256, 512 for NeRV-Diffusion-S, -B and -L configurations, respectively. gelu is used for activations in all blocks while tanh is used after the tailing toRGB layer.
for all upsampling transposed convolutions and k: for all other convolu-
B TIME INTERPOLATION
As discussed in 1 we train NeRV-Diffusion on UCF with an interval of 8 frames, and inter- polate the input time embeddings by an 8x factor to sample 128-frame videos. The results are presented in Figures Al and A2
Quantitative Verification. We measure long video semantic consistency in terms of object iden- tity and action logic preservation. Specifically, we leverage the Subject Consistency (SC) and Back- ground Consistency (BC) metrics in VBench, which extract the subject and background features via pre-trained DINO (Caron et al. 2021), and calculate their pairwise similarities across all frames for appearance preservation. Inspired by it, we further employ C3D, pre-trained action recognizer network, and feed with all sub-clips from the long videos to extract action features. We calculate pairwise similarities of the action features across all windows to measure the action logic preser- vation. These metrics don't 't only rely on consecutive distance but average along the whole video, accurately reflecting the drifting issues for long video generation. The results are listed in Table A2

---

Published as a conference paper at ICLR 2026
Table Al: Detailed architecture of our generative NcRV Decoder and modulated weight shape.
Batch size is omitted.
Feature Map Shape
Modulation Weight Shape
Layer
Input Init
8, 8, T, [7,4,, 8,8
Reshape
Conv Transposed Conv
[T. D,
[18, D, 3. or [21, 3/2,3, [64, 6,44,4
TT. 16,
Conv
[T,,166,10
[61, 61,,33
Conv
TT, T,D,16, 16,,16,10 16
[61,6433,3
Transposed Conv
T, T.3222,32
44,,64,4,4
Conv
T,2,32,32
(64, 64,,643333
Conv Transposed Conv
D.2,22,3
[64, 6,,
D,D,64
64, 64, 4,4
Conv
64, 64
[64, 64, 3,33
Conv
24,
[128, 64,
Transposcd Conv
T,2D, 128, 128
664, 4,4
Conv
[,2D, 128, 128
664, 3,3
toRGB Reshape to Output
TT,3, 128, 128 [T 128, 128, 3
[D,3,3,3]
Table A2: Quantitative measurement of time interpolation semantic consistency.
Long Consistency
VBench*
Action Sim.
SC
BC
Ground Truth
0.931
0.956
0.912
Ours
0.919
0.942
0.901
C INR WEIGHT INTERPOLATION
We further illustrate our generative NeRV's superiority in INR weight interpolation. DIGAN Yu et al., 2022) proposes interpolate between the latent noise vectors. When being interpolated be- tween the whole weights, their video INR presents non-continuous transitions as shown in Figure A3 (top). This is because 1) their latent vectors are decoded from Gaussian noise with a complex non- linear mapping network; 2) their INR weights are modulated with low-rank cross product, termed as Factorized Matrix Multiplication (FMM) in in/Skorokhodov al. (202I)). of the latent vectors, which break the arithmetic property. In contrast, our weight latent is directly used for modulation with a single linear affine layer from the KI. bottleneck, and is directly assigned as NeRV parameters with minimal transforms. Our generative NeRV presents smooth interpolation effect as shown in Figure A3 (bottom). This property also opens up the potential of general direct manipulations on the tokenized NeRVs in compositional manner with our NeRV-VAE.
D ADDITIONAL EXPERIMENTS AND RESULTS
D. VIDEO GENERATION
We provide generated samples of NeRV-Diffusion UCF in Figure A4
Qualitative Comparisons We present additional visual comparisons with LARP Wang et al. 2025a) on UCF in Figures A5 and A6 We sample both models with the same class label input. Compared to LARP, NeRV NeRv-Diffusinn also constructs holistic video representations but meanwhile still maintain the spatiotemporal integrity via the query time indices and spatial input embeddings, and thus produces more structural videos with less morphing or tearing.
2

---

Published as a conference paper at ICLR 2026
D.2 ADDITIONAL QUANTITATIVE METRICS.
We extend our reconstruction metrics to PNSR, SSIM and LPIPS between the output and imput videos, and extend our generation metrics cross-frame LPIPS (fI.PIPS) and C3D (Tran et al, 2013) I3D (Carreira & Zisserman 2017) -based Inception Score (Salimans et al. 2016) (IS) on the output videos. Due to the lack of pair-wise supervision in generation, we also evaluate gener- ators on the non-text subsets of VBench Huang al. (2024), including Subject Consistency (SC), Background Consistency (BC), Temporal Flickering (TH), Motion Smoothness (MS), Dynamic De- gree (DD), Aesthetic Quality (AQ) and Imaging Quality (IQ). For comparing methods, pretrained checkpoints for inference.
their
The results of reconstruction and generation UCF and K600 listed in Table A3 It worth noting that the ultimate purpose of NeRV-VAE (and the comparing tokenizers) to encode RGB videos into smooth latent for high-quality video generation, and we optimize it for best generative quality instead of soiely reconstruction faithfulness. Our models achieve SOTA generation performance on UCF across all dimensions, and reach comparable SOTA performance on K600 frame prediction.
Table A3: Additional reconstruction and generation metrics on UCF and K600.
(a) Additional reconstruction on UCF,
Reconstruction
PNSR
SSIM
LPIPS
rFVD
TATS (Ge et al.. 2022
22.43
0.765
0.108
162
Open-MAGVIT+ (Luo al. LARP-L Waag et al. 2025a
2024
25.84
0.862
0.045
16
27.87
0.891
0.038
20
NeRV-VAEA
26.63
0.879
0.043
22
(b) Additional reconstruction metrics K600.
Reconstruction
PNSR
SSIM
LPIPS
rFVD
LARP-L WWang al.) 2025a
28.22
0.867
0.035
NeRV-VAE-L
26.45
0.823
0.044
19
(c) Additional generation metrics on UCE
Generation
ILPIPSt
1St
VBencht
gFVDJ
SC
TF
MS
IQ
Ground Truth
0.031
86.24
0.954
0.977
0.981
0.988
0.282
0.410
0.443
TATS GGe et al 2022
0.018
68.79
0.910
0.958
0.978
0.980
0.314
0.348
0.437
332
VIDM Mei & Patel 2028
64.17
263
VideoDillusion Luo al 2023
80.03
LARP-L (Wang
0.025
68.79
0.955
0.977
0.985
0.991
0.218
0.398
0.393
102
NeRv-Diffusion-L
0.028
82.17
0.958
0.978
0.983
0.989
0.262
0.392
0.433
(d) Additional generation metrics on K600.
Generation
ILPIPS
1St
VBench?
gFVD,
SC
BC
TF
MS
DD
AQ
IQ
Ground Truth
0.031
31.20
0.950
0.945
0.978
0.988
0.266
0.406
0.461
LARP-L Wang et al.
2025a
0.029
23.51
0.933
0.973
0.981
0.989
0.292
0.337
0.293
17
L
0.034
27.09
0.928
0.967
0.977
0.979
0.354
0.392
0.432
22
D.3 DOWNSTREAM TASKS WITH CONTROLLABILITY
Our implicit diffusion model follows standard diffusion framework but only switches to neural weight latent, and thus be trained with versatile condition types. Additional condition input can be integrated with standard cross attention so that different modalities other than videos or im- ages are also supported. We extend NeRV-Diffusion to downstream tasks including image-to-video

---

Published as a conference paper at ICLR 2026
generation and unconditional video generation. To showcase the support of granular controllability of our neural weight latent, we also include an edge-to-vides generation experiment. We present qualitative results in Figure A7 These experiments demonstrate the native extensibility of NeRV- Diffusion spanning from none to fine-grained condition controls with high quality.
E ADDITIONAL DISCUSSIONS
E. DATASET DETAILS
UCF dataset contains over 13K real-world videos sourced from YouTube of average 180 frames, totally about 27 hours. K600 dataset contains about 500K videos of average 250 frames, summing up to about 57 days length.
E.2 TWO-STAGE GENERATIVE FRAMEWORK AND TRADE-OFF
Two-stage tokenizer-generator frameworks are widely adopted for their stability and efficiency com- pared to single-stage generation models, especially when scaled up to large-scale models and data. All generative tokenizers need to balance between the latent compactness and expressive richness, such as the KL constraint in VAE andthe codebook alignment for VQGANs.
The gap between gFVD and rFVD is controlled by the KI. loss scale. A high KI. loss scale will harm rFVD but reduce the gap between gFVD and rFVD, while a low KL loss scale will improve rFVD but enlarge the gap between gFVD and rFVD. We ablate its impact NeRV-Diffusion-S configuration Table A4
Table A4: Ablation on KL loss scales on NeRV-Diffusion-S configuration.
KL Loss Scale
1 x 10
5 x
10-6
1 x 10-5
5 x 10
rFVD
73
82
85
107
gFVD
198
186
184
202
Moreover, we conduct marginal tests and normality plots on our encoded neural weight latent, and compare it to LDM/SD's VAE latent. The statisties and visualizations are presented in Table A55 and Figure A8
Table A5: Latent Gaussianity statistics.
Gaussianity
mean
std
skew
kurtosis
LDM/SD-VAE 0.0165
5.2505
0.4756
0.1693
NeRV-VAE-L
-0.0036
0.9979
-0.0001
0.2568
E.3 GRANULARITY OF NEURAL NETWORK WEIGHTS
Our neural latent forms up the convolution weights of NeRV decoder, taking in unified spatiotempo- ral input embeddings to render pixel frames. While it is holistic and different from traditional frame- wise feature maps, it still performs convolutions, i.e. matrix input grid (in the opposite positions), and thus still maintains spatiotemporal information. This is major difference of our implicit representations between discrete holistic vidco latent such as LARP Wang et al. (2025a).
productions over the spatiotemporal
We design probing experiment to validate the spatiotemporal association between our neural weight latent and pixel frames. We crop RGB videos into random clips with random spatial and temporal ranges, and encode them using various video encoders. For each video encoder, we per- form K-Means its encoded latent, with the raw video sources the class labels. We calculate the
4

---

Published as a conference paper at ICLR 2026
purity and normalized mutual information (NMI) between the latent clusters and their real labels, ie. which raw video the corresponding clip is cropped from. A that our neural weight latent has spatiotemporal relations with pixel operations, indicating its potential to facilitate granular editing.
Table A6: Time interpolation semantic consistency.
Granularity
Purityt
NMI
SD-VAE
0.484
0.753
LARP
0.285
0.591
Ours
0.366
0.656
E.4 COMPARISON WITH OPEN-WORLD LARGE-SCALE VIDEO TOKENIZERS.
LTX (HaCohen al. 2024) and WAN WWan al. 2025) commercial foundation models that are trained on world large-scale data with considerable resources and extensive time. Though, list the quantitative comparisons in Table A7 We also include the VAE of Stable Diffusion (SD) and Stable Video Diffusion (SVD), which are also large-scale foundation models at earlier stages.
Table A7: Comparison of reconstruction performance on UCF with foundation video tokenizers.
Reconstruction (UCF)
PNSR
SSIM
LPIPS
rFVDJ
SD-VAE
24.79
0.783
0.068
63
SVD-VAE
27.24
0.856
0.058
39
LTX-VAE
31.29
0.909
0.061
32
WAN-VAE
31.14
0.935
0.022
NeRV-VAE-L
26.63
0.879
0.043
22
E.5 NEURAL WEIGHT LATENT DIMENSION ABLATION
We ablate different neural latent token dimensions with SeRV-Diffusion-S configuration in Table A8 It results in slight performance drop when being reduced, but not significant token num- bers shown in Table 3r We attribute this to that our neural weight latent holistic representations unlike traditional frame-wise feature maps, and the token dimension doesn't correspond to the RGB
color channel.
Table A8: Ablations neural weight latent dimensions.
Token Shape
128 x 32
128 x 64
128
96
128
128
FFVD
114
99
92
85
gFVDJ
230
212
197
184
E.6 ENCODING AND DECODING EFFICIENCY 720P
We create random data and test the encoding/decoding latency on 1280 x 720 video. The encoding takes approximately 1.9s, and the decoding takes approximately 1.8s, both on a single NVIDIA A6000 GPU.
E.7 PERFORMANCE CEILING OVER EFFICIENCY
So far we haven't t noticed such a ceiling as NeRV-Diffusion surpasses previous SOTA methods of both diffusion and autoregressive models. Meanwhile, the outstanding efficiency of NeRV-Diffusion
5

---

Published as a conference paper at ICLR 2026
also indicates its extra potential when scaling up. If performance is prioritized over efficiency, NeRV. Diffusion can also further scale up to comparable efficiency for extra performance boost.
F REPRODUCIBILITY STATEMENT
Our code and trained checkpoints will be made publicly available upon publication. We have dis- cussed our complete implementation details in and Appendix A including the model configu- ration and training recipe.

---

Published as a conference paper at ICLR 2026
Figure A1: 128-frame video generation results on UCF. NeRV VAE and NeRV-Diffusion only see one in every 8 frames in training, and is able to temporally generalize to unseen time stamps.

---

Published as a conference paper at ICLR 2026
Figure A2: 128-frame video generation results on UCF. NeRV VAE and NeRV-Diffusion only see
one in every 8 frames in training, and is able to temporally generalize to unseen time stamps.
8

---

Published as a conference paper at ICLR 2026
0.8
-
0.6
2
0.4
022
--0
2=1
- 0.6
= 0.4
- 0.
Figure A3: Interpolation of the whole parameters of the video INRs in DIGAN (top) and our gener-
ative NeRVs (bottom).

---

Published as a conference paper at ICLR 2026
Figure A4: Additional class-conditioned generation samples on UCF.
10

---

Published as a conference paper at ICLR 2026
LARP
Ours
LARP
Ours
LARP
Ours
LARP
Ours
Figure A5: Qualitative comparisons with LARP on UCF class-conditioned generation.
11

---

Published as a conference paper at ICLR 2026
LARP
OuEr
LARP
Ours
LARP
Ours
LARP
Ours
Figure A6: Qualitative comparisons with LARP on UCF class-conditioned generation.
12

---

Published as a conference paper at ICLR 2026
(a) Unconditional generation on UCF
(b) Image to video generation on K600.
(c) ladec-oonditioned video generation on K600.
Figure A7: Versatile condition types of video generation supported by NeRV-Diffusion.
13

---

Published as a conference paper at ICLR 2026
Global lafent nistogram
Global latent histegram
0.48
5.08*20
0.00
0.35
0.06
0.25
0.64
0.42
0.10
0.00
20
0.00
7.5
-5.0
5.0
3.5
(a) Histogram SD-VA DAF latent.
(b) Histogram of NeRV VAF latent.
Global latent Q-Q plat
Global latent Q-Q plot
30
20
Ordered Values
Ordered Values
2
0-
330
-6
Theoretical quantiles
Theoretical quantiles
(c) Quantile-Quanee 2D-VAE latent.
(d) Quantile-Quantiee NeRV-VAE latent.
Figure A8: Gaussianity verifications of the latent encoded by NeRV- VAE and SD- VAE.
14


Reviews:
[{"review_id": "szUYUfVECh", "reviewer": "Reviewer_kMwA", "summary": "This paper proposes NeRV-Diffusion, which synthesizes videos by generating implicit neural representation (INR) weights through diffusion models. The method first uses a hypernetwork-based tokenizer to encode videos into neural network parameters, then uses a diffusion transformer to denoise and generate in the weight space. The core idea is to represent videos holistically as a set of INR weights, rather than frame-by-frame feature maps.", "questions": "Please address the problems in the weakness section.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The idea of shifting from generating pixels to generating neural network weights is unique in the video generation field. Although there is precedent for INR generation (DIGAN 2022), applying diffusion models to video INR weight space still has exploratory significance.\n\nThe combined design of KL bottleneck, multi-head affine modulation, and channel-level parameterization to some extent solves the contradiction between weight distribution regularization and expressive capability.", "weaknesses": "Only compared decoder parameter count (3.5M-55M) and token count (128), missing actual inference speed.\n\nEvaluation metrics are too singular, only using FVD as one metric.\n\nVideo generation resolution is low and length is short, all experiments only at 128×128, unable to prove the method can scale to higher resolutions.\n\nThe dataset scale used is small and old, unable to verify the method's performance in real large-scale scenarios.\n\nComparison methods and baselines are outdated, missing comparisons with latest methods such as wan2.1, Self-Forcing, etc., and method metrics are too few, moreover not SOTA, advantages are not obvious.\n\nLacks qualitative comparison results display and analysis with other methods.\n\nInterpolation experiments lack quantitative metric verification, only on class-conditioned and frame prediction two tasks, unable to prove the method's generalizability, such as text-to-video, image-to-video, video editing, unconditional generation.\n\nThe core paradigm (generating INR weights) comes from DIGAN, improvements are mainly engineering tricks (KL constraint, multi-head modulation, scheduled sampling, etc.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7BQBOt33Ek", "reviewer": "Reviewer_FaBD", "summary": "NeRV-Diffusion is a novel two-stage generative framework for video synthesis that operates by generating the parameters of an Implicit Neural Representation (INR). Instead of synthesizing videos in pixel space or a traditional latent feature space, it first uses a hypernetwork-based autoencoder to \"tokenize\" videos, compressing them into a compact set of latent INR weights. This latent space is explicitly regularized to follow a Gaussian distribution. In the second stage, an Implicit Diffusion Transformer (DiT) is trained to model this latent space. To generate a new video, the DiT denoises a random noise vector into a new set of valid INR weights. These weights are then used to parameterize a convolutional INR (a \"Generative NeRV\"), which can decode the full-length video by taking continuous frame indices as input. The model achieves performance comparable to state-of-the-art non-implicit models on benchmarks like UCF-101 and Kinetics-600, while offering significant efficiency advantages and smooth interpolation properties.", "questions": "1. How does the performance (training stability, gFVD) and computational cost scale with respect to (a) video resolution (e.g. 512) and (b) video duration (e.g., 256 frames)? What is the growth rate of the INR parameter count (the token sequence length for the DiT)?\n2. Given the abstract nature of the weight space, what are the prospects for conditional generation beyond simple class labels or frame-gaps? Could this framework be guided by text, depth maps, or semantic layouts?\n3. How sensitive is the final generation quality to the fidelity of the first-stage autoencoder? How important is the D_KL loss weight in balancing reconstruction fidelity vs. a smooth, generative latent space?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By representing the video holistically as a single neural network, the model obviates the need for computationally expensive cross-frame attentions in the denoising network. This directly addresses a major computational bottleneck present in traditional frame-wise video diffusion models.\n2. The framework generates a compact, instance-specific decoder (the INR) rather than relying on a large, shared, universal decoder (like a VAE). INRs are known for being fast at the decoding (rendering) stage, making the visualization of generated videos computationally efficient.\n3. The model can be trained on sparse frames but generate smooth, high-frame-rate videos simply by sampling more frame indices at inference time (e.g., generating 128 frames from a 16-frame training setup).\n4. Traditional tokenizers that create frame-wise feature maps lead to redundant representations by ignoring inter-frame coherence. NeRV-Diffusion's approach of encoding the entire video into a single set of parameters is a much more compact and holistic representation.\n5. Despite being a new paradigm, the model demonstrates high-quality results. It significantly outperforms previous INR-based generative models (like DIGAN) and achieves comparable performance to non-implicit SOTA models (like MAGVITV2-AR) on standard gFVD benchmarks.", "weaknesses": "1. The framework is not end-to-end. It relies on a two-stage pipeline: (1) training the NeRV autoencoder, and (2) training the diffusion model on the frozen latent space. The final generation quality is fundamentally bottlenecked by the fidelity and \"Gaussian-ness\" of the latent space produced by the first stage.\n2. The latent space consists of neural network weights, which are highly abstract and have no intuitive spatiotemporal or semantic meaning. This makes granular, controllable editing extremely difficult, if not impossible, without retraining.\n3. The core challenge is forcing the INR weight space to be both highly expressive (to reconstruct complex videos) and simple enough to be modeled by a Gaussian distribution (for diffusion). The paper's numerous architectural modifications (multi-head affine, channel-wise parameterization et. al.) highlight the difficulty of balancing this trade-off. \n4. The experiments are conducted on low-resolution (128) and short (16-frame) videos. It is unclear how this framework scales to high-definition (HD/4K) or long-duration videos. A higher-resolution INR would require vastly more parameters, creating a much larger token sequence for the DiT to generate, which is a significant challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYVnqyBbSW", "reviewer": "Reviewer_WJ85", "summary": "The paper proposes a two-stage, INR-based generative pipeline: (1) a hypernetwork tokenizer encodes a video into a compact latent vector whose entries are used directly as the convolution kernels of a small NeRV decoder; (2) a Diffusion Transformer (DiT) denoises in this weight-token space to generate new videos. The authors have taken measures including KL bottleneck after the encoder, Gaussian bottleneck, channel-wise parameterization to ensure the weights follow (multivariate) normal distribution and facilitate smooth interpolations. On UCF-101 class-conditioned generation and Kinetics-600 frame prediction, the method reports competitive (g)FVD at 128×128, 16 frames, with fast per-frame decoding thanks to the tiny instance-specific decoder. It also shows smooth time and weight-space interpolations.", "questions": "- What might be the failure modes for the Gaussianity-diffusion trade-off case? Would this be harder to guarantee for larger decoder/videos or videos with more complicated spatiotemporal dynamics?\n- To scale up to higher resolutions, which one of NeRV capacity, weight latent size, DiT training, would be the first to break? Did you try increasing token size or decoder width beyond the ablation settings?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of generating leveraging the lightweightness and interpolation ability of INRs to produce videos with good spatiotemporal coherence (without having to rely on cross-attention) is interesting.\n- The model is compact and associated with fast decoding. Requiring only 128 tokens, a very small decoder, and a reasonable-sized DiT generator, the model still delivers pretty impressive performance.\n- The engineering efforts that make the weight-space diffusion work all make intuitive sense and well supported by the ablation results.", "weaknesses": "- **Narrow scale & task scope.** Though the use of UCF-101 and K600 is standard, all experiments are conducted only at 128x128 resolution with 16 frames, which is acceptable for early proofs-of-concept but could be disadvantageous for those larger baselines, as this setup is too simple to expose scalability at higher resolutions & longer videos.\n- **Heavy training.** The two-stage system requires longer training times and increases engineering burden, and it is expected that scaling up this framework might be even heavier; further, the DiT is still sizeable so it is hard to determine the actual runtime speed up saved.\n- **Gaussianity is not verified.** The paper doesn't show diagnostics (e.g., per-layer marginal tests, normality plots, etc.) to verify how Gaussian the INR weights actually are, and under what conditions would the Gaussianity assumption be counter-effective?\n- **Limited metrics.** The authors should consider reporting more metrics (e.g., tLPIPS, or some other more well-rounded metrics, like those from VBench or WorldScore), especially consider the performance is not reported for the K600 dataset.\n\nI think the paper has good novelty and the combination of diffusion (or generative models in general) with INRs for video generation/synthesis could be of great interests to the community. However, I am slightly concerned with how extendable & scalable this method is and how well it performs after more comprehensive evaluations, but I am willing to raise my score if these concerns are addressed by the authors' response.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KBdklBX9NH", "reviewer": "Reviewer_WpVC", "summary": "This paper proposes **NeRV-Diffusion**, a novel implicit video diffusion model that synthesizes videos by generating the weights of an INR. The framework consists of two stages:  \n\n- **Tokenization stage**: A hypernetwork-based encoder compresses raw videos into a compact set of neural weights (i.e., INR parameters). These weights directly instantiate a NeRV  decoder, which reconstructs the video given frame indices as input.  \n- **Generation stage**: An DiT is trained in this implicit weight space to denoise from Gaussian noise and generate new INR weights, thereby synthesizing novel videos.  \n\nExperiments show that NeRV-Diffusion matches or surpasses current state-of-the-art non-implicit video generation models (e.g., MAGVITv2-AR, LARP) on UCF-101 and Kinetics-600, while using significantly smaller model and latent sizes. It also supports high-quality **temporal interpolation** and **INR weight interpolation**.", "questions": "**On Efficiency**:  \n- Could you provide quantitative comparisons between NeRV-Diffusion and MAGVITv2 or Latte in terms of **inference latency** and **GPU memory usage**?  \n- When generating 128-frame videos, does the per-frame forward pass of the NeRV decoder become a bottleneck?  \n- What is the encoding/decoding efficiency when handling 720p video reconstruction and generation?  \n- How does the **training efficiency** (e.g., GPU-hours, convergence speed) compare to standard video VAEs?\n\n\n**On Model Architecture**:  \n- What exactly does **Multi-head Affine Modulation** do? How does it enable the reconstruction of a full video from a single set of latent tokens?  \n\n**On Latent Dimensionality**:  \n- Each token has a dimension of 128, which seems large compared to recent DiT-based models (e.g., Sora/WAN), whose VAEs use only 16 or 48 dimensions. Why not adopt a lower latent dimension?  \n- With only **128 tokens**, how is high-fidelity reconstruction still achievable? What factors determine this efficiency?  \n- Could you report **PSNR, SSIM, and LPIPS** scores for reconstruction?  \n- Even if training data differs, can you compare **compression rate** and reconstruction quality against LTX Video VAE or WAN 2.2 VAE?\n\n**On Performance Bottlenecks**:  \n- Why was the token count fixed at 128? What are the reconstruction and generation results with 512 or 1024 tokens?  \n- If one prioritizes **maximum performance over efficiency**, does this method have a relatively low ceiling compared to conventional latent diffusion approaches?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "- **Highly innovative**: First work to combine diffusion models with video INR weight generation, establishing a new paradigm of “holistic video synthesis via neural weights.”  \n- **Technically solid**: Key architectural improvements to NeRV (e.g., channel-wise parameterization, multi-head affine modulation) are well-motivated and validated via ablation studies.  \n- **Comprehensive experiments**: Evaluated on UCF-101 and Kinetics-600 against GANs, autoregressive (AR), and diffusion-based models, with multi-dimensional assessments including reconstruction, generation, frame prediction, and interpolation.", "weaknesses": "- **Lack of quantitative efficiency analysis**: Although “fast decoding” is claimed, the paper does not report inference speed (e.g., FPS), memory footprint, or direct comparisons with LDM/VQ-VAE.  \n- **Scalability and resolution limitations**: All experiments are conducted only on 128×128×16 videos. Performance at higher resolutions (e.g., 256×256) or longer sequences (>16 frames) is not evaluated.  \n- **Limited analysis of long-video interpolation**: While long videos are generated via time interpolation, it remains unclear whether semantic consistency (e.g., object identity, action logic) is preserved.  \n- **Insufficient comparison with recent tokenization methods**, such as LTX Video VAE or WAN VAE.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)