Title: PCAInit: Training-Free Initialization for Image-Based Neural Representations
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Image Reconstruction, Implicit Neural Representations (INRs), Representation Learning, Representational Alignment, Weight Initialization, Weight Space

Abstract: Implicit neural representations (INRs) have been widely used to model data as continuous functions parameterized by multi-layer perceptrons (MLPs). 
Although this approach is effective, it has two major limitations when applied to large-scale data such as video frames: (1) substantial GPU time is required for training and (2) subpar reconstruction quality. 
In this paper, we tackle these two limitations and investigate three initialization strategies for INR training, using the sinusoidal representation network (SIREN) architecture as a baseline.
Our first two methods leverage pretrained models, offering higher reconstruction quality at the cost of additional training time. 
Finally, our third method, PCAInit, is a novel training-free initialization approach derived from analyzing the relationship between image space and weight space through principal component analysis (PCA).
We show that this method achieves the best overall reconstruction quality without extra training time. For example, on a representative DAVIS 2017 video (bear, 480p), PCAInit improves PSNR by up to +37.1\% over SIREN and +26.7\% over meta-learned initialization, and also achieves the highest PSNR in the zero-shot super-resolution task. 
Furthermore, we show that PCAInit generalizes beyond video frames, achieving the best PSNR on collections of images as well. 
Our results reveal a promising research direction on the interplay between image space and weight space in INRs, opening new avenues for future research on efficient INRs with improved reconstruction quality and broader applicability.

TL;DR: We introduce PCAInit, a novel training-free initialization method derived from an analysis of the relation between image space and weight space through principal component analysis.