Neural Metamorphosis
Authors: name: Xingyi Yang; name: Xinchao Wang
Published: 2024-10-10 14:49:58

Abstract: This paper introduces a new learning paradigm termed Neural Metamorphosis
(NeuMeta), which aims to build self-morphable neural networks. Contrary to
crafting separate models for different architectures or sizes, NeuMeta directly
learns the continuous weight manifold of neural networks. Once trained, we can
sample weights for any-sized network directly from the manifold, even for
previously unseen configurations, without retraining. To achieve this ambitious
goal, NeuMeta trains neural implicit functions as hypernetworks. They accept
coordinates within the model space as input, and generate corresponding weight
values on the manifold. In other words, the implicit function is learned in a
way, that the predicted weights is well-performed across various models sizes.
In training those models, we notice that, the final performance closely relates
on smoothness of the learned manifold. In pursuit of enhancing this smoothness,
we employ two strategies. First, we permute weight matrices to achieve
intra-model smoothness, by solving the Shortest Hamiltonian Path problem.
Besides, we add a noise on the input coordinates when training the implicit
function, ensuring models with various sizes shows consistent outputs. As such,
NeuMeta shows promising results in synthesizing parameters for various network
configurations. Our extensive tests in image classification, semantic
segmentation, and image generation reveal that NeuMeta sustains full-size
performance even at a 75% compression rate.