Magnitude Invariant Parametrizations Improve Hypernetwork Learning
Authors: Jose Javier Gonzalez Ortiz
John Guttag
Adrian V Dalca
Venue: ICLR 2024 poster (2024)
Keywords: deep learning
hypernetwork
stability
optimization
Primary area: general machine learning (i.e., none of the above)
TL;DR: We make hypernetworks work better by identifying one of their main pathologies and fixing it, leading more stable and faster training.

Abstract: Hypernetworks, neural networks that predict the parameters of another neural network, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunately, existing hypernetworks are often challenging to train. Training typically converges far more slowly than for non-hypernetwork models, and the rate of convergence can be very sensitive to hyperparameter choices. In this work, we identify a fundamental and previously unidentified problem that contributes to the challenge of training hypernetworks: a magnitude proportionality between the inputs and outputs of the hypernetwork. We demonstrate both analytically and empirically that this can lead to unstable optimization, thereby slowing down convergence, and sometimes even preventing any learning. We present a simple solution to this problem using a revised hypernetwork formulation that we call Magnitude Invariant Parametrizations (MIP). We demonstrate the proposed solution on several hypernetwork tasks, where it consistently stabilizes training and achieves faster convergence. Furthermore, we perform a comprehensive ablation study including choices of activation function, normalization strategies, input dimensionality, and hypernetwork architecture; and find that MIP improves training in all scenarios. We provide easy-to-use code that can turn existing networks into MIP-based hypernetworks.

Published as a conference paper at ICLR 2024
MAGNITUDE INVARIANT PARAMETRIZATIONS IMPROVE HYPERNETWORK LEARNING
Jose Javier Gonzalez Ortiz MIT CSAIL
John Guttag MIT CSAIL Cambridge, MA guttagêmit. edu
Adrian V. Dalea
MIT CSAIL & MGH, HMS
Cambridge, MA josejg@mit.ccm edu
Cambridge, MA adal ca ca edu
ABSTRACT
Hypernetworks, neural networks that predict the parameters of another neural net- work, are powerful models that have been successfully used in diverse applications from image generation to multi-task learning. Unfortunately, existing hypernet- works are often challenging to train. Training typically converges far more slowly than for non-hypernetwork models, and the rate of convergence can be very sen- sitive to hyperparameter choices. In this work, we identify a fundamental and previously unidentified problem that contributes to the challenge of training hy- pemetworks: a magnitude proportionality between the inputs and outputs of the hypernetwork. We demonstrate both analytically and empirically that this can lead unstable optimization, thereby slowing down convergence, and sometimes even preventing any learning. We present a simple solution to this problem using a revised hypernetwork formulation that we call Magnitude Invariant Parametriza- tions (MIP). We demonstrate the proposed solution on several hypernetwork tasks, where consistently stabilizes training and achieves faster convergence. Further- more, we perform a comprehensive ablation study including choices of activation function, normalization strategies, input dimensionality,, and hypernetork archi- tecture; and find that MIP improves training in all scenarios. We also provide easy-to-use code that can turn existing networks into MIP-based hypernetworks.
INTRODUCTION
Hypernetworks, neural networks that predict the parameters of another neural network, are increas-
ingly important models in a wide range of applications such as Bayesian optimization (Krueger)
al. 2017: Pawlowski al. 2017), generative models (Alaluf
al. 2022, Zhang & Agrawala
2023
amortized model learning (Bae et al. 2022, Dosovitskiy & Djolonga 2020 Hoopes et al.
2022
continual learning (Ehret et al. 2021 von Oswald et al. al.] 2020), multi-task learning (Ma-
hhabadi al. 2021 Serrà et al.) 2019). and meta-lcarning (Bensadoun et al. 2021 Zhao et al.
2020). Despite their advantages and growing use, training hypernetworks is challenging. Compared non-hhypernetwork-aassee models, training existing hypernetworks is often unstable. At best this increases training time, and at worst can prevent training from converging at all. This burden lim- its their adoption, negatively impacting many applications. Existing hypernetwork heuristics, like gradient clipping (Ha et al., 20166 Krueger et al. 2017), are most often insufficient, while existing techniques improve standard neural network training often fail when applied to hypernetworks.
This work addresses a cause of training instability. We identify and characterize a previously unstud- ied hypernetwork design problem and provide a straightforward solution to address it. We demon- strate analytically and empirically that the typical choices of architecture and parameter initialization in hypernetworks cause a proportionality relationship between the scale of hypernetwork inputs and the scale of parameter outputs (Fig. Ta). The resulting fluctuations in predicted parameter scale lead to large variability in the scale of gradients during optimization, resulting in unstable training and slow convergence. In some cases, this phenomenon prevents any meaningful learning. To over- come the identified magnitude proportionality issue, we propose a revision to hypemetwork models: Magnitude Invariant Parametrizations (MIP). MIP effectively eliminates the influence of the scale of hypernetwork inputs on the scale of the predicted parameters, while retaining the representational power of existing formulations. We demonstrate the proposed solution across several hypernetwork

---

Published as a conference paper at ICLR 2024
(a) HyperNetwork Proportionality Issue
(b) Convergence Improvements
Default Hypernetwork
MIP Hypernelwork
0.25
(tdev(e)
0.20
Hypemetwork
Default
0.05
Initial
0.10
(ours)
Final
0.00
0.00 20000
0.0
0.5
1.0
0.0
0.5
1.0
40000 60000
80000
100000
Iteration
Figure 1: (a) Proportionality Issue. With default formulations, the scale of the predicted parame- ters 0 (measured in standard deviation) is directly proportional to scale of the hypernetwork input at initialization (initial), and even after training the model (final). Our proposed Magnitude Invari- ant Parametrizations (MIP) mitigates this proportionality issue with respect to y. (b) Convergence Improvements. Using MIP leads to faster convergence and results in more stable training than the default hypernetwork formulation. The latter experiences sporadic training instabilities (spikes in the training loss).
learning tasks, providing evidence that hypernetworks using MIP achieve faster convergence and more stable training than typical hypernetwork formulation (Fig. Tb
Our main contributions are:
We characterize a previously unidentified optimization problem hypernetwork training. and show that it leads to large gradient variance and unstable training dynamics.
We propose a solution: Magnitude Invariant Parametrizations (MIP), hypernetwork for- mulation that addresses the issue without introducing additional training or inference costs.
We rigorously study the proposed parametrization. We first compare it with the standard formulation and against popular normalization strategies, showing that it consistently leads to faster convergence and more stable training. We then extensively test it using various choices of optimizer, input dimensionality, hypernetwork architecture, and activation func- tion, finding that it improves hypernetwork training in all evaluated settings.
We release our implementation as an open-source PyTorch library, HyperLight. Hyper- Light facilitates the development of hypernetwork models and provides principled choices for parametrizations and initializations, making hypernetwork adoption more accessible. We also provide code that enables using MIP seamlessly with existing models.
RELATED WORK
Parameter Initialization. Deep neural networks experience unstable training dynamics in the pres- ence of exploding vanishing gradients (Goodfellow et al., 2016). Weight initialization plays critical role in the magnitude of gradients, particularly during the early stages of training. Com- monly, weight initialization strategies focus on preserving the magnitude of activations during the forward pass and maintaining the magnitude of gradients during the backward pass (Glorot & Ben-l gio 2010 He al. 2013). In the context of hypernetworks, early work made use of Glorot and Kaiming initialization (Balažević et al.) 2019 Pawlowski et al. (017). while more recent work pro- poses initializations that accounts for the architectural properties the primary network (Beck al., 2023 Chang et al.) 2019: Knyazev et al. 2021 Zhmoginov et al. (002).. However, most of these works assume hypernetwork inputs to categorical embeddings, which limits their applicability, and makes their formulations susceptible to the proportionality issue that we identify in this work.
Normalization Techniques. Normalization techniques control the distribution of weights and acti- vations, often leading to improvements in convergence by smoothing the loss surface (Bjorck al.) 2018 Toffe 2017 Lubana et al. 2021; Santurkar et al. 2018). Batch normalization is widely used to normalize activations using minibatch statistics, and methods like layer group normalization instead normalize across features (Ba et al., 2016 Ulyanov et al., 2016;; Wu & He, 2018). Other methods reparametrize the weights using weight-normmalzzation strategies to decompose direction
2

---

Published as a conference paper at ICLR 2024
and magnitude (Qiao al., 2019: Salimans & Kingma 2016). As we show in our experiments, these strategies fail to resolve the proportionality issue we study. They either maintain the propor- tionality relationship, eliminate proportionality by rendering the predicted weights independent of the hypernetwork input, eliminating the utility of the hypernetwork itself.
Adaptive Optimization. High gradient variance can be detrimental to model convergence in stochastic gradient methods (Johnson & Zhang 2013] Roux et al.) (012). Solutions to mitigate
this issue encompass adaptive optimization techniques, which aim decouple the effect of gradient direction and magnitude by normalizing by a history of gradient magnitudes (Kingma & Ba 2014] Zeiler 2012). Similarly, applying momentum reduces the instantaneous impact of stochastic gradi ents by using parameter updates based on an exponentially decaying average of past gradients (Nes-) terov 2013; Qian 1999). These strategies are implemented by many widely-used optimizers, such as Adam (Balles& & Hennig, 2018: Kingma & Ba, 2014). We show experimentally that although adaptive optimizers like Adam enhance hypernetwork optimization, they do not address the root cause of the identified proportionality issue.
Fourier Features. High-dimensional ing (Rahimi et al. 2007) and for positional encodings in language modeling applications to account for both short and long range relationships (Su et al., 2021, Vaswani et al., 2017). Additionally, implicit neural representation models benefit from sinusoidal representations (Sitzmann al. 2020) Tancik et al. 2020). Our work also uses low dimensional Fourier projections. We demonstrate their use as means project hypernetwork inputs to vector space with constant Euclidean norm.
Fourier projections have been used in feature engineer-
Residual Forms Residual and skip connections are widely used in deep learning models and often improve model training. particularly with increasing network depth (He al.) 201 6a Li et al.) 2018 Vaswani et al. 2017). Building on this intuition, instead of the hypernetworks predicting the network parameters directly, our proposed hypernetworks predict parameter changes, mitigating part of the proportionality problem at hand.
3 THE HYPERNETWORK PROPORTIONALITY PROBLEM
Preliminaries. Deep learning tasks generally involve model [(;; 0) - y. with learnable param- eters 0. In hierarchical models using hypernetworks, the parameters 0 of the primary network f are predicted by hypernetwok (():w) + based on a input vector y. Instead of learning pa- rameters O of the primary network f. only the learnable parameters w of the hypernetwork h are optimized using backpropagation. The specific nature of the hypernetwork inputs y varies across applications, but regularly corresponds to a low dimensional quantity that models properties of the learning task, and is often a simple scalar or embedding vector (Dosovitskiy & Djolonga, 2020
Hoopes et al. 2022 Lorraine & Duvenaud 2018 Ukai et al. 2018 Wang et al. (022)).
Assumptions. For analysis, we assume the following about the hypernetwork formulation: 1) The architecture is a series of fully connected layers o(Wz æ + 6) where W are the parameters, 6 the biases and o(z) the non-linear activation function; 2) The nonlinear activation is a piece-wise linear function with single switch the origin. Namely, it satisfies 6(x) 1 [2200 (axz + 1 [e-co) (3x),
for 3 3 > 0 (e.g.. LeakyReLU) 3) Bias vectors are initialized to zero. Existing hypernetworks
satisfy these properties for the large majority of applications (Dosovitskiy & Djolongal 2020 Ha
ee al. 2016 Lorraine& & Duvenaud, 2018 MacKay et al. 2019 Ortiz et al. 2023 Ukai el al. 2018
von Oswald et al. Wang et 2021).
Input-Output Proportionality. We demonstrate that under these widely-used settings, hypernet- work inputs and outputs involve proportionality relationship, and describe how this can impede hypernetwork training. We show that 1) at initialization, any intermediate feature vector .(()) at layer k will be proportional to hypernetwork input 7, even under the presence of non-linear activa- tion functions, and 2) this leads large gradient magnitude fluctuations detrimental to optimization.
We first consider the case where E R is a scalar value. Let h((7; use a fully connected architec- ture composed of series of fully connected layers
M(x:w) W(n) + ((n)
(*+1) ((W(()) (1k) + 6(3))
(1)
x (1) =7
3

---

Published as a conference paper at ICLR 2024
where z²(A) is the input vector of the 1thh fully connected layer with learnable parameters ((() and biases ()(). To prevent gradients from exploding or vanishing when chaining several layers, it is common to initialize the parameters W(()) and biases 6(0) so that either the magnitude of the activa- tions is approximately constant across layers in the forward pass (known as fan in), or so that the magnitude of the gradients is constant across layers in the backward pass (known as fan out) (Glorot & BBngiio 2010, He et al., 2015). In both settings, the parameters W are initialized using a zero mean Normal distribution and bias vectors 860) are initialized to zero. If y > 0, and (xx) has the common form specified above, at initialization the 7th entry of vector x=2) is
ow!"" + ((0) > x
since 6(0) and (W is independent of 7. Using induction, we assume that for layer h, (N) x y Vjj, and show this property for layer k + 1. The value of the 1th element of vector (x+++1) is
(k+1) ø (b,t + +E, W (k) 2) (0 = 7 (E, W (k)
(R) aj %
(3)
since 6,(0) = 0, and the term inside & is independent of y. If y is not strictly positive, we can reach the same proportionality result, but with separate constants for the positive and the negative range. This dependency holds regardless of the number of layers and the number of neurons per hidden layer, and also holds when residual connections are employed. When 7 is vector input, find similar relationship with the overall magnitude of the input and the magnitude of the output. Given the absence of bias terms, and the lack of multiplicative interactions in the architecture, the fully connected network propagates magnitude changes in the input.
Training implications. Since 0 = x((+1), this result leads to a proportionality relationship for the magnitude of the predicted parameters ||0||2 || and their variance Var(o) x |///|22 As the scale of the primary network parameters 0 will depend on y, this will affect the scale of the layer outputs and gradients of the primary network. In turn, these large gradient magnitude fluctuations lead to unstable training dynamics for stochastic gradient descent methods (Glorot & Bengio 2010).
Further Considerations. Our analysis relies on biases being at zero, which only holds initial- ization, and does not include normalization layers that are sometimes used. However, in our experi- ments, we find that biases remain near zero during early training, and hypernetworks with alternative choices of activation function, input dimensionality, or with normalization layers, still suffer from the identified issue and consistently benefit from our proposed parametrization (see Section 6
MAGNITUDE INVARIANT PARAMETRIZATIONS
To address the proportionality dependency, we make straightforward changes to the typical hypemnetorr formulation: 1) We introduce an encoding function that maps inputs into a constant- norm vector space, and 2) we treat hypernetwork predictions as additive changes to the main net- work parameters, rather than as the parameters themselves. These changes make the primary net- work weight distribution non- non- conp-rooortionll to the hypernetwokk input and stable across the range of hypernetwork inputs. Figure illustrates these changes to the hypernetwork.
Input Encoding. To address the proportionality problem, we map the inputs 7 E [0 to space with a constant Fuclidean norm ||Fa((7)l2 (cos(77722) sin(yn/22) /2)]. With this change, the input magnitude to the hypernetwork is constant, so ||x of y. For higher-dimensional inputs, we apply this transformation to each input individu- ally, leading to an output vector with double the number of dimensions. This transformation results in an input representation with a constant norm, thereby eliminating the proportionality effect.
1 using the function FLa(7)
For our input encoding, we first map each dimension of the input vector to the range [0, 1] to maxi- mize output range of E12. We use min-max scaling of the input: 7' = (? "/min)/ /7mmax - 7min). For unconstrained inputs, such as Gaussian variables, we first apply the logistic function o (x) 1/(1 + exp(-2)) ). If inputs span several orders of magnitude, we take the log before the min-max
scaling in (Bae et al. 2022 Dosovitskiy & Djolonga 2020).
Output Encoding, Residual forms have become a comerstone in contemporary deep learning ar- chitectures (He et al. 2016a Li al. 2018 Vaswani et al. 2017). Motivated by these methods,
4

---

Published as a conference paper at ICLR 2024
B12
M(sww)
Primary net work
Reshape
cuel
Dense Pense+ReLD
D
Dense
Reshape
concat
f(: 0)
6in(
Dense
Reshape
Figure 2: Magnitude Invariant Parametrizations for Hypernetworks. MIP first projects the hypernetwork inputs y to constant norm vector space. Then the outputs of the hypernetorr A are treated as additive changes to a set of independent learnable parameters 0 to generate the primary network weights 9. In blue highlight the main components of MIP, the input encoding Eg2 and the residual formulation 8 = + AO.
replace the standard hypernetwork framework with one that learns both primary network f parame- ters (as is typically learned in existing formulations) and hypernetwork predictions, which are used as additive changes these primary parameters. We introduce set of learnable parameters 6 and compute the primary network parameters as 0 = 0 + h(E))(y)):w.
Parameter Initialization We initialize the hypernetwork weights w using common initialization methods for fully connected layers that consider the number of input and output neurons to each layer, such as Kaiming or Glorot initialization. Then, we initialize the independent parameters ( in the same manner that we would initialize the parameters of an equivalent regular network. We provide further details and examples section A of the supplement.
EXPERIMENTAL SETUP
5.1
TASKS
We evaluate our proposed parametrization on several tasks involving hypernetwork- models.
Bayesian Neural Networks. Hypernetwork models have been used learn families of func- tions conditioned a prior distribution (Ukai et al.) 2018). During training, the prior representa- tion y E Rd sampled from the prior distribution ? ~ p(y) and used to condition the hypernetwork h(ri)) + 0 to predict the parameters of the primary network model fz; 0). Once trained, the family of posterior networks is then used to estimate parameter uncertainty or to improve model cal- ibration. For illustrative purposes network used to classify the MNIST dataset. Then, we tackle more complex setting where / (x; 0) is ResNet-like model trained the OxfordFlowers-100 dataset (Nilsback & Zisserman 2006). In both settings, we use the prior (0, 1) for each input.
first evaluate setting where f(: f) is a feed-forward neural
Hypermorph. Learning-bai based medical image registration networks f( Ty: + register moving image Im to fixed image y by predicting a flow or deformation field Φ between them. The common (unsupervised) loss balances an image alignment term Lsim and a spatial regularization Lim Φ, Yreg (o),
(smoothness) term L The learning objective is then L where y controls the off In Hypermorph (Hoopes et al. (022). . multiple regularization settings for medical image registration are learned jointly using hypernetworks. The hyperneworr is given the trade-off parameter 7 as input, sampled stochastically from (1 1) during training. We follow the same experimental setup, using U-Net architecture for the primary (registration) network and training with MSE for Lsim and total variation for Creg- We train models on the OASIS dataset. For evaluation, moving image, and measure the volume overlap to the fixed label maps (Balakrishnan et al. 2019).
the predicted flow field to warp anatomical segmentation label maps of the
Scale-Space Hypernetworks. We also with varying internal rescaling factors in the downsampling and upsampling layers, as done in Ortiz et al. 2023). In this setting, 7 corresponds to the scale factor. Given hypemetwork input 7, the
hypemeework to efficiently learn family of models
5

---

Published as a conference paper at ICLR 2024
(a) Primary Network Parameters
(b) Primary Network Activations
(c) Primary Network Gradients
Derbult Hypemetwork
Defsuttyyemeetoo
16
Default
14
MIP (ours)
Gradlent Magnitude [|9,C||
0.2
0.4
0.4
12
Dercky
MiP Hyypeneework
Noralzac Danikky
HIP ttypemetoonk
10
1.0
1.0
-02
0.2
Layer Output
Epoch
Figure 3: Distributions of primary network parameters (a) and layer activations (b). Mcasure- ments are taken at initialization for a default hy hypernetwork, our proposed MIP hypernetwork, and a conventional neural network with the same primary architecture. Distributions density estimates (KDE) of the values because the high degree of overlap between the distribu- tions. The MIP strategy leads to little change across input values and its distribution closely matches that of the non-hyyernetwork model. Evolution of Gradients (c) Gradient magnitude with respect hypernetork outputs ||V during early training. Standard deviation is computed across mini- batches in the same epoch. MIP leads to substantially smaller magnitude and standard deviation compared to the default parametrization.
shown kernel
hypernetwork h(r:w) + predicts the parameters of the primary network, which performs the spatial rescaling operations according to the value of y. We study setting where f(; 0) is convolutional network with variable resizing layers, the rescaling factor is sampled from (0, 0.5), and evaluate using the OxfordFlowers- 102 classification problem and the OASIS segmentation task.
5.2 EXPERIMENT DETAILS
Model. We implement the hypemnttwrk as neural network with fully connected layers and LeakyReLU activations for all but the last layer, which has linear output. Hypernetwork weights are initialized using Kaiming initialization on fan out mode and biases are initialized to zero. Unless specified otherwise, the hypernetwork architecture has hidden layers with 16 and 128 neurons respectively. We use this implementation for both the default (existing) hypernetworks, and our proposed (MIP) hypernetworks.
Training. We use two popular choices of optimizer: SGD with Nesterov momentum, and Adam. We search over range of initial learning rates and report the best performing models; further details are included in section B of the supplement.
Implementation. An important contribution of our work HyperLight, our PyTorch hypernetwork framework. HyperLight implements the proposed hypernetwork parametrization, but also provides modular and composable API that facilitates the development of hypernetork models. Using flyperl.ght, practitioners can employ existing non-hypernetwork model definitions and pretrained model weights, and can easily build models using hierarchical hypernetworks. Anonymized source code is available at litttps:: httpbb.ttuuuuuug httpp//cc......ooeeeccacaaeeaa/a
EXPERIMENTAL RESULTS
6.1 EFFECT OF PROPORTIONALITY ON PARAMETER AND GRADIENT DISTRIBUTIONS
First, we empirically show how the proportionality phenomenon affects the distribution predicted weights 0 and their corresponding gradients for the Bayesian neural networks MNIST. Figures 3a

---

Published as a conference paper at ICLR 2024
and 3b compare the distributions of the primary network weights and layer outputs for a range of values of hypernetwork input 7. The default hypernetwork parametrization is highly sensitive to changes in the imput, in contrast, MIP eliminates this dependency, with the resulting distribution closely matching that of the non-hhyernetwork models. Figure [Ta (in the introduction), shows that using the default formulation, the scale of the weights correlates linearly with the value of the hy- pernetwork input, and that, crucially, this correlation is still present after the training process ends. In contrast, MIP parametrizations lead weight distribution that is robust the input y, both at the start and end of training.
We also analyze how the proportionality affects the early phase of hypernetwrrk optimization by studying the distribution of gradient norms during training. Figure shows the norm of the pre- dicted parameter gradients V as training progresses. Consistent with our analysis, hypernet- works with default parametrization experience large swings in gradient magnitude because of the proportionality relationship between inputs and predicted parameters. In contrast, the MIP strategy leads to a substantially smaller variance and more stable gradient magnitude.
6.2 MODEL TRAINING IMPROVEMENTS
In this experiment, we analyze how MIP affects model convergence for the considered tasks. For all experiments, we found that MIP hypernetworks did not introduce a measurable impact in training runtime, report per-epoch steps.
Figure Ib (in the introduction) shows the training loss for Bayesian networks trained on MNIST. We find that MIP parametrizations result in smaller loss sooner during training, and the default parametrization suffers from sporadic training instabilities (spikes in the training loss), while MIP leads to stable training. Similarly, Figure 4a shows the test accuracy for Bayesian networks trained on OxfordFlowers. In this task, MIP also achieves faster convergence and better final model accu- racy for both choices of optimizer.
Figures 4b and present convergence curves for the other two tasks. For Hypermorph, MIP parametrizations are crucial when using SGD with momentum since otherwise the model fails to meaningfully train. For all choices of learning rate the default hypernetwork failed to converge, whereas with MIP parametrization it converged for large range of values. With Adam, networks train meaningfully, and MIP models consistently achieve similar Dice scores substantially faster. They are less sensitive to weight initializations. Though the Adam optimizer partially mitigates the gradient variance issue by normalizing by history of previous gradients, the MIP parametriza- tion leads to substantially faster convergence. Furthermore, for the Scale-Space segmentation, we find that for both optimizers MIP models achieve substantially faster convergence and better final accuracy compared to those with the default parametrization.
Comparison to normalization strategies. We compare the proposed parametrization to popular choices of normalization layers found in the deep learning literature. Using the default formulation, where the predicted weights start proportional to the hypernetwork input, we found that existing nor- malization strategies fall into two categories: they either keep the proportionality relationship present (such batch normalization), remove the proportionality by making the predicted weights inde- pendent of the hypernetwork input (such as layer or weight normalization). We provide further details in Section the supplemental material.
We test several of these normalization strategies. BatchNorm-P, adds batch normalization layers the primary network. LayerNorm-P, adds feature normalization layers to the primary network. LayerNorm-Bl adds feature normalization layers to the hypernetwork layers. WeightNorm, per- forms weight normalization, which decouples the gradient magnitude and direction. to weights pre- dicted by the hypernetwork (Ba et al., (2166) Loffe 2017] [Salimans & Kingmal 2016). Figure Sa shows the evolution of the test accuracy for the Scale-Spar Space hypernetworks trained on OxfordFlow- ers. We report wall clock time, since some normalization strategies, such as BatchNorm, substan- tially increase the computation time required per iteration. For networks trained with SGD, these normalization strategies enable training, but do not significantly improve on default hypernetworks when trained with Adam. Models trained with SGD momentum and hypernetwork feature nor- malization (LayerNorm-H) diverged early into training for all considered hyperparameter settings. Models trained with the proposed MIP parametrization lead to substantially faster convergence and better final model accuracy.
7

---

Published as a conference paper at ICLR 2024
(a) Bayesian Networks (Flowers)
(b) HyperMorph
(c) Scale-Spart (OASIS)
SG.D Momenntum
SCD Momenkum
Homertum
0.70
0.05
Datt
0.4
0.2
0.2
0.55
0.0
Acarao)
0.6
Dice Score
0.70
0.6
0.4
0.4
Hypernet Defout
Hypemet Defeuit
Hypemet
0.-
0.2
Defautt
HIP tourst
MP (ours)
MP (ours)
0.0
0.0
1000
2000
3000
4000
5000
500
1000
1500 Epoch
2003
2500
3000
200
800
1000
Epoch
Epoch
Figure 4: Models Convergence Improvements. Comparison between default hypemetworkss and hypernetworks with MIP for the Bayesian networks OxfordFlowers- 102 (a), HyperMorph (b) and Scale-Space hypernetworks trained on OASIS (c). In all cases, we find that the MIP parametrization leads faster model convergence without any sacrifice in final model accuracy compared to the default parametrization. In all cases we observe that MIP improves convergence and final model accuracy. We also find that for default hypernetworks using the Adam optimizer substantially helps the training process, however, incorporating MIP leads to even better training dynamics.
Initialization Schemes. We compare MIP and the default hypernetworks to hypernetworks that use the Hyperfan-in and Hyperfan-out initialization strategies from Chang et al. (2019). The Hy- perfan initialization takes into account the hypernetwork and primary network architectures when initializing model weights, improving model convergence and training stability. However, Hyperfan initializations are not designed for magnitude-eneoot inputs, so they are susceptible to the propor- tionality issue we identify.
Figure 5b presents convergence results for the Hypermorph task with SGD and Adam. We find that Hyperfan initializations do not resolve the training challenges when using SGD. For hypernetworks trained with Adam, MIP outperforms both Hyperfan variants.
Ablation Analysis. We study the contribution of each of the two main components of the MIP parametrizations: input encoding and additive output formulation. Figure 5c shows the effect on convergence for two tasks. We found that both components reduce the proportionality dependency between the hypernetwork inputs and outputs, and that each component independently achieves substantial improvements in model convergence. However, find that best results (fastest conver- gence) are consistently achieved when both components are used jointly during training.
6.3 ROBUSTNESS ANALYSIS
Hypernetwork Input Dimensionality. We study the effect of the number of dimensions of the input to the hypernetwork model. We evaluate on the Bayesian neural network task, and we vary the number of dimensions of the input prior. We train models with geometrically increasing number of input dimensions, dim(7) = 2,.,..., 32. Figure (in section C of the supplement) shows that the proposed MIP strategy leads to improvements model convergence and final model accuracy we increase the dimension of the hypernetwork input 7.
Choice of Hypernetwork Architecture. We assess model performance when varying the properties of the hypernetwork architecture. We vary the width (number of hidden neurons per layer) and depth (number of layers)- fully connected networks with 3, 4 and 5 layers and with 16 and 128 neurons per layer, well Figures|] 7 and g (in section C.2 of the supplement) show that the MIP improvements generalize to the all tested hypernetwork architectures with analogous improvements in model training.
exponentially growing number of neurons per layer Dim(z") 16 2".
8

---

Published as a conference paper at ICLR 2024
(a) Normalization Techniques
(b) Initialization Scheme
MIP Ablation
Nomentum
Momentum
Bayesian Neunal Networks (SGD)
0.8
Accuraay
0.6
That: Dice Score
0.70
Accuracy
0.2
0.0
0.55
Adom
Space Hypornetwork: (Adam)
0.0
0.72
AoL-ccuracy
0.6
0.6
Initalization
Hypernet
0.4
Delsult
0.4
Delaut
0.66
MIP tourst
HIP (both)
0.64
Hypertennin
0.2
HIP only
onl)
0.0
300
300
400
500
1000
25:00
2000
2503
3000
200
400
600
800
1000
Training Time [min]
Epoch
Epoch
Figure 5: (a) Normalization Strategies. Comparison of hypernetworks trained with various nor- malization strategies for Scale-Space hypernetworks trained of OxfordFlowers-IIn 102. MIP provides substantially better results than the considered normalization strategies, achieving faster model con- vergence and better final test accuracy. (b) Initialization Scheme. Comparison of hypernetworks with the default, MIP, Hyperfan-In and Hyperfan-Out initialization schemes on the Hypermorph task. Models with MIP train substantially better than models with Hyperfan initializations, espe- (c) Ablation Analysis. Convergence results for separate components of MIP on the Scale-Space Hypernetworks on OASIS using the Adam optimizer, and on the Bayesian networks trained with SGD on the OxfordFlowers 102 classification problem. Each component of the parametrization leads to improvements final model accuracy as well as training convergence, and best results achieved when using both components simultancously.
cially when using the SGD optimizer.
Choice of Nonlinear Function Activation. While our method motivated by the training in- stability present in hypernetworks with (Leaky)-ReLU nonlinear activation functions, explored applying it to other common choices of activation functions found in the literature: Tanh, GELU and SiLU (Hendrycks & Gimpel| 2016) [Ramachandran al. (017). Figure (in section C.3 of the supplement) shows that MIP consistently helps for all choices of nonlinear activation function, and the improvements are similar to those of the LeakyReLU models.
LIMITATIONS
All hypernetwork models used in our experiments are composed of fully connected layers and use activation and initialization choices commonly recommended in the literature. Similarly, we focused on two optimizers in our experiments, SGD with momentum and Adam. We believe that we would see similar results for other less common architectures and optimizers, but this remains to be inves- tigated. Furthermore, we focus on training models from scratch. As hypemetworks become popular transfer learning, we believe this will be an interesting avenue for future analysis of MIP.
CONCLUSION
We showed through analysis and experimentation that traditional hypernetwork formulations are susceptible to training instability, caused by the effect of the magnitude of hypernetwork input val- ues on primary network weights and gradients, and that standard methods such as batch and layer normalization do not solve the problem. We then proposed the use new method, Magnitude Invariant Parametrizations (MIP), for addressing this problem. Through extensive experiments, demonstrated that MIP leads to substantial improvements in convergence times and model accuracy across multiple hypernetwork architectures, training scenarios, and tasks. Given that using MIP never reduces model performance and can dramatically improve training, we expect the method to be widely useful for training hypernetworks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This research is supported by Quanta Computer, Inc, and Wistron Coporation. Additionally, the project was supported by NIH AG064022 and AG070998
REFERENCES
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inver- sion with hypernetworss for real image editing. In Proceedings of the IEEE/CVF conference computer Vision and pattern recognition, pp. 18511-18521, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016.
Juhan Bae, Michael R Zhang, Michael Ruan, Eric Wang, So Hasegawa, Jimmy Ba, and Roger Grosse. Multi-rate vae: Train once, get full rate-distortion curve. arXiv preprint arXiv:2212.03905, 2022.
Guha Balakrishnan, Amy Zhao, Mert R Sabuneu, John Guttag, and Adrian V Dalca. Voxelmorph: learning framework for deformable medical image registration. IEEE transactions on medical imaging, 38(8): 1788-1108, 2019.
Ivana Balažević, Carl Allen, and Timothy M Hospedales. Hypernetwork knowledge graph embed- dings. In International Conference on Artificial Neural Networks, pp. 553-565 133-565. Springer, 2019.
Lukas Balles and Philipp Hennig. Dissecting adam: The sign, magnitude and variance stochastic gradients. In International Conference on Machine Learning, pp. 404-413. PMLR, 2018.
Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypemetworks in meta-reinffrremmnt learning. In Conference on Robot Learning, pp. 1478-1487 PMLR, 2023.
Raphael Bensadoun, Shir Gur, Tomer Galanti, and Lior Wolf. Meta internal learning. Advances in Neural Information Processing Systems, 34:20645-20656, 2021.
Nils Bjorek, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. Advances in neural information processing systems, 31, 2018.
Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2019.
Lee Dice. Measures of the amount of ecologic association between species. Ecology, 26(3): 297-302, 1945.
Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. In International conference on learning representations, 2020.
Benjamin Ehret, Christian Henning, Maria R. Cervera, Alexander Meulemans, Johannes von Os- wald, and Benjamin Grewe. Continual learning in recurrent neural networks. In International Conference on Learning Representations, 2021. URL htttps://arxiii www.xiii.ggg/..eeeoo abs abaaabbs///00 /2006. 12109
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv:1609.00 1609.09106, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1034-1034, 2015.
10

---

Published as a conference paper at ICLR 2024
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In European conference computer vision, pp. 630-645. Springer, 201
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606. 08415, 2016.
Andrew Hoopes, Malte Hoffman, Douglas N. Greve, Bruce Fischl, John Guttag, and Adrian V. Dalca. Learning the effect of registration hyperparameters with hypermorph. Machine Learning for Biomedical Imaging, 1:1-30, 2022. ISSN 2766-905X. URL http:: :/me lomelba-jouue journall..
org/ 2022: 003
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch-normalized models. arXiv preprint arXiv:1702.03277 2017.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26:315-323, 2013.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 112.6994,,,6980, 2014.
Boris Knyazev, Michal Drozdzal, Graham Taylor, and Adriana Romero Soriano. Parameter prediction for unseen deep architectures. Advances in Neural Information Processing Systems, 34:29433-29444 2021.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: 2017.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances neural information processing systems, 31, 2018.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernet- works. arXiv preprint 180Iv.1802.0000 80419949, 2018.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. CoRR, abs/1711.05101, 2017. URL
Ekdeep S Lubana, Robert Dick, and Hidenori Tanaka. Beyond batchnorm: Towards a unified under- standing of normalization in deep learning. Advances in Neural Information Processing Systems, 14:4778-47911 2021.
Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning net- works: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint 19X3v:1903.0000 2019.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. arXiv preprint arXiv:2106.04489 2021.
Daniel S Marcus, Tracy H Wang, Jamie Parker, John G Csermansky, John C Morris, and Randy Buckner. Open access series of imaging studies (oasis): cross-sectional mri data in young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience, 1498 2007.
Yurii Nesterov. Introductory lectures on conver optimization: A basic course, volume 87, Springer Science &amp; Business Media, 2013.
M-E Nilsback and Andrew Zisserman. A visual vocabulary for flower classification. In 2006 IEEE Computer Sociery Conference on Computer Vision and Pattern Recognition (CVPR' (06), volume 2, pp. 1447-1454. IEFF, 2006.
11

---

Published as a conference paper at ICLR 2024
Jose Javier Gonzalez Ortiz, John Guttag, and Adrian V. Dalca. Amortized learning of dynamic feature scaling for image segmentation. arXiv preprint arXiv:2304.05444, 2023.
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1711.0129) 2017.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 (1):145-151, 1999.
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Micro-batch training with batch- channel normalization and weight standardization. arXiv preprint arXiv: 10520, 2019.
Ali Rahimi, Benjamin Recht, al. Random features for large-scale kernel machines. In NIPS, volume 3, pp. 5. Citeseer, 2007.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv:1710.059411 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 231. Springer, 2015.
Nicolas Le Roux, Mark Schmidt, and Francis Bach. A stochastic gradient method with an exponen- tial convergence rate for finite training sets. arXiv preprint arXiv: 220.62288, 2012.
Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterizatioo accelerate training of deep neural networks. Advances neural information processing systems, 29:901- 909, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Iyas, and Aleksander Mądry. How does batch nor-
malization help optimization? In Proceedings of the 32nd international conference neural information processing systems, pp. 2488-2498, 2018.
Joan Serrà, Santiago Pascual, and Carlos Segura. Blow: single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. arXiv preprint arXiv: 1906.00999, 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. arXiv preprint arXiv:2104.0988 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kad Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv:2006. 10739, 2020.
arXiv preprint
Kenya Ukai, Takashi Matsubara, and Kuniaki Uehara. Hypemetwork- implicit posterior es- timation and model averaging of cnn. In Asian Conference on Machine Learning, pp. 176-191. PMLR, 2018.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv:1607.08022. 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2024
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
URL https:s/aaxxx....rg////.o abs/ 11066 00695
Alan Q Wang, Adrian V Dalca, and Mert R Sabuneu. Regularizaation-agnosii compressed sensing mri reconstruction with hypernetworks. arXiv preprint arXiv:2101.02194, 2021.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
Matthew D Zeiler. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012.
Lymin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
Dominic Zhao, Johannes von Oswald, Seijin Kobayashi, João Sacramento, and Benjamin F Grewe. Meta-learning via hypernetworks. 2020.
Andrey Zhmoginov, Mark Sandler, and Maksym Vladymyrov. Hypertransformer: Model generation for supervised and superviiee few-shot learning. In International Conference Machine Learning. pp.
PMLR, 2022.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A MIP PARAMETER INITIALIZATION
Parameter initialization strategies for regular neural networks have remained fairly stable over the past half decade. The prevalent method consists of using Glorot or He initializations that are de- signed to preserve the magnitude of activations during the forward pass and maintain the magnitude of gradients during the backward pass (Glorot & Bengio 2010: He al. 2015).
Typically, the assumptions of these initialization schemes do not hold when applied to hypernet- work settings. This has prompted the development of several specialized initialization schemes tailored for hypernetwork formulations (Beck et al. 2023 Chang et al. 2019 Knyazev et al. (021)) Crucially, these techniques require incorporating the knowledge of the primary network when per- forming the initialization, and are designed for categorical inputs represented as embedding vectors. The guarantees these schemes provide do not hold when using magninude-encoded inputs such as
scalars.
We propose a simple yet effective initialization scheme based on the recommendations from the neu- ral network literature. First, the hypernetwork weights w are initialized using common initialization methods for fully connected layers. Then, the independent parameters 8 are initialized taking into consideration their role in the primary network.
We illustrate our initialization rules using examples with the Kaiming He fan-out scheme. Using He fan-out init, the weights and biases of neural network layer with Rin input neurons and naut output neurons are sampled
(o.-
G
W W~W' .
where G is the relative gain of the non-linearity function ((x). For ReLU, we have G = 2 whereas linear or sigmoid, we have G 1.
We differentiate the following cases:
Intermediate Hypernetwork Layer- We initialize a (Ni. Row) layer in the hypernetwork following the scheme we just outlined, i.e., W (O G/ (GGG/////ooo (Doa) and 0.
Final Hypernetwork Layer We consider two cases, but do consider biases be- cause they are redundant with 0g.
1. Layer predicting primary network weight of shape (nin- neut):
W N (o. v Thin
(5)
Layer predicting primary network bias: W
Independent Weights 0g For fully connected layer with (7 now) neurons, we initialize W ~ NO, (c/Too)))) and b = O.
From this initialization, were the weights of a regular neural network. Alternatively, if the primary network corresponds to a pretrained model, the independent weights 6g are initialized using the pretrained values, and can be optionally frozen during training.
observe that the set of independent weights 0g is initialized as if they
A.1 IMPLEMENTATION CONSIDERATIONS
Since the Og weights are redundant with the bias parameters of the final hypernetoor layer, we remove bias parameters from the final hypernetoork layer. In implementation, final layer, but we initialize its weights as if it were the multiple smaller layers, since otherwise the initialization would not follow the recommendations outlined in the previous section.
single
Under some hypemneworr configurations, all the primary network parameters are predicted a single forward pass of the hypernetwork. In this scenario, implement the parameters 0 as the
14

---

Published as a conference paper at ICLR 2024
bias vector terms of the last layer B(m), which proves to be as efficient as the default formulation. This is correct because we do not have ((a) in our formulation, and ((0) is equivalent to 0 in the computational graph, receiving the same gradients. Hence, we initialize 6(^) as a one-dimensional representation of the primary network parameters, subsequently reshaping construct B.
B ADDITIONAL EXPERIMENTAL DETAILS
B.1 DATASETS
MNIST. We train models on the MNIST digit classification task. We use the official MNIST database of handwritten digits. The MNIST database of handwritten digits comprises training set of 60,000 examples, and a test set of 10,000 examples. We use the official train-test split for training data, and further divide the training split into training and validation using a stratified 80%- 20% split. We use the digit labels and consider the 10-way classification problem.
OxfordFlowers-102.. We the OxfordFlowers-10 dataset, fine-grained vision classification dataset with 8.189 examples from 102 flower categories (Nilsback & Zisserman (206)). We utilize this dataset as it poses non-trivial learning task that does not quickly converge, and allows us to better study learning dynamics. We use the official train-test split for training data, and further divide the training split into training and validation using a stratified 80%-20% split. We perform data augmentation by considering random square crops of between 25% and 100% of the original image area and resizing images to 256 by 256 pixels. Additionally, we perform random horizontal flips and color jitter (brightness 25%, contrast 50%, saturation 50%). For evaluation we take the central square crop of each image and resize to 256 by 256 pixels.
OASIS We use version of the open-access OASIS Brains dataset (Hoopes et al. 2022 Marcus ct al. 2007), a medical imaging dataset containing 414 MRI scans from separate individuals, com- prised of skull-stripped and bias-corrected images that resampled into affinely- alligne,, com- mon template space. For each scan, segmentation labels for 24 brain substructures a 2D coronal slice are available. We use 64%, 16% and 20% splits for training, validation and test.
B.2 BAYESIAN NEURAL NETWORKS
Primary Network. For the MNIST task, we use LeNet architecture variant that uses ReLU acti- vations they have become more prevalent in modern deep learning models. Moreover, replace the first fully-connected layer with two convolutional layers of 32 and 64 features. We found this change did impact test accuracy in non-hyperntwwork models, but lead to more stable initial- izations for the default hypernetworks.
For the OxfordFlowers- 102 task, the primary network f features a ResNet-like architecture with five downsampling stages with (16, 32, 64, 128, 128) feature channels respectively. For experiments including normalization layers, such as BatchNorm and LayerNorm, the learnable affine parameters of the normalization layers are not predicted by the hypernetworks and are optimized like in regular neural networks via backpropagation.
Training. We train using categorical cross entropy loss. For both optimizers we use learning rate 7) 3 10-4 Nevertheless, we found consistent results with the ones we report using learning rates in the range 17 [10-
3). We sample y from the uniform distribution [/ 1].
Evaluation. For evaluation we use top-1 accuracy on the classification labels. In order to get a more fine-grained evolution of the test accuracy, evaluate on test set at 0.25 epoch increments during training. We report results with five model replicas with different random seeds.
B.3 HYPERMORPII
HyperMorph, learning based strategy for deformable image registration learns models with differ- ent loss functions in an amortized manner. In image registration, the y hypernetwork input controls the trade-off between the reconstruction and regularization terms of the loss.
Primary Network. For our primary network f we use a U-Net architecture (Ronneberger et al.] 2015) with convolutional encoder with five downsampling stages with two convolutional layers
15

---

Published as a conference paper at ICLR 2024
per stage of 32 channels each. Similarly, the convolutional decoder is composed of four stages with two convolutional layers per stage of 32 channels each. We found that models with more convolutional filters performed no better than the described architecture.
Training. We train using the setup described HyperMorph (Hoopes et al. 2022) using mean squared error for the reconstruction loss and total variation for the regularization of the predicted flow field. For the Adam optimizer we use 31 = 0.9 and G2 = 0.999 with decoupled de- cay Loshchilov & Hutter (2017) and 1j = 10-4 but found that learning rates [10-4, 10-3] lead to similar convergence results. For SGD with momentum, we tested learning rates y) {3 10 10-2 10-2 3 10-3 10-3 3, 3 x 10- 10-4, 10 10 }. In all cases the default hyper- network formulation failed to meaningfully train. We train for 3000 epochs, and sample ry uniformly in the range [0, 1] like the original work.
Evaluation. Like Hoopes et al. 2022), we use segmentation labels as the main means of evaluation and use the predicted flow field warp the segmentation label maps and measure the overlap to the ground truth using the Dice score (Dice) 1945), a popular metric for measuring segmentation quality. Dice score quantifies the overlap between two regions, with a score of I indicating perfect overlap and 0 indicating no overlap. For multiple segmentation labels, we compute the overall Dice coefficient as the average of Dice coefficients for each label. We report results with five model replicas with different random seeds.
B.4 SCALE-SPACE HYPERNETWORKS
We evaluate on a task where the hypemnetorr input " controls architectural properties of the pri- mary network. We use y to determine the amount of downsampling in the pooling layers. Instead of using pooling layers that rescale by fixed factor of two, we replace these operations by fractional bilinear sampling operation that rescales the input by a factor of
Primary Network. For classification tasks, our primary network f features a ResNet-like archi- tecture with five downsampling stages with (16, 32, 64, 128, 128) feature channels respectively. For experiments including normalization layers, such BatchNorm and LayerNorm, the learnable affine parameters of the normalization layers are not predicted by the hypernetworks and are opti- mized like in regular neural networks via backpropagation.
For segmentation tasks, we model the primary network f using a U-Net architecture [Ronneberger cet al. 2015) with a convolutional encoder with five downsampling stages with two convolutional layers per stage of 32 channels each. Similarly, the convolutional decoder composed of four stages with two convolutional layers per stage of 32 channels each.
Training. We sample the hypernetwork input y uniformly in the range [0, 0.5] where 0.5 corresponds to downsampling by We train the multi-class classification task using categorical cross-entropy loss, and train with a weight decay factor of 10-3, and with label smoothing Good- fellow et al. (2016); Szegedy al. (2016) the ground truth labels with uniform distribution of amplitude e = 0.I. For the segmentation tasks we train using a cross-entropy loss and then fine- tune using a soft-Dice loss term, as Ortiz et al. (2233. For both optimizers we use learning rate 17 = 1 x 10 -4. Nevertheless, we found consistent results with the ones we report using learning rates the range 17 = 10 10-31
16

---

Published as a conference paper at ICLR 2024
C ADDITIONAL EXPERIMENTAL RESULTS
C.I NUMBER OF INPUT DIMENSIONS
In this experiment, we study the effect of the number of dimensions of the imput to the hypernetwork model on the hypernetwork training process, both for the default parametrization and for our MIP parametrization. We evaluate using the Bayesian Hypernetworks, since we can vary the number of dimensions of the input prior without having to define new tasks. We train models with geometrically increasing number of input dimensions, dim(7)
.2......................... We apply the input encoding to each dimension independently. We study two types of input distribution: uniform (0, 1) and Gaussian (N 1). For MIP, we apply a sigmoid to the Gaussian inputs to constrain them to the [0.1] range as specified by our method. We evaluate the Bayesian hypernetworks task on the OxfordFlowers- 102 dataset with a primary convolutional network optimized with Adam.
1, 2, .................... .3.
Figure 6 shows the convergence curves during training. Results indicate that the proposed MIP parametrization leads to improvements in model convergence and final model accuracy for all num- ber of input dimensions the hypernetwork and for both choices of input distribution. Moreover, we observe that the gap between MIP and the default parametrization does not diminish as the number of input dimensions grows.
(a) Uniform Inputs (I's U(0, 1))
miyt
miyt 2
may
d
dnlyt- 26
5nly
Mopraned bebut
15-
2203
8202
vods Cooch
220 Szech
220 frech
50-00
fpach
Ezech dtrey) 4
Szeen stmiy) 3
2
dmivi
dmy
32
3.44
bebut MFfocud)
23.2
+2:33
12.0
J82z Eoech
4543
226
2823 Fpach
62.0
.591 Sooch
Ezech
(b) Gaussian Inputs (T; = NO 1))
Hypemet
1093
4902
z058
2000 direyt 4
orea
200e Erech
orea
2000
aree
2000
30/0
1poch
Umitt=
denivi -
driyl 52
Dirut
MPiouro
2026
100.3
51ech
C1001
Epech
Epoch
Figure 6: Number of dimensions of hypernetwork input. Test loss (top row) and test accuracy (bottom row) for Bayesian hypernetworks trained on the OxfordFlowers classification task for in- creasing number of dimensions of the hypemetwork input 7 We report results for different prior input distributions: Uniform (a) and Gaussian (b). For each setting, we train 3 independent repli- cas with different random initialization and report the mean (solid line) and the standard deviation (shaded region). We see significant improvements in model training convergence when the hyper- network uses the proposed MIP parametrization.
17

---

Published as a conference paper at ICLR 2024
C.2 CHOICE OF HYPERNETWORK ARCHITECTURE
In this experiment we test whether increasing the choice of hypernetwork architecture size has an effect on the improvements achieved by incorporating Magnitude Invariant Parametrizations (MIP). We study varying the width (the number of neurons per hidden layer) and the depth (the number of hidden layers) independently as well jointly. For the depth, we consider networks with and 5 layers. For width, we consider having 16 neurons per layer, 128 neurons per layer, or having an exponentially growing number of neurons per layer (exp), following the expression Dim(x") 16 2".
We compare training networks using the default hypernetwork parametrization and MIP for the HyperMorph task. Figure shoow convergence curves for the evaluated settings, for several random initializations. Additionally, Figure g shows the distribution final model performances the range of inputs y [0, 1]. We find that MIP models converge faster without sacrificing final model accuracy.
shows
Width 16
Width 128
Width exp
0.74
Val Dice Score
0.72
Segth =3
0.70
0.68
0.74
Val Dice Score
Depth
0.72
Hypernet
Default
0.70
MIP (ours)
0.68
0.74
Val Dice Score
0.72
0.70
0.68
0
1000
2000
3000
1000
2000
3000
1000
2000
3000
Epoch
Epoch
Epoch
Figure 7: Model convergence for several configurations of depth and width of the hypernetwork architecture for default and MIP hypemnetworss. Results are for HyperMorph on OASIS. Shaded regions measure standard deviation across hypernetwork initializations.
Width 16
Width 128
Width exp
Test Dice Score
0.765
0.760
Hypernet
0.755
Default
MIP
0.750
3
5
Hypemetwork Depth
Figure 8: Test dice score for several configurations of depth and width of the hypernetwork chitecture for default and MIP hypernetworks. Results are for HyperMorph on OASIS. Box-plots are reported over the range of hypernetwork inputs y. For all hypernetwork architectures, MIP parametrizations consistently lead to more accurate models.
18

---

Published as a conference paper at ICLR 2024
C.3 CHOICE OF NONLINEAR ACTIVATION FUNCTION
While our method is motivated by the training instability present in hypernetworks with (Leaky)- ReLU nonlinear activation functions, we explored applying it other popular choices of activation functions. We consider popular activation functions GELU and SiLU (also known as Swish) that are close to the ReLU formulation, well as the Tanh nonlinear function Hendrycks Ramachandran et al. (2017).
Gimpel (0166);
We evaluate on the Bayesian hypernetworks task on the OxfordFlowers-102 dataset with primary convolutional network trained optimized with Adam. Figure 9 shows the convergence curves for Bayesian hypernetworks with primary convolutional network trained on the OxfordFlowers clas- sification task optimized with Adam. We see that MIP consistently helps for all choices of nonlinear activation function, and the improvements are similar to those of the LeakyReLU models.
Tanh
3.0
Teat
2.5
Hypernet
Default
2.0
(ours)
1.5
500
1000
1500
2000
2500 3000
500
1000 1500 Epoch
2000
2500 3000
500
1000 1500 Epoch
2000 2500 3000
Epoch
GELU
SILU
Tanh
Test Accuracy
0.8
0.6
Hypemntt Defoult (ours)
500
1000
1500 2000
2500
3000
1000
1500 2000 2500 Epoch
3000
500
1000
1500 Epoch
2000
2500
Epoch
Figure 9: MIP on alternative nonlinear activation functions Test loss (top row) and test aceu- racy (bottom row) for Bayesian hypernetworks trained on the OxfordFlowers classification task for various choices of nonlinear activation function the hypernetorrk architecture: GELU, SiLU and Tanh. For each setting, we train 3 independent replicas with different random initialization and report the mean (solid line) and the standard deviation (shaded region). We see significant improve- ments in model training convergence when the hypernetwork uses the proposed MIP parametriza- tion.
C.4 FINAL MODEL PERFORMANCE
Table I: Final model results on the test set for the considered tasks and models. We report the average performance averaged across the range of y inputs. We find MIP does not decrease model performance in any setting, while providing substantial improvements in several of them, especially when using the SGD optimizer. Standard deviation across random initializations is included in parentheses.
Adam
SGD
Task
Data
Default
MIP
Default
MIP
MNIST
98.1 (1.1)
99.1 (0.3)
99.2 (0.2)
99.0 (0.2)
Bayesian NN
DxfordFlowers--o 102
78.1 (1.9)
83.2 (0.3)
1.4 (0.1)
75.4 (0.5)
HyperMorph
OASIS
71.0 (0.3)
72.1 (0.3)
54.3 (0.4)
70.5 (0.2)
Scale-Space HN
OASIS
81.4 (0.3)
84.4 (0.6)
75.3 (2.7)
78.8 (1.4)
19

---

Published as a conference paper at ICLR 2024
C.5 NORMALIZATION STRATEGIES
Before developing MIP parametrizations we tested the viability of existing normalization strategies (such Layer Weight normalization) deal with the identified proportionality phenomenon. While normalizing inputs and activations is a common practice in neural network training, hyper- networks present different challenges, and applying these techniques can actually be detrimental to the training process. Hypernetoorks predict network parameters, and many of the assumptions behind parameter initialization and activation distribution do not easily translate between classical networks and hypernetworks.
An important distinction is that the main goal of our formulation is to ensure that the hypernetwork input has constant magnitude, not that is normalized (i.e., zero mean, unit variance). A normal- ized variable 2 (0, 1) does not have constant magnitude (i.e., 1.2 norm), over its support, so normalization techniques do not solve the identified magnitude dependency and actually lead to undesirable formulations. To show this, let æ E R be a hypernetwork activation vector, and y [0, 1] the hypernetwork input. Then, according to the identified proportionality in Section 3.2, we know that æ = y2.. Here x is the activation when the input is y and is a vector independent of y. The normalization output will be
E[2)] 72 - -
72 >[==] 2 [[=]
Norm xx)
Stdev(z) [r] Stdev [70)
Stdevv Stdev(z]
making the output independent of the hypernetwork input y- Following this reasoning, strategies like layer norm, instance norm or group norm in the hypernetwork will make the output of the model independent of the hypernetwork input, rendering the hypernetwork unusable for scalar inputs. For batch normalization cases it depends upon whether different hypernetwork inputs are used for each element in the minibatch. Otherwise, the proportionality will still hold as the batch mean and standard deviation will be the same for all entries in the minibatch. Our experimental results confirm this. Hypernetworks with layer normalization fail to train in most settings. In contrast, we found consistently that training substantially improves when using our MIP formulation. See Figure 5a in the main body which shows that none of the tested normalization strategies is competitive with MIP in terms of model convergence or final model accuracy.
not, the same logic applies as in the feature normalization strategies.
Batch Normalization Applying batch normalization fails to deal with the proportionality phe- nomenon because it normalizes statistics that are independent of the magnitude of y keeping the proportionality (loffe) 2017). In our experiments, batch normalization performed similar to the default formulation when included either the hypernetwork or the primary network, failing to address the proportionality relationship. For instance, all of the results in Figure 6 use batch nor- malization layers, as recommended for ResNet-like architectures. In this case, MIP still provides a substantial improvement in terms of model convergence and training stability.
Feature Normalization Feature normalization techniques such as layer normalization, instance normalization or group normalization do remove the proportionality phenomenon we identify (Ba et al. 2016: Ulyanov et al. 2016). However, by doing so they make the predicted weights indepen- dent of the input hyperparameter, limiting the modeling capacity of the hypernetwork architecture Moreover, in our empirical analysis, networks with layer normalization in the hypernetwork layers failed to train entirely, with the loss diverging early in training.
Weight Normalization We also considered techniques that decouple the gradient magnitude and direction such as weight normalization (Qiao al. 2019). Performing weight normalization on the hypernetwork predictions effectively decouples the gradient magnitude and direction. We find that convergence is substantially lower compared to the default parametrization. Moreover, final model performance does not match the default parametrization.
20


Reviews:
[{"review_id": "rxza8u6vOB", "reviewer": "Reviewer_mEvR", "summary": "The authors study the problem of improving the stability and efficiency of training hypernetworks. They first observe a previously unidentified problem with hypernetwork training, namely that for certain hypernetwork architectures the scaling of the input to the hypernetwork leads to the proportional scaling of the outputs. This can lead to destabilization of the training procedure and slow training. The authors propose a a novel hypernetwork formulation that removes this proportionality property. Across several architectures, tasks and optimizers, their framework leads to improved training stability and convergence.", "questions": "I have summarized above some points that were not very clear to me. I would be willing to increase my score if they were to be clarified.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "While normalizing inputs to neural network models is already well established best practice, to the best of my knowledge the specific application of this best practice for hypernetwork inputs has not been studied as much. I consider the fact that the input encoding approach of the authors is straightforward a plus. The experimental validation of the key claims of the papers is extensive.", "weaknesses": "I am not really sure if the output encoding part of the framework fits well with the problem that the authors claim to solve. It is not clear how output encoding relates to the input and output proportionality problem. It also makes interpreting the experimental results where both input and output encoding are used harder. Intuitively, output encoding allows the model to learn the task even if the hypernetwork does nothing so it is not clear if we improve hypernetwork training or just make the hypernetwork path less critical and rely on \"classical\" parameter learning which is typically more stable. \n\nAnother thing that is not clear to me is that it is not clear to me what is special about hypernetworks so that input normalization needs to work differently than \"classical\" networks. The scaling property would hold for a \"classical\" network as well for the appropriate activations. And yet Appendix C.5 claims that standard normalization techniques of \"classical\" networks would not work for hypernetworks. Appendix C5 seems to suggest that \"classical\" normalization techniques could make the output independent of the input. It is not clear to me why this is a problem only for hypernetworks and not for \"classical\" networks.\n\nAlso it would be useful to clarify why one could not just divide the hypernetwork input by its norm, at least for the case of a multi-dimensional hypernetwork inputs. In my mind, if the problem was just the scaling of the output, this should have worked. The suggested methods go beyond just fixing the scale of the input vector as a whole so it is not very clear if the problem is indeed the input output proportionality or some more general feature scaling issue.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9jfZjrwZlz", "reviewer": "Reviewer_6gdA", "summary": "The paper presents a solution, Magnitude Invariant Parametrizations (MIP), to a previously unidentified optimization problem in hypernetwork training that causes large gradient variance and unstable training dynamics. MIP, by modifying the typical hypernetwork formulation, addresses this issue without adding training or inference costs. The authors extensively test MIP, demonstrating improved stability and faster convergence in hypernetwork training across various settings. They also release HyperLight, an open-source PyTorch library, to ease the implementation of MIP and promote hypernetwork adoption. Through rigorous analysis and experimentation, the paper showcases MIP's potential in substantially enhancing hypernetwork training, marking a significant advancement in this domain.", "questions": "Mentioned in the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper include the identification of a novel optimization problem in hypernetwork training, the proposal of a new formulation (MIP) that addresses this issue without extra computational costs, extensive testing and comparative analysis demonstrating MIP's effectiveness, and the provision of an open-source library, HyperLight, to facilitate the practical adoption of the proposed solution in hypernetwork models. Through rigorous analysis and extensive experimentation, the paper makes a significant contribution towards improving the stability and convergence speed in hypernetwork training, providing a promising direction for the community.", "weaknesses": "The paper mainly focuses on fully connected layers and common activation, initialization choices, and optimizers (SGD with momentum and Adam) in its experiments, which may not encompass a broader spectrum of hypernetwork architectures or other types of networks. There's also a mention of unexplored territories like the effect of MIP on transfer learning and other less common architectures and optimizers, indicating a scope for broader empirical validation. Furthermore, the impact of MIP on real-world applications or larger-scale problems is not thoroughly explored, which might be crucial for the adoption of this technique in practical settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2tfAVDGyR", "reviewer": "Reviewer_7ccS", "summary": "This work proposes a novel parametrisation for hypernetworks that is magnitude invariant (MIP). The main motivation for MIP stems from the author’s observation that the hypernetwork output, when using piece-wise linear activations, has a magnitude proportional to the hypernetwork input. The authors argue that this proportionality is detrimental for the hypernetwork optimization as it affects the gradient variance. MIP is then introduced as a way to remove this dependence on the magnitude by parametrising the input to the hypernetwork in terms of Fourier features that have a constant norm throughout training. Then, in order to allow for hypernetwork outputs that are non-proportional to the hypernetwork inputs, the authors propose a residual parametrisation where an auxiliary weight matrix is trained directly and the hypernetwork output is used as an additive correction to that weight matrix. The authors then show that these two modifications to standard hypernetwork training, solve the proportionality problem and improve hypernetwork performance across the board.", "questions": "I find the overall paper to be well written, the arguments clear and the proposed solution convincing. Therefore, I am happy to recommend acceptance. My questions and suggestions are the following:\n- It would be interesting to see whether the residual formulation of the hypernetwork closes meaningfully the gap between fully task specific parameters and the ones predicted by the hyper network, i.e., $\\theta^0 + h(E_{L2}(\\gamma); w)$. For example, what is the performance if on a new task $t$ one starts from $\\theta^0$ and just optimises for a specific number of steps on that task to get $\\theta_t^*$? Is $\\theta_t^* - \\theta_0$ related to $h(E_{L2}(\\gamma); w)$? Is the performance of $\\theta_t^*$ similar to the performance of $\\theta^0 + h(E_{L2}(\\gamma); w)$?\n- Does the update given by the hypernetwork in this residual formulation need to be dense? Is the hypernetwork only adapting a few dimensions of $\\theta^0$?\n- It is not clear why for non-scalar inputs $\\gamma$, i.e., $\\gamma \\in \\mathbb{R}^D$ with $D \\geq 2$  a simple unit normalisation transformation, i.e., $\\hat{\\gamma} = \\frac{1}{\\|\\|\\gamma\\|\\|_2}\\gamma$ would not work for removing the dependence on the magnitude. It seems to me that in this case each $\\hat{\\gamma}$ would just correspond to a different point on the hypersphere and the output of the hypernetwork would not be independent of the input.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The authors identify a novel issue that seems to be important (based on the improvement delta from MIP) for hypernetwork training.\n- The specific MIP parametrisation is novel and practically broadly useful for any task that involves hypernetworks. \n- The experiments are relatively extensive in terms of tasks and ablation / robustness studies.\n- The paper is mostly well written and clear in the presentation of the main ideas and results.", "weaknesses": "- While the authors do empirically show that MIP benefits training, it is not clear whether the increased variance could also be controlled with, e.g., appropriately chosen (i.e., lower) learning rates and (i.e., higher) momentum, in the original parametrisation (which could attain similar performance, albeit slower).\n- This is something that the authors themselves identify, but given that hypernetworks are becoming popular for fast adaptation of pertained models, e.g., [1], it is important to see whether the magnitude proportionality effect is also detrimental there. \n\n[1] HyperDreamBooth: HyperNetworks for Fast Personalisation of Text-to-Image Models, Ruiz et al., 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NE1Vc6iMLd", "reviewer": "Reviewer_YXs9", "summary": "This paper identifies a problem that contributes to the challenge of training for hypernetworks: The magnitude proportionality between the inputs and outputs of the hypernetwork. The authors demonstrate how this proportionality can result in unstable optimization processes. To mitigate this issue, they introduce a magnitude-invariant parameterization method for hypernetworks, demonstrating its effectiveness in stabilizing the training process and accelerating convergence.", "questions": "- I might miss the related information. Apologies if that is the case. Are there any results of combining additive output with BatchNorm/LayerNorm in Figure 5? It is interesting to see such results because it verifies that MIP has something that normalization layers cannot offer. If proportionality is the main reason, what are the theoretical reasons that MIP can solve the problem but normalization cannot?\n- If the problem is proportionality, how about we use other activation functions such as ELU or GELU? Does that solve the problem? If not, can we still say the problem is proportionality?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The techniques proposed in this paper are indeed helpful in stabilizing the training of hypernetworks. Given that hypernetworks are widely used in machine learning. These techniques can be useful for practitioners.", "weaknesses": "- The discussion regarding why input/output proportionality results in unstable training is somewhat insufficient. It is a well-known fact that neural networks with piecewise linear activation functions, such as ReLU networks, exhibit this proportionality between inputs and outputs. However, if proportionality is the main reason, why ReLU networks are not well-known for having such problems for classical tasks such as regression/classification (not in hypernetworks)? Therefore, I think more explanations or discussions are needed in the text.\n- The two strategies introduced in the paper, namely input encoding and additive output formulation, are a bit ad hoc from my perspective. While these methods undoubtedly offer practical benefits for hypernetworks, I am not sure if the contribution is significant enough. A more thorough exploration and explanation of the theoretical underpinnings that underscore the significance of these techniques would enhance the paper's contribution and provide a clearer justification for their adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)