Scale-Space Hypernetworks for Efficient Biomedical Image Analysis
Authors: Jose Javier Gonzalez Ortiz
John Guttag
Adrian V Dalca
Venue: NeurIPS 2023 poster (2023)
Keywords: hypernetworks
amortized learning
computer vision
rescaling
convolutional neural networks
pareto efficiency
TL;DR: We learn a single hypernetwork model capable of characterizing the accuracy-efficiency for different resizing factors

Abstract: Convolutional Neural Networks (CNNs) are the predominant model used for a variety of medical image analysis tasks. At inference time, these models are computationally intensive, especially with volumetric data.In principle, it is possible to trade accuracy for computational efficiency by manipulating the rescaling factor in the downsample and upsample layers of CNN architectures.However, properly exploring the accuracy-efficiency trade-off is prohibitively expensive with existing models.To address this, we introduce Scale-Space HyperNetworks (SSHN), a method that learns a spectrum of CNNs with varying internal rescaling factors.A single SSHN characterizes an entire Pareto accuracy-efficiency curve of models that match, and occasionally surpass, the outcomes of training many separate networks with fixed rescaling factors.We demonstrate the proposed approach in several medical image analysis applications, comparing SSHN against strategies with both fixed and dynamic rescaling factors.We find that SSHN consistently provides a better accuracy-efficiency trade-off at a fraction of the training cost. Trained SSHNs enable the user to quickly choose a rescaling factor that appropriately balances accuracy and computational efficiency for their particular needs at inference.

Scale-Space Hypernetworks for Efficient Biomedical Imaging
Jose Javier Gonzalez Ortiz"
John Guttag MIT CSAIL Cambridge, MA
Adrian Dalca MIT CSAIL & MGH, HMS Cambridge, MA adalca@mit.edd edu
MIT CSAIL Cambridge, MA
josejgamit edu
guttagEmit.edt edu
Abstract
Convolutional Neural Networks (CNNs) the predominant model used for variety of medical image analysis tasks. At inference time, these models are com- putationally intensive, especially with volumetric data. In principle, is possible to trade accuracy for computational efficiency by manipulating the rescaling factor in the downsample and upsample layers of CNN architectures. However, properly exploring the accuracy-efficiene trade-off is prohibitively expensive with existing models. To address this, we introduce Scale-Space HyperNetworks (SSHN), a method that learns spectrum of CNNs with varying internal rescaling factors. A single SSHN characterizes an entire Pareto accuracy-efficiency curve of models that match, and occasionally surpass, the outcomes of training many separate networks with fixed rescaling factors. We demonstrate the proposed approach in several medical image analysis applications, comparing SSHN against strategies with both fixed and dynamic rescaling factors. We find that SSHN consistently provides a better accuracy-etto accuracy-effcienccy trade-off at fraction of the training cost. Trained SSHNs enable the user to quickly choose rescaling factor that appropri- ately balances accuracy and computational efficiency for their particular needs at inference.
Introduction
Convolutional Neural Networks (CNNs) are a widely-used and fundamental tool in medical image 52 (88). However, their practical deployment hindered by their high computa- tional requirements, especially in resource-constraing
analysis 2
20 37
environments often encountered in medical applications 65 71 This has led to techniques lower computational demands while preserving
model accuracy. Popular strategies to address this issue encompass quantization [28 33 [6], prun-
ing (5) 19
22
35 and factoring [10 34 70]. which focus on reducing the parameter count and the
size of the convolutional kernels. These techniques invariably introduce a trade-off between model accuracy and computational efficiency. To understand this trade-off, developers need train many models and select the best performing one based on balance of the aforementioned metrics.
In this work, propose alternative approach aimed at improving the computational efficiency of prevalent medical imaging models, such as U-Net networks, which internally resize image features at different scales (58). While existing convolutional architectures usually reduce the spatial scale by a fixed factor of two 24 27 40 66), we study more general architectures that enable this factor to vary continuously. By reducing the spatial dimensions of intermediate features, we reduce the computational cost at both training and inference stages, as the convolutional cost is directly linked the size of the intermediate spatial feature maps.
*Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Hyperparameter Search Approach
Proposed Amortized Learning with Hypernetworks
1. Train N networks with varying rescaling amounts independently
2. Choose from trained models or train more using narrower range
1. Train single hypemeeworr
2. Rapidly churacterize
conditioned on rescaling
Pareto frontier of models
CINN with y 0.1
Rescaling Factor yp
CNN with 0.2
Accurec
Vesunae
Hypernetwork
CNN with o 0.3
CNN with yp
CNN with y == 0.9
Inference Cost = /(p)
Inference Cost f(P)
Figure 1: Strategies for exploring the accuracy-efficiency trade-off of varying the CNN rescaling factor. A standard approach (left) requires training N models with different rescaling factors independently. Our proposed amortized learning strategy (right) exploits the similarity between the tasks and trains single hypernetwork model. Once trained, efficiently characterize the Pareto frontier between model accuracy and computational cost. This enables rapid and precise model choices depending the desired trade-off inference.
rapidly evaluate many rescaling settings and
However, like with other techniques for reducing computation, characterizing the trade-off between accuracy and efficiency for a range rescaling factors can computationally demanding, as requires training multiple independent models. To address this challenge, we introduce a hypernet- work learning framework, Scale- Space Hypernetworks (SSHN), which enables single model to capture complete landscape of models representing continuous range of rescaling ratios. Once trained, this single SSHN model can be used to rapidly generate the Pareto frontier for the trade-off' between accuracy and efficiency a function of the rescaling factor (Figure [ This enables user to choose, depending on the inference setting, a rescaling factor that appropriately balances accuracy and computational efficiency for their case.
We evaluate our approach using several medical image segmentation and registration tasks including different biomedical domains and image types. We demonstrate that the proposed hypernetorr based approach able to learn models a wide range of feature rescaling factors, and that inference networks derived from the hypernetwork perform at least as well, and in most cases better than, networks trained with fixed rescaling factors. SSHNs enable us to find that a wide range of rescaling factors achieve similar accuracy results despite having substantially different inference costs.
Our main contributions are:
We introduce Scale-Space HyperNetworks (SSHN), a single model that, given a rescaling factor, predicts the weights for segmentation network that uses that amount rescaling in the spatial downsampling and upsampling layers. By reducing the spatial dimensions of intermediate features, we reduce the inference cost.
We evaluate our method on several medical image analysis tasks and show that this approach makes it possible to characterize the trade-off between model accuracy and inference efficiency far faster and more completely than existing approaches.
Using SSHNs, demonstrate that variety of medical imaging tasks, many rescaling factors lead to similar results despite having substantially different inference costs. We use SSHN to rapidly explore this trade-off for a dataset of interest, showing that it can lead to 50% inference FLOPPs reduction without sacrificing model accuracy.
We show that learning a single model with varying rescaling factors has regularization effect, which consistently improves the accuracy of networks trained using our method with respect to networks trained with a fixed rescaling factor. For some tasks, SSHN consistently outperforms regular models by 2-3% for the same rescaling factor.

---

2 Related Work
Resizing in Convolutional Neural Networks. Resizing layers are a fundamental operation common
to most modern convolutional neural architectures [24 55 66 78). These are used downscale or upscale the spatial dimensions of intermediate feature maps, providing a way for the network to aggregate visual context at multiple scales. Resizing layers have been implemented in variety of ways, including max pooling, average pooling, bilinear sampling, and strided or atrous convolutions [3] (|99663) Recent work has investigated using a wider range of rescaling operations by stochastically downsampling intermediate features [20
42]
or learning differentiable resizing modules that replace
resizing operations in the network [36 45 57 These techniques learn the differentiable modules
as part of the training process, optimizing them to improve model accuracy. The final result is a model with features optimally resized for the data available at training. contrast, our goal is to learn landscape of models with varying rescaling factors that characterize the entire trade-off curve between accuracy and efficiency. This also enables users to choose an inference-time resizing factor that appropriately balances efficiency and accuracy for their dataset.
Hypernetworks, neural network models that generate the weights for another neural network 21 39 60), are effective models that have been successfully used in a wide range of applications. These
include neural architecture search 6 72.. Bayesian optimization 41 [53, weight pruning 46
continual learning [68], multi-task learning 66 meta-learning [3 and knowledge editing
Recently, hypernetworks have been employed in hyperparameter optimization problems [25] 49
[9]. In these gradient-baser -ased optimization approaches, hypernetwork is trained to predict the weights of a primary network conditioned on a given hyperparameter value. Our work shares similarities with sypernetwork-basee neural architecture search techniques, as both strategies explore a multitude of potential architectures during training [7]. However, our work features key distinction: we do not optimize the architectural properties of the network during training. Instead, we focus on learning variety of architectures jointly, and enable choosing the architecture during inference based on the needs of the user.
3 Scale-Space HyperNetworks
Convolutional Neural Networks (CNNs) are parametric functions و 3 (x; 0) that map input values x output predictions ý using a set of learnable parameters 0. The network parameters 0 are optimized to minimize a loss function L given dataset 'D using stochastic gradient descent strategies. For example, in supervised scenarios,
0- = arg min
Σ (((r,)))) Z.EED
Most CNNs architectures use some form of rescaling layers with a fixed rescaling factor that change the spatial size of intermediate network features, most often by halving (or doubling) along each spatial dimension. We define a generalized version of convolutional models fo 0) that perform feature rescaling by a continuous factor E [0, 1]. Specifically, f (2; 0) has the same parameters and operations as fxx; 0), with all intermediate feature rescaling operations being determined by the factor yp. For instance, in existing image models, downsampling layers R(z) map an input tensor of shape (C, H, W) to an output tensor of shape (C, H/2, W/2) where and W are the height and width of the feature map and C' is the channel dimension. In contrast, we consider resizing layers of the form (x) that produce an output tensor of shape (C, [W [P)l]).
To characterize model accuracy as a function of the rescaling factor o using standard techniques requires training multiple instances of f (-; 8), one for each rescaling factor, leading substantial computational cost. Instead, we propose a framework to learn a family of parametric functions (= 0) jointly, using an amortized learning strategy that learns the effect of rescaling factors yp on the network parameters 0. We employ a function h1pp; w) with learnable parameters w that maps the rescaling ratio &o set of convolutional weights for the function S (=; We model h( w) with another neural network, or hypernetwork, and optimize its parameters w based on the learning objective
-n arg min Σ
L (Fi (()p:)))))))))))))))))))))))))))
(2)
x.yED
3

---

Hypemeework A(po
Primary Network fy(m; 0)
Resizing Layer Ry(w)
W
Roli
Resize
Denso
ConveRoLU
H
Resiz
Onnee Rel
Dense
ReLlU
Concat
4
Resize
Dense
ConyeReLu
Concal
[
84
Dense
R
0.
W
grad
nograd
Figure 2: Diagram of the proposed hypernetwork-basee model used to jointly learn family of segmentation networks with flexible feature rescaling. At each iteration, a rescaling factor yp is sampled from a prior distribution p(y)) Given ço, the hypernetwork h(p:w) predicts the parameter values 9 of the primary network f (x: 9). Given a sample datapoint z, we predict y = (x; f) and compute the loss L(ů. u) with respect to the true label y. The resulting loss gradients only lead to updates for learnable hypernetwork weights w. The hypernetwork implemented as Multi Layer Perceptron (MIP) and the primary network uses UNet-like architecture. On the right, illustrate how the resizing layer (I) variably resizes the spatial dimensions of a given input based on the scale factor s.
At each training iteration, we sample a resizing ratio yp ~ p(() from a prior distribution p(()) which determines both the rescaling of internal representations and, via the hypernetwork, the weights of the primary network ())
The hypernetwork model has more trainable parameters than regular primary network, but the number of convolutional parameters 0 the same, At inference time, we leave out the hypernetwork entirely, using the predicted weights for given y,, incurring only the computational requirements of the primary network.
Implementation. We model the hypernetwor h(p:w) using fully connected neural network with several layers, a common choice hypernetwork literature 7 21 25
67 68.. We encode the hypernetwork input as a vector [sc, so] to prevent biasing the fully connected layers towards the magnitude of yp. Given an input value sp, the hypernetwork h predicts a set of outputs (fp, (fp.01.... .I..... 0x corresponding to the convolutional filters of the primary network. Each weight is predicted as flat vector and then reshaped to the appropriate dimensions in the primary network. In the applications in experiments, the primary network follows a UNet-like structure, the most prevalent and widely-used design medical imaging tasks [30 (58) We modify the architecture by replacing the discrete downsampling and upsampling layers with variable resizing layers determined by y,, using differentiable bilinear interpolation operations. Figure illustrates Spale HyperNetworks applied to three stage U-Net architecture.
4 Experimental Setup
4.1 Tasks
We evaluate on two fundamental medical image analysis tasks: segmentation and registration.
Segmentation. In segmentation tasks, the primary network [(x; 0) - maps an input scan to an output segmentation map y. We use supervised training and optimize an objective of the form L(g. v). where y is the ground truth segmentation map and ý is the predicted segmentation map. In our

---

experiments, we first pre-train the networks using a categorical cross entsopy loss and then finetune them using soft Dice-score loss [4]66, We evaluate each segmentation method using the Dice score [1]. which quantifies the overlap between two regions and is widely used in the segmentation literature. Dice is expressed as Dice(y. j) = (2ly ÿ / + lől). A Dice score of I indicates perfectly overlapping regions, while 0 indicates no overlap. For datasets with more than one label,
we average the Dice score across all foreground labels.
Registration. In registration tasks, moving image Zm is registered to a fixed image 2 by applying flow or deformation field ø, like 8y = I' Φ. Existing work has shown that parametric model f + é can be optimized to learn this mapping (2). This is commonly achieved with an unsupervised objective that optimizes both an image alignment term Sim between the fixed and moved image I'm 9, as well as term encouraging spatial regularization of the flow field: Leeg The learning objective is then -= f.sim ( Φ Jy) + XregCrey(()) (6), where Are controls the amount of flow-fiel field regularization. We follow the experimental setup as in (2), using U-Net architecture for the primary (registration) network, MSE for Sim and total variation for freg For evaluation, we use the predicted flow field to warp anatomical segmentation label maps of the moving image, and measure the volume overlap to the fixed label maps.
4.2 Datasets
We four popular and quite different biomedical imaging datasets OASIS 25 50 PanDental D.
WBC (76) and CAMUS (4). OASIS consists of 414 brain MRI scans that have been skull-stripped, bias-corrected, and resampled into an affinely aligned, common template space. For each scan, segmentation labels for 24 brain substructures from the FreeSurfer protocol
[8] in a 2D mid-coronal slice are available. PanDental includes 215 XRay scans of the lower half of patients' heads, with label for the entire mandible and a label for all teeth. CAMUS is an ultrasound dataset, containing 2D apical nour-chamber and two-chamber view sequences acquired from 500 patients. WBC consists of 400 microscopy images of white blood cells, with labels for the cell nucleus and cytoplasm. In each dataset, we trained using (64%. 16%. 20%) train, validation, test split of the data.
4.3 Baseline Methods
We compare our method against three other strategies for creating an efficiency-accurace Pareto curve: Fixed, Stochastic, and FiLM.
Fixed. We train a set of conventional U-Net models with fixed rescaling factors. These are identical to the primary network, and are trained using fixed rescaling factor & (i.e., no hypernetwork is used). We train several baseline models at 0.05 rescaling factor increments from 0 to 0.5.
Stochastic. In recent work, convolutional networks leading to improved generalization [42]. We incorporate a standard U-Net baseline, where scale factor yp is stochastically sampled during training. Under this setting, single model is learned, whose parameters 0 are optimized to work with all possible rescaling factors within the sampled range.
trained with stochastic rescaling factor,
FiLM Feature- wise Linear Modulation (FiLM) layers conditioning mechanism for convolutional models [16] (4).. FiLM modules map an input 2 to series of multiplicative and additive vectors ? and 3 that are used to affinely transform the intermediate feature maps at different points of the network. Recent work has used this mechanism to learn models in a amortized way, by stochastically sampling the conditioning input 2 during training and thus enabling Pareto frontier of models (13) We test model incorporating FiLM operations after every convolutional operation. Similar to our approach, the scale factor y is stochastically sampled during training, and used to modulate the FiLM modules in the primary network.
4.4 Experimental Details
Primary Network Architecure. For our experiments, we use the U-Net architecture, since it the most widely-use segmentation CNN architecture in the medical imaging literature [30, 58). We use bilinear interpolation layers to downsample or upsample a variable amount based on the scale factor y for downsampling. For the downsampling layers we multiply the input dimensions by yo and round to the nearest integer. For the upsampling layers we upsample to match the size of the feature map that is going to be concatenated with the skip connection. We use five encoder stages and four
5

---

CA5I5
PanDental
WBC
CAMUS
3.40
0.90
0.52
0.85
0.90
0.50
0.58
Dice
3.80
0.82
0.86
0.70
0.75
Method
SSHN (ours)
Reed
0.75
0.52
0.65
Stochastic
RLM
0.70
0.50
0.50
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
0.0
0.2
0.4
Scale Factor 9
Scale Factor
Scale Factor
Scale Factor 9
Figure 3: Accuracy Results. Comparison between family of networks trained with fixed amounts of feature rescaling (Fixed), our proposed SSHN method and the other variable resizing methods (Stochastic, FiLM). We report results in the test set for the four considered segmentation datasets. In all settings the SSHN model outperforms the individually trained models (Fixed), whereas the other dynamic resizing baseline fail to meaningfully improve upon them.
decoder stages, with two convolutional operations per stage and LeakyReL.U activations [48] For networks trained on OASIS Brains we use 32 features at each convolutional layer. We found that the results we discuss below held across various architectural settings. Hypernetwork. We implement the hypemnetwrr using three fully connected layers. Given vector of [,p 1 yp] input, the output size is set to the number of convolutional filters and biases in the segmentation network. We use the vector representation (p, 1 - +p] to prevent biasing the fully connected layers towards the magnitude of sp. The hidden layers have 10 and 100 neurons respectively and LeakyReLU activations [48] on all but the last layer, which has linear outputs. Hypernetwork weights are initialized using Kaiming initialization (23), and bias vectors are initialized to zero. Evaluation. For each experimental setting train five replicas with different random seeds and report mean and standard deviation. Once trained, we use the hypernetwork to rapidly evaluate a range of yo, using 0.01 intervals. This is more fine-grained than is feasible for the Fixed baselines, since all evaluations for method use the same model, whereas the Fixed baseline requires training separate models for each rescaling factor. Training. We use the Adam optimizer [38] and train networks until the loss in the validation set stops improving, During training we sample the rescaling from (0, 0.5) We provide additional platform and reproducibility details in the supplement, and we will publicly release our code.
Experimental Results
We present two sets of experiments. The first set evaluates the accuracy and efficiency of segmentation models generated by our hypernetwork, both in absolute terms and relative models generated with fixed resizing. In the second set of experiments we study why SSHN achieve improved results.
5.1 Accuracy and Computational Cost
Accuracy. First, assess the ability of the hypernetwork model to learn continuum of primary network weights for various rescaling factors, and evaluate how the performance of models using those weights compare to that of models trained independently with specific rescaling factors. Figure 3 shows the Dice score for the segmentation task, on the held-out test split of each dataset, as a function of the rescaling factor ş.. We see that in some cases many rescaling factors achieve similar performance. For example, in the OASIS task, SSHN achieves a mean Dice score across brain structures above 0.89 for most rescaling factors-suggesting that inference cost be reduced without sacrificing accuracy.
The Stochastic and FiLM models fail to consistently match the performance of the Fixed baselines. In contrast, the networks predicted by SSHN are consistently more accurate than equivalent networks trained with specific rescaling factors, suggesting that the SSHN models, unlike the other approaches, are successfully adapting the predicted weights based on the scaling factor. Table 5 shows the best performing model variant found for each approach.
Figure preseenssss results for registration of the OASIS dataset. Again, SSHN networks out perform the independently trained networks. In contrast, the Stochastic and FiLM methods perform slightly worse than the Fixed baseline. Experiments later in the paper delve deeper into this improvement in model generalization.

---

Dice Score
gixed(g=1/2)
0.6
g=/4)
pSHN( 1/2)
0.4
Cerebrat (42 (42.2%)
Cerebral-Cx (35 99%)
Thalamus (5.2%)
Ventrai-DC (4.2%)
Hopacampus (3.7 9.7%))
Latural-Vent. (2. 2.7%
Brain Stam 1.9%
Putamen (1.8%)
Pallidum (0.7%)
VatVent.. (0.6%)
BrotWent. 5..5%)
Caudata (0.%%)
Choraid Plous (0.2%)
Figure 4: Dice coefficient across various brain structures for the baseline method (Fixed) trained and evaluated وي 0.5, and the SSHN approach evaluated various rescaling factors yp for the OASIS segmentation task. Anatomical structures are sorted by mean volume in the dataset (in parentheses). Labels consisting left and right structures (e.g. Hippocampus) are combined. We abbreviate the labels: white matter (WM), cortex (CX) and ventricle (Vent). The SSHN model achieves similar performance for all labels despite the reduction in feature map spatial dimensions.
We also explore how varying the rescaling factor yo affects the segmentation of smaller labels the image using the OASIS dataset. Figure presents breakdown by neuroanatomical structures for Fixed baseline with = 0.5 and a subset of rescaling factors for the SSHN model. The proposed hypernet- work model achieves similar per- formance for all labels regardless of their relative size, even when downsampling by substantially larger amounts than baseline.
Table 5: Segmentation Results, Test Dice score values for the different methods evaluated the considered segmen- tation tasks. Standard Deviation reported across training replicas.
Method
CAMUS
OASIS
PanDental
WBC
Fixed Stochastic FiLM
83.5 1 0,3
89.1 1 0.0
88.9 1 0.2
90.9 1 0.6
0.1
84.8 1 0.2
0.7
88.1 +: 2.0
83.2 0.3
87.2
- 0.6
8.
87.1 +: 1.4
SSHN (ours)
86.2 -:: 0.3
90.7 + 0.1
89.9 : 0.3
92.7 + 0.4
0.72
0.71
Computational Cost. We ana- lyze how each rescaling factor af- fects the inference computational requirements of each segmenta- tion network. We measure the number of floating point opera- tions (FLOPs) required to use the network for each rescaling factor at inference time. We also eval- uate the amount of time used to train each model.
Dice
Method
Fixed
(ours)
0.67
Stochastic
FILM
0.66
Scale Factor
Inf Inference LOPS
Figure 6: Registration Results. Results for the learnable registration task on OASIS. Test set results function of the rescaling factor (left) and as a function of the computation cost, in GFLOPs (right) for the considered methods. The SSHN model matches the independently trained networks (Fixed) for most values of the rescaling factor yo
Figure depicts the trade-off be- tween test Dice score and the in- ference computational cost for all segmentation tasks, and Tigure the results in the registration task on OASIS. We observe that the choice of rescaling factor has substantial effect on the computational requirements. For instance, for the OASIS and PanDental tasks, reducing the rescaling factor from the default yp 0.5 to yp = 0.4 reduces computational cost by over 50% and by 70% for y = 0.3. all cases, we again find that hypernetwork achieves better accuracy-FLOPP trade-off curve than the set of baselines. Table in the supplement report detailed measurements for accuracy Dice score), inference cost (in inference GFLOPs) and training time (in hours) for all methods approaches.

---

CA515
PanDental
WBC
CAMUS
0.40
0.90
0.92
0.85
0.50
0.00
0.88
0.80
0.84
0.58
0.75
Methed
0.70
(ours)
0.75
0.82
0.65
Stochastic
0.78
0.6
0.6
1.0
0.6
1.0
Inference GFLOPs
Inference GFLOPs
Inference GFLOPS
Inference GFLOPS
Figure Efficiency Results. Trade-off curves between test model accuracy (Dice score) and inference computational cost (GFLOPs). SSHN models successfully characterize the trade-off between model accuracy and efficiency while outperforming models trained with fixed amounts of feature rescaling (Fixed). In all datasets, the inference computational cost of SSHN can be reduced by least 40% without a significant change in model aceuracy. Results are averaged across network initializations, and variance is indicated with shaded regions.
Because of identical inference cost (for given rescaling factor) and improved segmentation quality of SSHN, its Pareto curve dominates the networks with fixed resizing. SSHN therefore requires less inference computation for comparable model quality. For example, for segmentation networks trained on OASIS, for yp E [0.25, 0.5] there is no loss in quality, showing that a more than 20% reduction in FLOPS can be achieved with no loss of performance. SSHN achieves similar performance to the best baseline with o = 0.15, while reducing the computational cost of inference by 50%. Characterizing Pareto frontier using SSHN requires substantially fewer GPU-hours than the
the sccuracy-efficiene traditional approach, while enabling a substantially more fine-scale curve. In our experiments, the set of baselines required over order of magnitude (10x) longer train compared the single hypernetwork model, even when we employ a coarse-graiie aaase--raiied baseline search.
5.2 Analysis Experiments
In this section perform additional experiments designed to shed light on why the hypernetwork models achieve higher test accuracy than the baseline models.
Varying Prior Width. We first study the quality of hypernetwork segmentation results with varying width of the prior distribution p()). Our goal is to understand how learning with varying amounts of rescaling affects segmentation accuracy and generalization. We train set of SSHN models on the OASIS dataset but with narrow uniform distributions (((055 - r, 0.5 + r), where r controls the width of the distribution. We vary Dice scores for training and test set on the OASIS segmentation task, as well as a baseline ONe for comparison. With 0, the hypernetwork is effectively trained with a constant &p = 0.5, and slightly underperforms compared to the bascline UNet. However, as the range of the prior distribution grows, the test Dice score of the hypemetwork models improves, surpassing the baseline UNet performance.
[0, 0.01, 0.05) and evaluate them with yp = 0.5 Figure 8 shows
Figure X suggests that the improvement over the baselines can be least partially explained by the hypernetoork learning a more robust model because it is exposed to varying amounts of feature rescaling. This suggests that stochastically changing the amount of downsampling at each iteration has regularization effect on learning that leads to improvements in the model generalization. Wider ranges of rescaling values for the prior p(ye) have a larger regularization effect. At high values of improvements become marginal and the accuracy of model trained with the widest interval p(p) = (1(044, 0.6) is similar to the results of the previous experiments with p(p) = 24 1).
Weight transferability. We study the effect of using a set of weights that was trained with a different rescaling factor. Our goal is understand how the hypernetwork adapts its predictions as the rescaling factor changes. We use fixed weights 6' = h+0.5; w) as predicted by the SSHN model with input &f = 0.5, and run inference using different rescaling factors şp For comparison, we apply the same procedure to set of weights of a Fixed U-Net baseline trained with fixed presents segmentation results on OASIS for varying rescaling factors sp Transferring weights from the baseline model leads to a rapid decrease in performance away from y = 0.5. In contrast, the weights generated by the hypernetwork are more robust and can effectively be used in the range xp [0.45, 0.6]. Nevertheless, weights learned for a specifie rescaling factor xp do not generalize well
0.5. Figure P
8

---

0.910
Test Dice Score
0.905
Test Dice D Score
0.6
0.900
0.4
0.895
Fixed (0=
0.2
T-Fixed p 0.5
0.890
SSHN (0 0.5)
T-SSHN 0-0.5
0.0
0.00
0.02 0.04 0.06
0.08 0.10
0.0
0.2
0.4
0.6
0.8
1.0
Width of training prior p(p)
Rescaling Factor @
Figure 8: Segmentation Dice on OASIS Test data a function of the prior distribution range 2r. Results are evaluated with 0.5 in all settings, regardless of the training prior, and we include baseline UNet (Fixed) model. Uniform priors for oo are of the form (0.55 r, 0.5 r), i.c. they linearly increase around 0.5. Standard deviation (shaded regions) is computed over random ini- tializations.
Figure 9: Segmentation results for models re- sulting from transferring weights learned for a given rescaling factor and evaluated at dif- ferent factors. The convolutional parameters learned by the hypernetwork (T-SSHN) transfer nearby factors substantially better than the weights of the baseline model (T-Fixed), Stan- dard deviation (shaded regions) is computed over random seeds.
to wide range of different rescaling factors, whereas weights predicted by the hypernetwork transfer substantially better to other rescaling factors.
Limitations
We evaluated our proposed framework on image segmentation and registration, prevalent tasks in medical image analysis. We believe that our method can be extended to other tasks like object detection or classification. Constructing such extensions is exciting area for future research. Furthermore, we focus on the U-Net as a benchmark architecture because it a popular choice for segmentation networks across many domains [58
13 32
2 4 It also features key properties,
such as an incoder-decode structure and skip connections from the encoder to the decoder, that are used in other segmentation architectures g 74 []
Nevertheless, in Section C. the Supplemental material, we report results for other popular segmentation architectures, including PSPNet, FPN and SENet, and find analogous results.
Conclusion
We introduce SSHN: hypernetwork-based approach for efficiently learning family of CNNs with different rescaling factors an amortized way. Given trained model, SSHN enables efficiently generating the Pareto frontier for the trade-off between inference cost and accuracy. This makes possible to rapidly search for a rescaling factor that maximizes accuracy subject to computational cost constraints. For example, using the hypernetwork, demonstrate that using larger amounts of downsampling can often lead to substantial reduction in the computational costs of inference, while sacrificing nearly accuracy. Finally, demonstrate that for variety of biomedical image analysis tasks and datasets, SSHN models models achieve accuracy at least as good, and more often better than, CNNs trained with fixed rescaling ratios, while producing more robust models. This strategy enables the construction of a new class of cost-adjustable models, which can be adapted to the resources of the user and minimize carbon footprint while providing competitive performance in biomedical image analysis.
Acknowledgements
This research is supported by Quanta Computer, Inc, and Wistron Coporation, Additionally, the project was supported by NIH AG066422 and ROLAG070988.

---

References
[1] Amir Hossein Abdi, Shohreh Kasaci, and Mojdch Mehdizadeh. Automatic segmentation of mandible in panoramic x-ray. Journal of Medical Imaging, 2(4):044003, 2015.
[2] Guha Balakrishnan, Amy Zhao, Mert Sabuncu, John Guttag, and Adrian V Dalca. Voxelmorph: a learning framework for deformable medical image registration. IEEE transactions on medical imaging, 38(8): (103):178818800 1800, 2019.
[3] Yoshua Bengio, Ian Goodfellow, and Aaron Courville. Deep learning, volume MIT press Cambridge, MA, USA, 2017.
[4] Benjamin Billot, Douglas N. Greve, Koen Van Leemput, Bruce Fischl, Juan Eugenio Iglesias, and Adrian Dalca. A learning strategy for contrast-agnostie mri segmentation. In Tal Arbel, Ismail Ben Ayed, Marleen de Bruijne. Maxime Descoteaux, Herve Lombaert, and Christopher Pal, editors, Proceedings of the Third Conference on Medical Imaging with Deep Learning. volume 121 of Proceedings of Machine Learning Research, pages 75- 93. PMLR, 06-08 Jul 2020.
[5] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag, What is the state neural network pruning? arXiv preprint arXiv:2003.03033 2020.
[6] Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Smash: one-shot model architecture search through hypernetworks. arXiv preprint arXiv:1708.05344 2017.
[7] Oscar Chang, Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernet- works. In International Conference Learning Representations, 2019.
[8] Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.0555 2017.
[9] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pages 801-818, 2018.
[10] François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 1251-1258, 2017.
[11] Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. arXiv preprint 2 2021.
[12] Lee R Dice. Measures of the amount of ecologic association between species. Ecology, 26(3):297-302, 1945.
[13] Alexey Dosovitskiy and Josip Djolonga. You only train once: Loss-conditional training of deep networks. International conference on learning representations, 2020.
[14] Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. The importance of skip connections in biomedical image segmentation. In International Workshop Deep Learning Medical Image Analysis, International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis.
[15] Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. The importance of skip connections in biomedical image segmentation. In Deep learning and data labeling for medical applications, pages 179-187. Springer, 2016.
[16] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 3(7) 2018.
[17] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Neural architecture search: A survey. The Journal of Machine Learning Research, 2991):1977-2017, 2019.
[18] Bruce Fischl. Freesurfer. Neuroimage, 62(2):774-781, 2012.
10

---

[19] Trevor Gale, Erich Elsen, and Sara Hooker. The state of sparsity in deep neural networks, 2019.
[20] Benjamin Graham. Fractional max-pooling arXiv preprint arXiv: 1412.6171.677 2014.
[21] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: arXiv:/609.09 1609.09106,
2016.
[22] Song Han, Huizi Mao, and William J. Dally. Deep compression:
Compressing deep neural network with pruning, trained quantization and huffman coding. In Yoshua Bengio and Yann LeCun, editors, 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[23] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages
2015.
[24] Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[25] Andrew Hoopes, Malte Hoffman, Douglas N. Greve, Bruce Fischl, John Guttag, and Adrian Dalca. Learning the effect of registration hyperparameters with hypermorph. Machine Learning for Biomedical Imaging, 1:1-30, 2022.
[26] Jie Hu, Li Shen, and Gang Sun. gquezze-and-excitttin networks. Proceedings the IEEE conference computer vision and patlern recognition, pages 7132-7141, 2018
the
[27] Gao Huang. Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708. 2017.
[28] Itay Hubara. Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 41 4114-4122 2016
[29] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-baa based biomedical image segmentation. Nature methods, 18(2)200-211, 2021.
[30] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nuu-net: self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 2822)200-211, 2021.
[31] Fabian Isensee, Jens Petersen, Andre Klein, David Zimmerer, Paul F Jaeger, Simon Kohl, Jakob Wasserthal, Gregor Koehler, Tobias Norajitra, Sebastian Wirkert, et al. nnu-net: Self-adapting framework for u-net-based medical image segmentation. arXiv preprint arXiv: 10486, 2018.
[32] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. CVPR, 2017.
[33] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient intege--rrithmetic-nnf inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2704-2713, 2018.
[34] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint arXiv:1405.386 2014.
[35] Steven A. Janowsky. Pruning versus clipping in neural networks. Physical Review A. 39(12):6600-6600 June 1989.
[36] Chen Jin, Ryutaro Tanno, Thomy Mertzanidou, Eleftheria Panagiotaki, and Daniel C. Alexander. Learning to downsample for segmentation of ultra-high resolution images. arXiv preprint arXiv:2109.11071 2021.
11

---

[37] Asifullah Khan, Anabia Sohail, Umme Zahoora, and Aqsa Saeed Qureshi. A survey of the recent architectures of deep convolutional neural networks. Artificial Intelligence Review, 53(8):5455-5511 2020.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[39] Sylwester Klocek. Lukasz Maziarka, Maciej Wotczyk, Jacek Tabor, Jakub Nowak, and Marek Smieja. Hypernetwork functional image representation. In International Conference on Artificial Neural Networks, pages 496-510. Springer, 2019.
[40] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[41] David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks. arXiv preprint arXiv: arXiv: 1710.04799 2017.
[42] Jason Kuen, Xiangfei Kong, Zhe Lin, Gang Wang, Jianxiong Yin, Simon See, and Yap-Peng Tan. Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Pultern Recognition, pages 7929-7938, 2018
[43] Sarah Leclere, Erik Smistad, Joao Pedrosa, Andreas Østvik, Frederic Cervenansky, Florian Espinosa, Torvald Espeland, Erik Andreas Rye Berg, Pierre-Mare Jodoin, Thomas Grenier, et al. Deep learning for segmentation using an open large-scale dataset in 2d echocardiography. IEEE Iransactions on medical imaging, 38(9):2198)) (99))))199-2210, 2019.
[44] Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2117-2125, 2017.
[45] Shikun Liu, Zhe Lin, Yilin Wang, Jianming Zhang, Federico Perazzi, and Edward Johns, Shape adaptor: A learnable resizing module. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[46] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3296-3305, 2019.
[47] Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypernetworss. arXiv preprint arXiv: 1802.09999 2018.
[48] Andrew L Maas, Awni Y Hanmun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Citeseer, 2013.
[49] Matthew MacKay, Paul Vicol, Jon Lorraine, David Duvenaud, and Roger Grosse. Self-tuning networks: Bilevel optimization of hyperparameters using structured best-response functions. arXiv preprint arXiv; 03088, 2019.
[50] Daniel S Marcus, Tracy H Wang, Jamie Parker, John G Csernansky, John C Morris, and Randy L. Buckner. Open access series of imaging studies (oasis): cross-sectional mri data young, middle aged, nondemented, and demented older adults. Journal of cognitive neuroscience, 1989)1148-15577,, 2007.
[51] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571. IEEE, 2016.
[52] Ozan Oktay, Jo Schlemper, Loic Le Folgoc, Matteew Lee, Mattias Heinrich, Kazunari Misawa, Kensaku Mori, Steven McDonagh, Nils Y Hammerla, Bernhard Kainz, et al. Attention u-net: Learning where to look for the pancreas. arXiv preprint
2018.
12

---

[53] Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1111.01997. 2017.
[54] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville, Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
[55] Ilija Radosavovic, Justin Johnson, Saining Xie. Wan- Yen Lo, and Piotr Dollár. On network design spaces for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 1882-1890, 2019.
[56] Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pages 225-542.55 Springer, 2016.
[57] Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour: Learning strides in convolu- tional neural networks. In International Conference on Learning Representations, 2022.
[58] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisterr assisted intervention, pages 234-241. Springer, 2015.
[59] Leonardo Rundo, Changhee Han, Yudai Nagano, Jin Zhang, Ryuichiro Hataya, Carmelo Militello, Andrea Tangherloni, Marco S Nobile, Claudio Ferretti, Daniela Besozzi, et al. Use- net: Incorporating squeeze-and-excitaton blocks into u-net for prostate zonal segmentation of multi-institutional mri datasets. Neurocomputingg 065:31-43, 2019.
[60] Jürgen Schmidhuber. A 'self-referential weight matrix. In International Conference on Artificial Neural Networks, pages 446-450. Springer, 1993.
[61] Joan Serrà, Santiago Pascual, and Carlos Segura. Blow: a single-scale hyperconditioned flow for non-parallel raw-audio voice conversion. arXiv preprint arXiv:1906.0077 2019.
[62] Nahian Siddique, Paheding Sidike, Colin Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: theory and applications. arXiv preprint arXiv:2011.01116 2020.
[63] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin Riedmiller. Striving for simplicity: The all convolutional net. arXiv preprint arXiv:1412.6806, 2014.
[64] Carole H Sudre, Wenqi Li, Tom Vercauteren, Sebastien Ourselin, and M Jorge Cardoso. Gener- alised dice overlap as a deep learning loss function for highly unbalanced segmentations. In Deep Learning in Medicul Image Analysis and Multimodal Learning for Clinical Decision Support: Third International Workshop, DLMIA 2017, and 7th International Workshop, MI.- CDS 2017, Held in Conjunction with MICCAI 2017, Québec City, QC, Canada, September 14, Proceedings 3. pages 240-248. Springer, 2017.
[65] Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel Emer. Efficient processing of deep neural networks: A tutorial and survey. arXiv preprint arXiv:1703.09039 2017.
[66] Mingxing Tan and Quoe V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1194, 2019.
[67] Kenya Ukai, Takashi Matsubara, and Kuniaki Uchara. Hypernetwork- based implicit posterior estimation and model averaging of enn. In Asian Conference on Machine Learning, pages 176-191. PMLR, 2018
[68] Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F Grewe. Continual learning with hypernetworks. arXiv preprint arXiv: 006600695, 2019.
[69] Alan Q Wang, Adrian Dalca, and Mert R Sabuneu. Hyperrecon: Regularization-agnostie es- mri reconstruction with hypernetworks. In Machine Learning for Medical Image Reconstruction: 4th International Workshop, MIMIR 2021, Held in Conjunction with MICCAI 2021, Strasbourg, France, October 1, 2021, Proceedings 4, pages 3-13. Springer, 2021.
13

---

[70] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[71] Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-awadd pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5687-5695, 2017.
[72] Chris Zhang. Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprini arXiv:/810.05745 2018.
[73] Dominic Zhao, Johannes von Oswald, Seijin Kobayashi, João Sacramento, and Benjamin Grewe. Meta-lcarning via hypemetworks. 2020.
[74] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2881-2890 2017.
[75] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2881-2890 2017.
[76] Xin Zheng, Yong Wang, Guoyou Wang, and Jianguo Liu. Fast and robust segmentation of white blood cell images by self-supervised learning. Micron, 107:55-71, 2018.
[77] Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei- Shi Zheng, Jonathan Li, and Alexander Wong. Squeeze-and and-attention networks for semantic segmentation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13065-13074, 2020.
[78] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018
14


Reviews:
[{"review_id": "Zly2w1TfRz", "reviewer": "Reviewer_GRpN", "summary": "The paper presents Scale-Space HyperNetworks (SSHN), a method that predicts the weights for a segmentation network for range of rescaling factors. The proposed approach makes it possible to characterize the trade-off between model accuracy and inference efficiency faster, reducing the overall computational cost. Further, the paper demonstrates SSH demonstrate improved generalization on a variety of medical imaging tasks and datasets.", "questions": "Please look at weakness.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Employing a function h with learnable parameters to map the rescaling ratio to a set of convolutional weights is an interesting and novel approach and the overall framework.\n- The paper is evaluated across two critical medical image analysis tasks and is demonstrated to perform better compared to fixed and other variable resizing methods. \n- Paper also demonstrate efficiency and other analysis, including varying prior width and weight transferability, which allows deeper understanding of the framework and I'm confident that future works in this direction can benefit from such analysis. \n\n", "weaknesses": "The weakness that I found in this work is a limited explanation of the method, which might create confusion and readability issues for the readers. Here are a few suggestions/questions to address them:\n1. The implementation section should be clearly explained and might need to be expanded to include clear differences between how the network is trained and how it is used in inference or in evaluation. This is important because, in the later part of the paper, there are descriptions (line 198) like \"once trained, we use the hyper network to rapidly evaluate a range of phi..\". \n2. The paper mentions how a hyper-network has more parameters. It would be nice to indicate how much more those parameters.\n3. In Fig 2, the hyper network branch goes to both the encoder and decoder. But since there is a concat layer, how can a variable resizing factor be used? \n4. The training time result should be included in the main text as this further strengthens the argument of the paper. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the constructive feedback and comments. Addressing the\nraised questions:\n\n1.  **Model Development** - We will revise the manuscript to better\n    explain the model development process and to differentiate between\n    training and inference. To further clarify, the steps we take are:\n\n    -   Training - The hypernetwork is trained by generating the primary\n        network weights from randomly sampled rescaling factors.\n\n    -   Scale selection - Once trained, the hypernetwork is used once per rescaling factor to predict the weights of each primary network. These weights are used to evaluate the accuracy on a held-out set of data, producing a Pareto accuracy-efficiency frontier.\n    A rescaling factor is then chosen based on the trade-off characteristics, which determines which primary network parameters will be used.\n\n    -   Inference - A single set of primary network weights is used for\n        inference at the chosen rescaling factor. The hypernetwork is no\n        longer needed at this point, and does not contribute to the\n        inference computational cost.\n\n    We will revise the text to make this distinction more explicit and\n    clear.\n\n2.  **Parameter Counts** - We will incorporate additional detail about\n    the parameter counts. In our experiments, the hypernetwork model has\n    approximately 100x more learnable parameters than the Fixed\n    baseline. Importantly, however, the hypernetwork predicts the\n    weights of a primary network that has identical structure as the\n    Fixed baselines.\n\n3.  **Decoder Resizing** - For resizing layers in the decoding branch,\n    the feature maps are resized to match the spatial dimensions of the\n    tensors from the skip connections that are concatenated afterwards.\n    We will revise the text to better describe this detail of the\n    implementation.\n\n4.  **Training Runtime** - We will revise the results section to\n    incorporate training runtimes.\n\n\n"}, {"review_id": "jjwkQ94IvI", "reviewer": "Reviewer_veWP", "summary": "The paper introduces a method that learns a spectrum of CNNs with different rescaling factors. The method relies on using a hyper-network to generate the parameters of the model for a given rescaling factor --- this enables the users to choose the desired accuracy-efficiency trade-off with a single architecture. \n\nWhile the method is general, the authors demonstrate in two challenging structured prediction tasks (namely, segmentation and registration) the benefits of their approach in terms of accuracy-efficiency trade-off while incurring a small amount of extra training cost. ", "questions": "- What are the additional computational costs of generating weights through a hyper-network at inference time? Is this included in the calculation of inference costs in Figure7? \n- It is unclear why FiLM is a sensible baseline to compare against.\n- Is the cost of training a single SSHN smaller than training multiple CNNs independently for fixed rescaling factors? If so, this could be highlighted in the manuscript. \n- While the simplicity of the proposed approach is a strength rather than a weakness, one could envision setting different rescaling factors in different parts of the architecture. Such extension seems almost trivial implementationally, but training the hyper-network may become more challenging. It would be great to hear your thoughts on the potential benefits and risks of increasing such degrees of freedoms.\n", "limitations": "N/A", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "\n- The proposed method is a creative adoption of hyper network to the practical challenge of enabling the users to choose the desired accuracy-efficiency trade-off at inference time\n- Moreover, a comprehensive set of experiments demonstrate SSHN attain considerably better accuracy-efficiency trade-off than relevant baselines in multiple datasets for two challenging tasks\n- The manuscript is written very clearly and its motivation is strong.", "weaknesses": "- I would like to hear a more detailed explanation as to why learning a single model with varying rescaling factors lead to a consistent improvement in accuracy over the models trained with a fixed rescaling factor.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the constructive feedback and comments. Addressing the\nraised questions:\n\n### Weaknesses:\n\n-   **Accuracy Improvements** - This is an important question, and we\n    don't have a definitive answer. However, the results of the\n    experiment described in Section 5.2, suggest that varying the\n    resolution of intermediate features in the network induces a\n    regularization effect similar to how image scaling techniques in\n    data augmentation pipelines regularize learning.\n\n### Questions:\n\n-   **Inference Costs** - We would like to clarify how the model\n    development process is performed in our method:\n\n    1.  Training - The hypernetwork is trained by generating the primary\n        network weights from randomly sampled rescaling factors.\n\n    2.  Scale selection - Once trained, the hypernetwork is used once per rescaling factor to predict the weights of each primary network. These weights are used to evaluate the accuracy on a held-out set of data, producing a Pareto accuracy-efficiency frontier.\n    A rescaling factor is then chosen based on the trade-off characteristics, which determines which primary network parameters will be used.\n\n    3.  Inference - A single set of primary network weights is used for\n        inference at the chosen rescaling factor. The hypernetwork is no\n        longer needed at this point, and does not contribute to the\n        inference computational cost.\n\n    Therefore, since the hypernetwork is not used for performing\n    inference, Figure 7 only includes the computational costs of the\n    primary network. We tried to explain this in our method section\n    (L117-119), and will revise our manuscript to better clarify this in\n    the result section as well.\n\n-   **FiLM baseline** - We compare against FiLM because recent work (You\n    only train once: Loss-conditional training of deep networks \\[13\\])\n    employed FiLM modules as a way to perform amortized learning across\n    multiple loss weightings, efficiently characterizing the loss Pareto\n    frontiers. To the best of our knowledge, this is the closest\n    baseline for the problem statement of learning a family of models in\n    an amortized way.\n\n-   **Training Cost** - Yes, it takes approximately 10x less time to\n    train a single SSHN than to train the set of Fixed baselines\n    independently. We report training times in Table 3 of the\n    supplement. We will revise the text to better highlight this fact.\n\n-   **Different Rescaling Factors** - We agree with the observation. We\n    carried out this experiment and report results for the OASIS dataset\n    in Section C.2 of the supplement. We found that while it is feasible\n    to train a hypernetwork model with separate rescaling factors, it\n    fails to meaningfully improve the accuracy-efficiency Pareto\n    frontier while requiring longer training times to converge.\n\n\n"}, {"review_id": "rz3dqGAsBq", "reviewer": "Reviewer_wkRF", "summary": "CNNs, particularly those handling 3D data, can pose computational challenges due to their high expense. To tackle this, researchers frequently scale down the input data, a practice that often compromises accuracy. This paper presents SSHN, a technique designed to learn a variety of CNN models, each with unique scaling factors. With a marginal increase in training duration, SSHN is demonstrated to enhance the trade-off between accuracy and efficiency, making it a promising solution to the limitations of conventional methods.", "questions": "1. At the moment, training time has been only discussed in the supplementary material. Can you comment on the computational burden of SSHN and its implications? Would this warrant updating the limitations section?\n2. Could you provide information on the memory footprint of training the SSHN in the volumetric data regime? Would it be feasible?\n3. Could you elaborate on how the Fixed baseline is used during inference time and how the rescaling factor is chosen?\n4. Have 3D datasets in your experiments been treated as 2D datasets? Have you considered using a 3D U-Net as the primary model?", "limitations": "The authors have discussed certain limitations including the choice of model architecture and tasks. However, the authors could further elaborate on how method's training time and memory footprint for volumetric data, as this could be a significant limitation for practical applications.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is well-written, with clear visuals and figure captions. The clarity of the text aids the understanding of the proposed concepts.\n* The authors have chosen a reasonable variety of datasets for evaluation, strengthening the validity of their claims.\n* Baseline approaches considered in the paper are relevant, providing a solid ground for comparison.\n* The authors have diligently reported training five randomly initialized models, ensuring their findings are not reliant on a specific random seed.", "weaknesses": "* The paper states that CNNs are computationally intensive at inference time. However, in this reviewer's experience, the training time is often the more challenging aspect, especially with volumetric data. This warrants further clarification.\n* The memory footprint of SSHN in the volumetric data regime is usually a big concern, as it often leads to memory consumption issues. It is understood that the authors did not investigate this aspect.\n* It is unclear how the Fixed baseline was utilized during inference. For instance, it is not specified how the best rescaling factor is chosen for a specific dataset, or if the models are used as an ensemble.\n* The handling of 3D datasets (such as OASIS and CAMUS) in experiments is unclear. It is uncertain whether they are treated as 2D datasets, and if the authors have considered at all using a 3D U-Net as their primary model. If datasets were used as 2D images, this must be mentioned in the limitations section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the constructive feedback and comments. Addressing the\nraised questions:\n\n1.  **Training Cost** - We would like to clarify how the model\n    development process is performed with our method:\n\n    -   Training - The hypernetwork is trained by generating the primary\n        network weights from randomly sampled rescaling factors.\n\n    -   Scale selection - Once trained, the hypernetwork is used once per rescaling factor to predict the weights of each primary network. These weights are used to evaluate the accuracy on a held-out set of data, producing a Pareto accuracy-efficiency frontier.\n    A rescaling factor is then chosen based on the trade-off characteristics, which determines which primary network parameters will be used.\n\n    -   Inference - A single set of primary network weights is used for\n        inference at the chosen rescaling factor. The hypernetwork is no\n        longer needed at this point, and does not contribute to the\n        inference computational cost.\n\n    In our experiments, training a SSHN model takes 10x less time than\n    training the set of Fixed baselines, and is only 1.8x more costly than\n    training a single U-Net model. Once trained, the hypernetwork\n    is only used once to generate the weights of the primary network,\n    which are then used to make the predictions for different inputs, so\n    the computational cost is identical between SSHN and Fixed for a\n    given rescaling factor.\n\n2.  **Memory Footprint** - We agree that we should say more about memory\n    consumption. Our memory consumption measurements results follow\n    similar trends to the FLOP measurements. Models with smaller\n    rescaling factors substantially reduce memory consumption while\n    maintaining predictive quality for a range of rescaling factors. At\n    training time we sample $\\varphi \\sim \\mathcal{U}(0,0.5)$, so our\n    memory consumption is marginally more than that of a regular U-Net.\n    At inference time, since we do not use the hypernetwork, our memory\n    consumption is no more than that of a regular U-Net.\n\n3.  **Fixed baseline** - In our experiments, we do not chose a specific\n    rescaling factor for each dataset, as that depends on the downstream\n    accuracy-efficiency considerations. Therefore, in our experiments,\n    we compare methods using the entire accuracy-efficiency frontier.\n    For the Fixed baseline, we evaluate models independently, with each\n    one of them corresponding to a separate rescaling factor and\n    reported an individual datapoint (Figures 3,6 and 7).\n\n4.  **3D Data** - Yes, in our experiments, we used 2D mid-slices for the\n    3D datasets. We do this because 3D models are more computationally\n    demanding, and the Fixed baseline requires training many individual\n    models with different rescaling factors. Our method can be applied\n    to 3D U-Net models, where the computational and memory improvements\n    might turn out to be even more significant. We believe this is an\n    interesting area of future research.\n\n\n"}, {"review_id": "0QZhEm0PZE", "reviewer": "Reviewer_Lhzq", "summary": "This paper proposes to learn a spectrum for CNNs with varying internal rescaling factors and demonstrates the effectiveness of the proposed approach in several medical image analysis applications including segmentation and registration with fixed and dynamic rescaling factors. Overall, the approach is simple but sound and powerful, and the empirical results clearly support the claim. I recommend the paper for acceptance.", "questions": "- How would your results change if you used an architecture more recent than U-Net for the tasks you evaluated on?\n- Your proposed method seems to have a more smooth behavior compared to the other benchmarked resizing methods (Stochastic, FiLM) in Figure 3 -- do you have an intuition on why that might be?", "limitations": "Limitations are adequately addressed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Very clear presentation.\n- Strong empirical results including one that nicely uncovers that many rescaling factors lead to similar results despite having substantially different inference costs for a variety of medical imaging tasks.\n- Simple but powerful method that can characterize the trade-off between model accuracy and inference efficiency faster and better than the existing approaches.", "weaknesses": "n/a", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the constructive feedback and comments. Addressing the\nraised questions:\n\n-   **Primary Network Architecture** - We perform an architecture\n    ablation experiment in Section C.1 of the supplement. For the OASIS\n    dataset, we find similar trends to the ones of the U-Net\n    architecture.\n\n-   **Smooth behaviour** - We believe this is because the hypernetwork\n    formulation lets our model adapt better to changes in resolution. We\n    explored a related aspect in our *Weight transferability* experiment\n    in Section 5.2, where we find that SSHN weights transfer well to\n    neighboring rescaling factors.\n\n\n"}, {"review_id": "qW4gJsMsWz", "reviewer": "Reviewer_VkYJ", "summary": "The authors propose a unified approach based on Hypernetworks (HN) to model the accuracy-efficiency pareto front for medical applications.\nThe authors claim the following contributions:\n- Introducing Scale-Space HyperNetworks (SSHN) a single model that given a rescaling factor generates weights for corresponding model with reduced spatial dimensions of intermediate features.\n- Demonstrating the performance of SSHN on varing datasets while reducing the number of FLOPs by up to 50%.\n", "questions": "- Line 105 - instead of $(C,\\lceil\\varphi H\\rceil,\\lceil \\varphi H \\rceil)$ should be $(C,\\lceil\\varphi H\\rceil,\\lceil \\varphi W \\rceil)$.\n- Line 128-129 adding $R_\\varphi$ will make it clearer for the reader, in addition, consider moving these lines to ~Line 105 where you first present the notation $R_\\varphi$.\n- Why not normalizing $\\varphi$ such that $[0, 0.5] \\rightarrow [0,1]$ this will increase $\\varphi$’s resolution.\n- Did you try to use different prior over $\\varphi$, $p(\\varphi)$ instead of uniform?\n", "limitations": "See weaknesses and questions.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is well-written and easy to follow.\n- It tackles an important problem of computational heterogeneity during inference.\n- The proposed method is simple and easy to implement/reproduce.\n", "weaknesses": "- The authors failed to cite pioneering HN works [1,2,3]. Specifically [1] which is highly relevant and related to this work.\n- The novelty of this paper is limited. Pareto Front Learning (PFL) [1] is a widely explored field within machine learning, encompassing various lines of research. It has gained significant recognition and has a well-established presence due to its diverse range of investigations and studies. PFL is a computational approach that aims to find the optimal trade-offs between multiple conflicting objectives. It involves identifying a set of solutions that lie on the Pareto front, representing the best possible outcomes for each objective, without sacrificing performance in other areas. By exploring the Pareto front, decision-makers can make informed choices that balance competing objectives and achieve more comprehensive and balanced solutions. It seems like the paper’s core novelty has already been presented by previous works. I suggest the authors to better explain how their approach differs from highly related works in PFL [1,4].\n- I wonder how well the proposed approach works on non-medical datasets e.g. Cityscapes, NYU, etc.\n- How well the model generalizes to unseen scales i.e. scales that are not in the range of $p(\\varphi)$?\n- I suggest the authors add a section regarding their design choices for example (i) the prior scaling factor and (ii) the rescale module $R_\\varphi$. Ablation experiments should be included as well in the main manuscript or supplementary.\n\nCitations:\n\n[1] Learning the Pareto Front with Hypernetworks, Navon et al.\n\n[2] HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editing, Alaluf et al.\n\n[3] Personalized Federated Learning using Hypernetworks, Shamsian et al.\n\n[4] Controllable Pareto Multi-Task Learning, Lin et al.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe appreciate the constructive feedback and comments. Addressing the\nraised questions:\n\n### Weaknesses:\n\n-   **Prior Work** - We agree that the work cited in the review is\n    relevant to our method, and we will revise our manuscript to draw\n    the connections and contextualize the research.\n\n-   **Pareto Front Learning** - The goal of our work is substantially\n    different from that of the referenced work \\[1\\]. In particular, PFL\n    focuses on jointly optimizing the Pareto frontier of multiple\n    predetermined training objectives across a single architectural\n    setting. In contrast, our approach is about optimizing a single\n    objective (in our case segmentation accuracy) across a range of\n    architectural settings that introduce various computational\n    trade-offs. To the best of our knowledge, our work is the first to\n    explore amortizing the cost of training models with different\n    internal rescaling factors, a key aspect in convolutional model\n    efficiency, and that has not been explored in any of the references.\n    The technical obstacles associated with our goal are different from\n    those faced by PFL. For us, metrics of interest like computational\n    cost (FLOPs), required memory, or model latency are not\n    differentiable, and thus cannot be easily incorporated in the\n    learning process in the way the cited work does. Our approach allows\n    practitioners to use the learned hypernetwork to efficiently\n    generate the accuracy-cost frontier for their given hardware\n    constraints, and then choose the model to be deployed based on\n    validation metrics that do not need to be defined during the\n    training process.\n\n-   **Non-medical Datasets** - Our work focuses on medical datasets\n    since in this domain it is common to train models from scratch,\n    which is often computationally intensive. In contrast, natural image\n    segmentation models typically use a pretrained encoder backbone trained at a fixed\n    set of resolutions.\n\n-   **Scaling shift** - Our model performs best when evaluated with\n    rescaling factors within $p(\\varphi)$. Outside this range\n    segmentation performance degrades. However, the assumption is that\n    if a scale is of interest it should be in the range of $p(\\varphi)$.\n\n-   **Ablations** - We include ablation experiments regarding the model\n    architecture in Section C.1 of the manuscript. We will provide\n    further design details regarding the resizing module $R_\\varphi$.\n    However, we are unsure about what alternative choices of rescale\n    module  $R_\\varphi$ would be of interest. We believe that bilinear\n    interpolation is the most prevalent fractional resizing mechanism in\n    the literature.\n\n### Questions:\n\n-   We are thankful for the corrections, and we will revise the\n    manuscript to include them.\n\n-   We do normalize $[0, 0.5] \\rightarrow [0, 1]$ and then apply a\n    $(x, 1-x)$ encoding. We will revise our implementation details.\n\n-   We did explore using an *area uniform* prior\n    $p(\\varphi) \\sim \\sqrt{\\mathcal{U}(0,1)}$ but we did not find any\n    significant differences in the produced Pareto curves.\n\n"}]

Author rebuttal:

Meta-review: The paper was reviewed by 5 reviewers. On the one hand, the reviewers appreciated the ideas presented in the paper, found the presentation of the paper clear and highlighted the potential relevance of the network parameter prediction for diverse rescaling factors in medical imaging segmentation.  On the other hand, the reviewers were concerned with the reported experimental results, brought up several relevant prior works and pointed out missing implementation details. Authors submitted the rebuttal. In the post rebuttal phase, four out of five reviewers were inclined towards acceptance. The main criticism of unconvinced reviewer were around paper's novelty and validation. The reviewer brought up relevant prior works. AC agrees with the reviewer that brought up work is indeed relevant. Validation could also be strengthened. The main points of validation criticisms are related to potential underfitting of baselines as the fixed setup is under-performing w.r.t. SSHN. Adding an ablation over the baseline parameters (with parameters growing to the capacity of SSHN) would make the submission stronger. AC concludes that given current experimental setup one of the claims of the paper about the superiority of SSHN ( "... SSHN consistently outperforms regular models by 2-3% for the same rescaling factor.") might not have enough experimental support. Overall, the arguments towards acceptance outweigh the reasons to reject this submission. Thus AC recommends to accept. The AC expects the authors to introduce the reviewers suggestions in the final version of their manuscript. Moreover, the AC asks the authors to (1) either modify the claim in question or include an ablation over parameters to ensure that the baselines are not underfitting to the data and (2) properly credit the related works brought up by the reviewers.
Decision: Accept (poster)