SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Computational Cost. We analyze how each rescaling factor affects the inference computational requirements of each segmentation network. We measure the number of floating point operations (FLOPs) required to use the network for each rescaling factor at inference time. We also evaluate the amount of time used to train each model.
Figure 7 depicts the trade-off between test Dice score and the inference computational cost for
Table 5: Segmentation Results. Test Dice score values for the different methods evaluated on the considered segmentation tasks. Standard Deviation is reported across training replicas.
Method	CAMUS	OASIS	PanDental	WBC
Fixed	83.5 ± 0.3	89.1 ± 0.0	88.9 ± 0.2	90.9 ± 0.6
Stochastic	83.3 ± 0.1	84.8 ± 0.2	85.6 ± 0.7	88.1 ± 2.0
FiLM	83.2 ± 0.3	87.2 ± 0.6	85.4 ± 0.6	87.1 ± 1.4
SSHN (ours)	86.2 ± 0.3	90.7 ± 0.1	89.9 ± 0.3	92.7 ± 0.4
Figure 6: Registration Results. Results for the learnable registration task on OASIS. Test set results as a function of the rescaling factor (left) and as a function of the computation cost, in GFLOPs (right) for the considered methods. The SSHN model matches the independently trained networks (Fixed) for most values of the rescaling factor  $\varphi$ .
all segmentation tasks, and Figure 6 shown the results in the registration task on OASIS. We observe that the choice of rescaling factor has a substantial effect on the computational requirements. For instance, for the OASIS and PanDental tasks, reducing the rescaling factor from the default  $\varphi = 0.5$  to  $\varphi = 0.4$  reduces computational cost by over  $50\%$  and by  $70\%$  for  $\varphi = 0.3$ . In all cases, we again find that hypernetwork achieves a better accuracy-FLOPs trade-off curve than the set of baselines. Table 1 in the supplement report detailed measurements for accuracy (in Dice score), inference cost (in inference GFLOPs) and training time (in hours) for all methods approaches.
Figure 7: Efficiency Results. Trade-off curves between test model accuracy (Dice score) and inference computational cost (GFLOPs). SSHN models successfully characterize the trade-off between model accuracy and efficiency while outperforming models trained with fixed amounts of feature rescaling (Fixed). In all datasets, the inference computational cost of SSHN can be reduced by at least  $40\%$  without a significant change in model accuracy. Results are averaged across network initializations, and variance is indicated with shaded regions.
Because of identical inference cost (for a given rescaling factor) and improved segmentation quality of SSHN, its Pareto curve dominates the networks with fixed resizing. SSHN therefore requires less inference computation for comparable model quality. For example, for segmentation networks trained on OASIS, for  $\varphi \in [0.25, 0.5]$  there is no loss in quality, showing that a more than  $20\%$  reduction in FLOPS can be achieved with no loss of performance. SSHN achieves similar performance to the best baseline with  $\varphi = 0.15$ , while reducing the computational cost of inference by  $50\%$ . Characterizing the accuracy-efficiency Pareto frontier using SSHN requires substantially fewer GPU-hours than the traditional approach, while enabling a substantially more fine-scale curve. In our experiments, the set of baselines required over an order of magnitude ( $10\times$ ) longer to train compared to the single hypernetwork model, even when we employ a coarse-grained baseline search.
5.2 Analysis Experiments
In this section we perform additional experiments designed to shed light on why the hypernetwork models achieve higher test accuracy than the baseline models.
Varying Prior Width. We first study the quality of hypernetwork segmentation results with varying width of the prior distribution  $p(\varphi)$ . Our goal is to understand how learning with varying amounts of rescaling affects segmentation accuracy and generalization. We train a set of SSHN models on the OASIS dataset but with narrow uniform distributions  $\mathcal{U}(0.5 - r, 0.5 + r)$ , where  $r$  controls the width of the distribution. We vary  $r = [0, 0.01, \dots, 0.05]$  and evaluate them with  $\varphi = 0.5$  Figure 8 shows Dice scores for training and test set on the OASIS segmentation task, as well as a baseline UNet for comparison. With  $r = 0$ , the hypernetwork is effectively trained with a constant  $\varphi = 0.5$ , and slightly underperforms compared to the baseline UNet. However, as the range of the prior distribution grows, the test Dice score of the hypernetwork models improves, surpassing the baseline UNet performance.
Figure 8 suggests that the improvement over the baselines can be at least partially explained by the hypernetwork learning a more robust model because it is exposed to varying amounts of feature rescaling. This suggests that stochastically changing the amount of downsampling at each iteration has a regularization effect on learning that leads to improvements in the model generalization. Wider ranges of rescaling values for the prior  $p(\varphi)$  have a larger regularization effect. At high values of  $r$ , improvements become marginal and the accuracy of a model trained with the widest interval  $p(\varphi) = \mathcal{U}(0.4, 0.6)$  is similar to the results of the previous experiments with  $p(\varphi) = \mathcal{U}(0, 1)$ .
Weight transferability. We study the effect of using a set of weights that was trained with a different rescaling factor. Our goal is to understand how the hypernetwork adapts its predictions as the rescaling factor changes. We use fixed weights  $\theta' = h(0.5; \omega)$  as predicted by the SSHN model with input  $\varphi' = 0.5$ , and run inference using different rescaling factors  $\varphi$ . For comparison, we apply the same procedure to a set of weights of a Fixed U-Net baseline trained with a fixed  $\varphi = 0.5$ . Figure 9 presents segmentation results on OASIS for varying rescaling factors  $\varphi$ . Transferring weights from the baseline model leads to a rapid decrease in performance away from  $\varphi = 0.5$ . In contrast, the weights generated by the hypernetwork are more robust and can effectively be used in the range  $\varphi \in [0.45, 0.6]$ . Nevertheless, weights learned for a specific rescaling factor  $\varphi$  do not generalize well
Figure 8: Segmentation Dice on OASIS Test data as a function of the prior distribution range  $2r$ . Results are evaluated with  $\varphi = 0.5$  in all settings, regardless of the training prior, and we include a baseline UNet (Fixed) model. Uniform priors for  $\varphi$  are of the form  $\mathcal{U}(0.5 - r, 0.5 + r)$ , i.e. they linearly increase around 0.5. Standard deviation (shaded regions) is computed over 5 random initializations.
Figure 9: Segmentation results for models resulting from transferring weights learned for a given rescaling factor and evaluated at different factors. The convolutional parameters learned by the hypernetwork (T-SSHN) transfer to nearby factors substantially better than the weights of the baseline model (T-Fixed). Standard deviation (shaded regions) is computed over 5 random seeds.
to a wide range of different rescaling factors, whereas weights predicted by the hypernetwork transfer substantially better to other rescaling factors.
We evaluated our proposed framework on image segmentation and registration, a prevalent tasks in medical image analysis. We believe that our method can be extended to other tasks like object detection or classification. Constructing such extensions is an exciting area for future research. Furthermore, we focus on the U-Net as a benchmark architecture because it is a popular choice for segmentation networks across many domains [58, 15, 32, 31, 2, 4]. It also features key properties, such as an encoder-decoder structure and skip connections from the encoder to the decoder, that are used in other segmentation architectures [8, 74, 9]. Nevertheless, in Section C.1 of the Supplemental material, we report results for other popular segmentation architectures, including PSPNet, FPN and SENet, and find analogous results.
We introduce SSHN: a hypernetwork-based approach for efficiently learning a family of CNNs with different rescaling factors in an amortized way. Given a trained model, SSHN enables efficiently generating the Pareto frontier for the trade-off between inference cost and accuracy. This makes it possible to rapidly search for a rescaling factor that maximizes accuracy subject to computational cost constraints. For example, using the hypernetwork, we demonstrate that using larger amounts of downsampling can often lead to a substantial reduction in the computational costs of inference, while sacrificing nearly no accuracy. Finally, we demonstrate that for a variety of biomedical image analysis tasks and datasets, SSHN models models achieve accuracy at least as good, and more often better than, CNNs trained with fixed rescaling ratios, while producing more robust models. This strategy enables the construction of a new class of cost-adjustable models, which can be adapted to the resources of the user and minimize carbon footprint while providing competitive performance in biomedical image analysis.
This research is supported by Quanta Computer, Inc, and Wistron Coporation. Additionally, the project was supported by NIH R01AG064027 and R01AG070988.
[70] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[71] Tien-Ju Yang, Yu-Hsin Chen, and Vivienne Sze. Designing energy-efficient convolutional neural networks using energy-aware pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5687-5695, 2017.
[72] Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. arXiv preprint arXiv:1810.05749, 2018.
[73] Dominic Zhao, Johannes von Oswald, Seijin Kobayashi, João Sacramento, and Benjamin F. Grewe. Meta-learning via hypernetworks. 2020.
[74] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[75] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[76] Xin Zheng, Yong Wang, Guoyou Wang, and Jianguo Liu. Fast and robust segmentation of white blood cell images by self-supervised learning. Micron, 107:55-71, 2018.
[77] Zilong Zhong, Zhong Qiu Lin, Rene Bidart, Xiaodan Hu, Ibrahim Ben Daya, Zhifeng Li, Wei-Shi Zheng, Jonathan Li, and Alexander Wong. Squeeze-and-attention networks for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13065–13074, 2020.
[78] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8697-8710, 2018.