PePR: Performance Per Resource Unit as a Metric to Promote Small-scale Deep Learning

Venue: NLDL_org_2025_Conference
Published: 2024-09-05 14:09:26

Abstract: The recent advances in deep learning (DL) have been accelerated by access to large-scale data and compute. These large-scale resources have been used to train progressively larger models which are resource intensive in terms of compute, data, energy, and carbon emissions. These costs are becoming a new type of entry barrier to researchers and practitioners with limited access to resources at such scale, particularly in the Global South. In this work, we take a comprehensive look at the landscape of existing DL models for vision tasks and demonstrate their usefulness in settings where resources are limited. To account for the resource consumption of DL models, we introduce a novel measure to estimate the performance per resource unit, which we call the PePR score.  Using a diverse family of 131 unique DL architectures (spanning 1M to 130M trainable parameters) and three medical image datasets, we capture trends about the performance-resource trade-offs. In applications like medical image analysis, we argue that small-scale, specialized  models are better than striving for large-scale models. Furthermore, we show that using pretrained models can significantly reduce the computational resources and data required. 
We hope this work will encourage the community to focus on improving AI equity by developing methods and models with smaller resource footprints.

Main paper: 
 introduction: 
 The question of material costs of technology, even in light of their usefulness, should not be ignored [1] . This is also true for technologies such as deep learning (DL) that is reliant on large-scale data and compute, resulting in increasing energy consumption and corresponding carbon emissions [2] . These growing resource costs can hamper their environmental and social sustainability. [3, 4] .Considerations towards improving the environmental impact of DL are garnering attention across different application domains. This has resulted in calls for action broadly, and also within the medical 1 Pronounced pepper.2 Source code will be made available image analysis community, to improve the resource 045 efficiency of DL models [5, 6] and to report the 046 energy and carbon costs [7] . Another important im-047 plication of the growing resource costs of DL is the 048 risk of disenfranchising practitioners with limited 049 access to resources. This is captured in Figure 2 -050 a) which shows the number of publications (per 051 capita) within DL across the world for 2013 and 052 2022 3 . Several regions in the world categorised as 053Global South are continuing to lag behind in research 054 in DL [8] . While there are also multiple other struc-055 tural reasons for this trend, the increasing resource 056 costs of performing research within DL can become 057 a new form of entry barrier that can aggravate this 058 disparity [9] .In light of these observations, this work argues for 060 focusing on small-scale DL in the era of large-scale 061 DL. We hypothesize that the current situation with 062 the increasing resource consumption is due to the sin-063 3 The data for the visualisation in Figure 1 was curated by querying for the number of research publications per country on the topic of "Artificial Intelligence" in Ope-nAlex.org. The population data per country was queried from data.WorldBank.org. Regional aggregation was performed using OECD standards and further refined into the ten regions. Curated data will be provided along with the source code.gular focus on task-specific performance metrics that are not grounded in material costs. We also argue that access is a prerequisite to improving equity in DL and in use of these methods in healthcare. These arguments are supported by a comprehensive analysis of performance and resource costs of DL-based computer vision models. We study the properties of 131 models ranging from 1M to 130M trainable parameters, on three medical image classification tasks to capture interesting trends. We provide qualitative evidence for the usefulness of using pretrained models in resource-constrained regimes. Finally, we present a novel composite measure of performance and resource consumption. We call this the performance per resource unit (PePR) score. Using the PePR-score we characterise the behaviour of smallscale and large-scale DL models. We demonstrate that in resource-constrained regimes, small-scale DL models yield a better trade-off between performance and resource consumption.Related Work: Pareto optimisation of performance and resource constraints has been primarily investigated within the context of neural architecture search (NAS) [10] . More recently, methods have been proposed to explore models using specific resource constraints such as energy consumption [11, 12] or carbon footprint [13] . The work in [11] proposes a resource-aware performance metric similar to our contribution in this work which, however, is concerned with non DL models. Within application domains such as medical image analysis, there has been little emphasis on the joint optimisation of performance and energy consumption [14] . The question of equitable AI within healthcare has been posed in works like [15] primarily from the context of fairness and not from resource/access perspectives. 
pepr-score: 
 In this work, we assume a DL model to be an entity that consumes resources such as data, energy, time, or CO2eq. budget as input and provides some measurable predictive performance on downstream tasks of interest. In contrast to conventional performance metrics that are not grounded in material costs, we envision a score that can take the resource costs into account. To this end, we introduce the notion of performance per resource unit (PePR), denoted as P ePR ∶ [0, 1] × [0, 1] → R, which relates (normalised) performance P ∈ [0, 1] of a model with the resources consumed and defined asEQUATIONIn this definition, R is the resource cost normalised to lie in [0, 1], or explicitly R = (R abs -R min )/(R max -R min ) for some absolute re-source cost R abs and some R min , R max fixed across 117 models within an experiment. 
data & experiments: 
 To demonstrate the usefulness of the PePR-score, we curated a large collection of diverse, neural network architectures and experiment on multiple datasets.The model space used in this work consists of 131 neural network architectures specialised for image classification. The exact number of 131 architectures was obtained after seeking sufficiently diverse models which were also pretrained on the same benchmark dataset. We used the Pytorch Image Models (timm) model zoo to access models pretrained on ImageNet-1k resulting in 131 models spanning 1M to 131M trainable parameters. We randomly sub-sampled the available models in Pytorch Image Models library [17] , which during our experiments had about 700 models. We chose as many unique architectures as possible that were all pre-trained on the same ImageNet dataset. This resulted in the 131 models used in our work, covering CNNs, vision transformers, hybrid models, and efficient architectures.We categorise these models along two dimensions i) CNN or Other depending on if the architecture is a generic CNN primarily consisting of convolutional 5 Because of the bound for the second integrand in (2): f (0) 2 ≤ f (r) 1+r ≤ 1The model space is illustrated in Figure 2-c Experiments and Results: We performed three main experiments with our large collection of models: i) Study the influence of pretraining on test performance ii) Evaluate the role of number of training points iii) Compute PePR-E score and compare the trade-off between test performance and energy consumption as the cost. Results from all three experiments are summarized in Figure 3 .We had access to pretrained weights for all 131 models, which made it possible to investigate the influence of using pretraining when resources are constrained. We either fine-tune or train-from-scratch all models for 10 epochs. In Figure 3-a ), across the board, we notice that using pretrained models are significantly better compared to training models from scratch for the same number of epochs (p < 0.001).Another resource that can be lacking, on top of compute/energy, is the amount of training data. We study this by only using 10% of the training data, for each of the three datasets, and reporting the average test performance per model in Figure 3-b ). Even though there is a significant test performance difference (p < 0.001) when only using 10% of the data compared to using 100% of the data, it could be still useful in making some preliminary choices.The overall test performance averaged across the three datasets is plotted against the number of parameters, along with architecture classes, in Figure 3-c ). There was no significant group difference in test performance between small-and large-scale models. Similarly, there was no significant difference between models that are Efficient and Not Efficient, or between CNN and Other.Finally, in Figure 3-d ) we visualise the PePR-E score for all the models, which uses the energy Our experiments reported in Figure 3 and Table 1 re-veal interesting trends about the interplay between 344 test performance and resource consumption. We noticed no significant difference in performance between the small-scale and large-scale models in the regime where they were fine-tuned with pretrained weights for 10 epochs. This captures the problem with focusing only on test performance, as it could easily yield large-scale models even when small-scale models could be adequate. However, when using the PePR-E score, we see a significant performance difference with the small-scale models achieving a higher PePR-score (p < 0.05). This emphasises the usefulness of taking resource costs into account, which can be easily done using any variations of the PePR score.Energy, or other resource, consumption awareness can also be incorporated using multi-objective optimisation [12] . PePR score can be thought of as one way to access the solutions on the Pareto front with an emphasis on low-resource footprint. This is captured in Figure 4 which overlays the Pareto set (in orange) and all other models over the PePR scores. The knee point of this Pareto front is pointing towards maximising PePR-E score (brighter regions).PePR score is a composite metric that offers a trade-off between performance and resource consumption. It can be used instead of multi-objective optimisation of the two objectives separately. As shown in our experiments, PePR score can be used to compare models that use different extents of resources. Current reporting in deep learning for im- Limitations: We used a training or fine-tuning budget of 10 epochs in this work to reduce the compute resources used. This can be a limitation, as different models learn at different rates. To show that our experimental results are not artifacts of this choice, we looked at the performance of models that have been trained to convergence on ImageNet (which formed the basis of pre-training) using the public dataset from [17] . We performed a similar analysis of validation set performance of the converged models, The PePR-M scores are shown in Conclusions: Using a large collection of DL mod-412 els we have shown that using pre-trained models 413 yields significant gains in performance, and should 414 always be considered. We have also shown that 415 when resource consumption is taken into account, 416 small-scale DL models offer a better trade-off than 417 large-scale models. Specifically, the performance 418 achieved per unit of resource consumption for small-419 scale models in low-resource regimes is higher. We 420 proposed the PePR score that offers an inbuilt trade-421 off between resource consumption and performance. 422 The score penalises models with diminishing returns 423 for a given increase in resource consumption.Questions around how best to improve equity in 425 research and healthcare are neither easy nor straight-426 forward, go far beyond the ways in which we use 427 specific types of DL, and cannot be fixed through 428 technological solutionism [36] . Nevertheless, using 429 small-scale DL can help mitigate certain types of 430 inequities by reducing some of the barriers that are 431 currently in place for researchers and practitioners 432 with limited access to resources. Small-scale DL 433 can be developed and run on end-point consumer 434 hardware which is more pervasive than specialised 435 datacenters with high performance computing in 436 many parts of the world. With this work, we sin-437 cerely hope that by focusing on reducing the resource 438 costs of DL to improve access the larger question of 439 equity in DL for healthcare will be grappled with 440 by the community. Standard scaling might not always be appropriate. Outliers may have to be considered, and in other instances R min , R max might depend on the experimental set-up.PePR score is P ePR (R = 1, P = 1; α = 1) = 0.5. In-Vol. 34. 09. 2020, pp. 13693-13696. doi: 
118: 
 The salient features of the PePR-score that make 119 it useful as an alternative objective that takes re-120 source costs into account are as follows: We also note that in cases where performance is 160 deemed to be more important than resource con-161 sumption, PePR score can be adjusted to reflect 162 this. For instance, one can employ P ePR (R, P ; α) = 163 α ⋅ P /(α + R) with a scaling factor α ≥ 1. Setting 164 a large α value, say α = 100, would prioritise per-165 formance and disregard the effect of the resource 166 consumption. As an example, consider the PePR 167 score for the most resource intensive model that also 168 = 139 0, R 1 = 1 because R min = R 0 , R max = R 1 .P ePRc (R; f ) = P ePR (R, f (R)),(2)185where in cases of ties the smallest value is picked.186 Furthermore, in order to be able to compare models Since f (0) ≤ f (r), r ≤ 1, if we assume f to be increasing, we also have that PePR increases in intervals where f ′ > 1 and decreases in intervals whereP ePRc (R; f ) = f (0) + ∫ R 0 f ′ (r) 1 + r dr -∫ R 0 f (r) (1 + r) 2 dr.f ′ < f (0)/2.5 This captures the idea that the maxima of the PePR curve lie at points of diminishing returns as captured by f ′ , which is also visualized in Figure 2-b ). 
 
 Appendix:  


Decision: This paper introduces the PePR score, a metric designed to evaluate the trade-off between performance and resource consumption in deep learning models, with a particular focus on resource-constrained environments like medical imaging. The study is timely, tackling AI equity concerns by highlighting the resource efficiency of smaller models. The PePR score offers a novel, interpretable way to quantify resource use across various DL models, making a positive impact in promoting sustainable practices in AI.

However, the study’s reliance on short training durations and a specific domain (medical imaging) limits its broader applicability. Furthermore, the absence of statistical validation, alongside the reliance on pre-trained models, raises concerns about the robustness and flexibility of the findings. While the PePR score is a promising metric, extending evaluations to varied datasets, domains, and training regimes would strengthen its credibility and adoption potential.

Overall, the paper presents a valuable contribution, offering insights into efficient DL practices in resource-constrained settings, but would benefit from further validation and refinement to maximize its impact.

Reviews: review: title: title: Interesting problem, but more results would improve the clarity of the paper
paper_summary: paper_summary: This paper discusses the performance-resource tradeoff in small- and large-scale DL models. The authors shed light on the inaccessibility to large-scale computing and data, which hinders researchers' ability, especially in the global south and healthcare-AI sectors. The paper proposed a new metric that considers both performance and resources for DL models. An evaluation of more than 100 models for medical image classification datasets shows that small-scale models can perform similarly to large-scale models under resource-constrained regimes. The authors also claim that fine-tuning using a pretrained model should always be a first option rather than training from scratch.
main_review: main_review: The authors provided detailed responses in the rebuttals, which addressed several of my concerns. The paper contains interesting results and insights worth sharing with the ML research community. Hence, I recommend accepting the revised version of the paper.
strength_weakness: strength_weakness: - Although the paper is well-motivated, the authors haven't discussed the results more profoundly or mapped their observations onto realistic cases. For instance, additional metrics on the usage and adoption of specific models/datasets for healthcare systems per region would be interesting to emphasize the resource equity barrier problem highlighted by the authors.
- The PePR metric's intended usage is still unclear. How do the authors envisage using this metric in a multi-objective optimization problem, and what would be the realistic use case?
- The process of models selection for the study is not well detailed. What criteria did the authors use to choose CNN and other models? Given the variety of model architectures, training methods, and task complexity, How could the authors ensure a representative coverage of models, tasks, and datasets so as to prevent any unbiased results? This needs to be justified further.
- The paper lacks additional details on the fine-tuning method and the rationale behind the training for 10-epoch. What if the fine-tuning is performed for less/more than 10 epochs? Would the paper observations/results still hold?
- In Figure 3, it's clear that CNN and others show no difference, so such categorization is not needed. The scatter points can reflect DL models in general. 
- The discussion section could be more comprehensive (e.g., . what recommendation would the authors propose to tackle the equity problem)
- Ablation studies, particularly on fine-tuning methods and budget, models architectures and scale, and complexity of datasets, are crucial to confirm the generality of the authors' observations. This would further validate the robustness of their results.