# Evaluating DL Model Scaling Trade-Offs During Inference via an Empirical Benchmark Analysis
###### Abstract  
With generative Artificial Intelligence (AI) capturing public attention, the appetite of the technology sector for larger and more complex Deep Learning (DL) models is continuously growing. Traditionally, the focus in DL model development has been on scaling the neural network's foundational structure to increase computational complexity and enhance the representational expressiveness of the model. However, with recent advancements in edge computing and 5G networks, DL models are now aggressively being deployed and utilized across the cloud-edge-IoT continuum for the realization of in situ intelligent IoT services. This paradigm shift introduces a growing need for AI practitioners, as a focus on inference costs, including latency, computational overhead, and energy efficiency, is long overdue. This work presents a benchmarking framework designed to assess DL model scaling across three key performance axes during model inference: classification accuracy, computational overhead, and latency. The framework's utility is demonstrated through an empirical study involving various model structures and variants, as well as publicly available datasets for three popular DL use cases covering natural language understanding, object detection, and regression analysis. deep learning; artificial intelligence; cloud computing; benchmarking +
Footnote †: journal: Computer Science  
the computational complexity of the network to increase the representational power of the DNN and improve accuracy while also generalizing better over newly introduced tasks [14]. A recent study by OpenAI revealed that since 2012, the amount of computational effort required by DL models has increased exponentially and doubled every 4 months [15]. Nonetheless, while IoT and edge computing hardware are vastly improving, they are not doing so at a rate capable of catering to the advancements in deep learning [16; 17]. Hence, although the AI community was, in the past, focused solely on improving model accuracy metrics, a paradigm shift is now being observed where runtime performance during inference and energy efficiency are gaining industry interest [18; 19]. This brings us to the focal point and motivation that support our work. With so many different scaling strategies for DNNs, it becomes extremely difficult for users to select a model variant that will meet different QoS requirements. Models can scale in terms of depth, width, and clarity of input data, and different parameters can be set (e.g., batch size and training epochs). All these configurations can significantly impact the performance of a DL application. The majority of existing works that empirically evaluate model scaling focus on either a single model structure [13; 20] and/or introduce findings that are tailored to solely evaluating accuracy and overhead during the model training stage [21; 22]. We deviate from the norm and focus on the impacts of different model scaling strategies during runtime inference. Specifically, we focus on three key performance axes: classification accuracy, computational overhead, and latency. By offering a detailed examination of trade-offs during model inference, our work aims to guide AI practitioners in selecting and tuning DL models for efficient and effective deployment of intelligent IoT services in resource-constrained environments. The main contributions of our work are summarized as follows:  
* We introduce a high-level description of a benchmarking framework for the assessment of the various trade-offs that occur when scaling the underlying network of a DL model across different dimensions (i.e., depth and width). This modular and extensive framework provides performance insights on the impact of DL model scaling during runtime inference in light of classification accuracy and model loss, computational overhead, and the model's latency footprint. * We demonstrate the utility of our framework by introducing an empirical study evaluating DL model scaling strategies and their impacts on the QoS of runtime inference.