# Evaluating DL Model Scaling Trade-Offs During Inference via an Empirical Benchmark Analysis
## 8 Conclusions and Future Work  
This article presents an empirical benchmarking study focused on understanding the correlation between DL model scaling and three performance axes during runtime inference. These axes cover accuracy, computational effort, and latency. This distinguishes our work from other studies that solely studied model accuracy and training duration during the training phase for different model variants. To this end, we designed and presented a prototype of a DL model scaling benchmark framework, along with details about its execution and functionalities. Moreover, we introduced three diverse DL use cases from the domains of natural language understanding, object detection, and regression analysis. All three use cases embrace open DNN model architectures, multiple and different scaling techniques, and the use of popular datasets for workload generation. With the three developed DL use cases, we then performed an empirical analysis with the use of our benchmark framework to discover various insights and trade-offs introduced across the three performance axes during runtime inference when examining multiple model variants per use case. Some key observations from the analysis are the following. Across all three DNN use cases, increasing model complexity generally led to improved inference quality--higher accuracy for classification tasks and lower MSE for regression tasks--but only up to a point. As model complexity continued to increase, the improvements began to plateau. Moreover, scaling the model complexity beyond a certain point that is both use-case and database-dependent, may lead to the model overfitting to the training data, capturing noise rather than meaningful patterns, which can degrade performance on new data. We also observed an almost linear relationship between model complexity and computational overhead (measured in FLOPs). In addition, the inference latency increased with model complexity, correlating with computational overhead, although the relationship was not perfectly linear due to additional factors like data pre-processing effort and memory access patterns influenced by model optimizations. Finally, our study highlights that models with the same complexity but different configurations can have significantly different impacts on accuracy and latency. Our future work includes several key areas of development to enhance the capabilities and applicability of our benchmarking framework. Initially, we plan to introduce new use cases covering more real-world problems that will include more complex DNN models, like LLMs or generally generative AI. To that end, we plan to integrate our system with online model repositories (e.g., HuggingFace) in order to further automate our analysis. Secondly, we envision an extension of our monitoring subsystem to capture not only DNN performance metrics but also underlying utilization metrics, e.g., compute utilization (GPU/CPU), memory usage, disk I/O, etc. This will reveal hidden correlations between the structures of specific DNN models and their effects on the execution environment. Furthermore, our plans include the introduction of an intuitive user interface, along with a visualization library that will include post-experimentation advanced plotting and analysis functions. This new module will alleviate the difficulties faced by end users in extracting correlations between different parameters, find the sweet spot for their deployment, and extract useful insights. Finally, we would like to create a recommendation service that suggests (near-)optimal configurations for DNN-enabled applications without requiring every possible permutation to be tested. This service will be especially beneficial for beginners, providing a strong starting point to reduce evaluation time and help them choose the best-fit DNN model for their requirements. Conceptualization, D.T., M.S. and P.M.; methodology, D.T., P.M. and M.S. ; software, P.M. and M.S. ; validation, P.M.; writing--review and editing, M.S., D.T. and P.M.; project administration, D.T.