# Evaluating DL Model Scaling Trade-Offs During Inference via an Empirical Benchmark Analysis
## 3 DNN Model Scaling Benchmark Framework  
To perform the benchmarking of a diverse set of DNN use cases, we created an automated process where a user needs to specify a dataset, the selected DNN use case, and the user's preferences about the model scaling parameters. Figure 2 depicts a high-level overview of our pipeline. The framework provides a lightweight Python library through which users can submit their preferences by either filling in the library's programming interfaces or by passing their parameters written in a configuration file (in YAML format). An example of the configuration file is depicted in Listing 1. In this example, the user selects _MLP_ as the DNN use case (_use_case_). Then, the user must define the parameter space (_parameter_space_) of the respective use case, which includes the available parameters, along with the selected evaluation values. In this example, the user sets three different values for network depth (_network_depth: [4; 5; 7]_), two values for network width and batch size parameters (_network_width: [32; 64]_; _batch_size: [32; 128]_), and three values for epochs (_epochs: [1; 3; 5]_). The system generates all possible permutations of these parameters, resulting in a search space of 36 unique runs for this use case. Next, the monitoring metrics that for the system to expose are selected. The available metrics include _computational_overhead_, _accuracy_, and _latency_. We then examine a set of execution-related parameters, including the boolean parameter _training_, which specifies whether the system retrains the model; _dataset_, which indicates the dataset location; and _times_, which defines the number of experimental executions. It is worth noting that how the dataset is loaded depends on the specific use-case implementation, so the _dataset_ parameter could refer to either a file or a folder where the dataset resides (e.g., a folder of images for object detection and classification use cases). After that, the framework generates a set of model scaling parameters based on user preferences and combines them with a specified ML use case. If users wish to create their own use cases, they should extend the DNN Model Scaling Benchmark programming interface and add them to the _use-case repository_. By default, our system includes a _use-case repository_ with three predefined DL use cases--_natural language understanding_, _regression  
Figure 2: Pipeline of performance evaluation trade-offs. analysis_, and _object detection and classification_--each with its respective scaling parameters (see Section 4). With the parameters and chosen use case defined (via the configuration file or programmatically), the system generates a model whose structure reflects the specified _model parameter set_, resulting in a _compiled model_. ```
use_case:MLP parameter_space: network_depth:[4,5,7] network_width:[32,64] batch_size:[32,128] epochs:[1,3,5] metrics: - computational_overhead - accuracy - latency training: True dataset:\"...\" times:1
```  
The next phase of our pipeline is _execution_. This phase has two steps, namely _training_ and _inference_. Specifically, the pipeline splits the dataset into training and testing subsets. This split is configurable with default values of 80% for training and 20% for the testing set. After that, the pipeline performs the training, updating the weights of the _compiled model_. We should note that the training step is optional, since the users may introduce their own pre-trained models. Next, the trained model is utilized to perform inference on the testing subset, extracting the performance metrics, namely _inference latency_, _accuracy/error_, and _computational overhead_. Since the test subset is already annotated, we can easily extract accuracy for classification tasks or error for regression tasks. For the inference latency, the framework keeps the timestamp before the inference process starts and the timestamp when it is over, subtracts them, and divides by the number of data points.