Insights into resource utilization of code small language models serving with runtime engines and execution providers
Creators: name: Durán López, Francisco
Publisher: name: Zenodo
Year: 2025

description: ## Summary

### Contributions- Actionable guidelines for practitioners- Measuring the impact of deep learning serving configurations on energy and performance- An analysis of deep learning serving configurations

### DL serving configurationsA duplet of a runtime engine and an execution provider: <[Runtime engine], [Execution provider]>- Runtime engines  - Default Torch (TORCH)  - ONNX Runtime engine (ONNX)  - OpenVINO runtime (OV)  - Torch JIT (JIT)- Execution providers  - CPU Execution Provider (CPU)  - CUDA Execution Provider (CUDA)

## Repository Structure

The repository is structured as follows:

- app  | API, schemas- dataset  | input dataset generation- experiments  | Notebooks and scripts to process profilers datasets- manuals  | Self-contained manuals related to the serving infrastructure- model_selection  | This folder contains models selection scripts and metadata- scripts  | Environment scripts and bash scripts for automated experiments- testing  | Scripts to send request to server- requirements.txt: The dependencies of our implementation- runall_update.sh: Bash script to start server and run experiments- code_slm_selection.csv: Selection of used code SLM

## Replication package

### 1. Data management

- Needed: HumanEval dataset- Output: New input dataset- files:   - dataset/*

### 2. Modelling

- Needed: Selection criteria- Output: Selected models- files:  - code_slm_selection.csv

### 3. Development

- Needed: Development of serving infrastructure, selected models- Output: Serving infrastructure- files:  - app/

### 4. Operation

- Needed: Deployed serving infrastructure- Output: results (profilers datasets)- files:  - testing/

1. Edit experiment parameters (time,files,...):   1. server settings      - app/models_code_load.py: Model classes        - MAX_LENGTH tokens   2. experiment settings      - testing/utils.py: experiment settings, python script        - input dataset      - repeat.sh: repeat n experiments runs or just execute runall        - runall_update.sh: experiment settings, bash script          - run server          - run experiments for each runtime engine2. Run server and experiments: runall.sh  bash  nohup ./repeat.sh > repeat.out 2>&1 & 

  Or:  bash  nohup ./runall.sh > results/runall.out 2>&1 &  3. Obtain results/*

### 5. Research output

- Needed: Profilers datasets- Output: Research output, data analysis and, support files to answer RQs- files:  - experiments/  - figures  - tables  - statistical results

Files in `experiments/`

- `visualize_{profiler}` - Visualization of raw data obtained from profilers.- `01_get_info_{profiler}` - Preprocessing of raw data obtained from profilers (script).- `02_get_time_marks` - Get time marks of inferences done during experiment (script)- `03_analysis_{execution_provider}` - Process data for analysis (notebook).- `04_aggregation` - Aggregated data (notebook).- `05_aggregated_plots` - Box plots (notebook).- `06_tests` - Obtaining statistical results of used statistical tests (script).- `07_tests_merge` - Merge test results, organized by dependent variable (notebook).- `08_analysis` - Notebook to analyze results (notebook).- `09_result_tables` - Creation of paper table (notebook).

## Models

- [codeparrot-small](https://huggingface.co/codeparrot/codeparrot-small)- [tiny_starcoder](https://huggingface.co/bigcode/tiny_starcoder_py)- [pythia-410m](https://huggingface.co/EleutherAI/pythia-410m)- [bloomz-560m](https://huggingface.co/bigscience/bloom-560m)- [starcoderbase-1b](https://huggingface.co/bigcode/starcoderbase-1b)- [bloomz-1b1](https://huggingface.co/bigscience/bloomz-1b1)- [tinyllama](https://huggingface.co/TinyLlama/TinyLlama-1.1B-Chat-v1.0)- [pythia-1.4b](https://huggingface.co/EleutherAI/pythia-1.4b)- [codegemma-2b](https://huggingface.co/google/codegemma-2b)- [phi2](https://huggingface.co/microsoft/phi-2)- [stablecode-3b](https://huggingface.co/stabilityai/stablecode-instruct-alpha-3b)- [stablecode-3b-completion](https://huggingface.co/stabilityai/stablecode-completion-alpha-3b-4k)

## Energy tracking tools

- [EnergiBridge](https://github.com/tdurieux/EnergiBridge)- [nvidia-smi](https://developer.nvidia.com/nvidia-system-management-interface)- [Wattmeter](https://vitriko.eu/regleta-inteligente-netio-powerbox-4kf)

## Help

### TestingDataset:testing/inputs.txt

Run server:uvicorn app.api_code:app  --host 0.0.0.0 --port 8000  --reload --reload-dir app

Make inferences:python3 testing/main.py -i torch -r 5 | tee -a results/out_torch.logpython3 testing/main.py -i onnx -r 5 | tee -a results/out_onnx.logpython3 testing/main.py -i ov -r 5 | tee -a results/out_ov.logpython3 testing/main.py -i torchscript -r 5 | tee -a results/out_torchscript.log

Results are saved in results/

### Useful Guides1. [API creation](manuals/01_create_api.md). Guide to create an API to deploy ML models.2. [Add pretrained model](manuals/02_add_models.md). Guide to add pretrained ML models (from HuggingFace, hdf5 format, pickle format) to do inferences through an API.3. [Deploy ML models in a cloud provider (General)](manuals/03_deploy_general.md). Guide to deploy ML models using an API in a cloud provider.4. [See more](manuals/)

### Other repos1. https://madewithml.com, API2. https://github.com/se4ai2122-cs-uniba/SE4AI2021Course_FastAPI-demo, API3. https://github.com/MLOps-essi-upc