# On the Cost of Model-Serving Frameworks: An Experimental Evaluation

Pasquale de Rosa, Yérom-David Bromberg, Pascal Felber, Djob Barbe

Thystere Mvondo Djob, Valerio Schiavoni

► To cite this version:

Pasquale de Rosa, Yérom-David Bromberg, Pascal Felber, Djob Barbe Thystere Mvondo Djob, Valerio Schiavoni. On the Cost of Model-Serving Frameworks: An Experimental Evaluation. IC2E 2024 - IEEE International Conference on Cloud Engineering, IEEE Computer Society, Sep 2024, Paphos, Cyprus. pp.221-232, 10.1109/IC2E61754.2024.00032. hal-04894045

HAL Id: hal-04894045

https://hal.science/hal-04894045v1

Submitted on 21 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# On the Cost of Model-Serving Frameworks:
An Experimental Evaluation

Pasquale De Rosa

University of Neuchâtel

Neuchâtel, Switzerland

pasquale.derosa@unine.ch

Yérom-David Bromberg

University of Rennes, CNRS, INRIA, IRISA

Rennes, France

david.bromberg@irisa.fr

Pascal Felber

University of Neuchâtel

Neuchâtel, Switzerland

pascal.felber@unine.ch

Djob Mvondo

University of Rennes, CNRS, INRIA, IRISA

Rennes, France

barbe-thystere.mvondo-djob@inria.fr

Valerio Schiavoni

University of Neuchâtel

Neuchâtel, Switzerland

valerio.schiavoni@unine.ch

**Abstract**—In machine learning (ML), the inference phase is the process of applying pre-trained models to new, unseen data with the objective of making predictions. During the inference phase, end-users interact with ML services to gain insights, recommendations, or actions based on the input data. For this reason, serving strategies are nowadays crucial for deploying and managing models in production environments effectively. These strategies ensure that models are available, scalable, reliable, and performant for real-world applications, such as time series forecasting, image classification, natural language processing, and so on. In this paper, we evaluate the performances of five widely-used model serving frameworks (TensorFlow Serving, TorchServe, MLServer, MLflow, and BentoML) under four different scenarios (malware detection, cryptocurrency prices forecasting, image classification, and sentiment analysis). We demonstrate that TensorFlow Serving is able to outperform all the other frameworks in serving deep learning (DL) models. Moreover, we show that DL-specific frameworks (TensorFlow Serving and TorchServe) display significantly lower latencies than the three general-purpose ML frameworks (BentoML, MLFlow, and MLServer).

**Index Terms**—model serving, inference, benchmarking

# I. INTRODUCTION

Machine learning systems (ML) focus on algorithms and models that enable computers to learn from data and make predictions or decisions. Common applications of ML include price predictions of financial assets [36], [64], recommendation systems [79], [83], object detection [39], [42], speech recognition [38], [43], and more. A typical ML processing pipeline consists broadly of two phases. The training phase, i.e., the process of building a model from data, is generally computationally expensive. It is executed mostly by complex model structures on large datasets [45]. State-of-the-art neural network architectures (such as recurrent neural networks - RNNs [44], [50], and convolutional neural networks - CNNs [57]), despite their excellent recognition accuracy, come at very high computational costs induced by their training phase [65]. More specifically, CNNs, typically used in image classification tasks [75], can require several days to train on multiple GPUs even with downsampled input images [56].

The inference (or prediction) phase is the process of building predictions from the trained model. It is generally considered less expensive since it does not require multiple iterations over batched samples of the dataset [45]. This phase consists of a single forward pass<fnref n="1" /> over the test set. If considering models fitted with a very high number of parameters, e.g., neural networks, it is generally smaller in size than the data used for training [55].

While the training phase is time-consuming, the inference should normally execute in *almost* real-time, as is typically embedded in user-facing applications that provide low-latency predictions at scale [45]. To design these user-facing prediction services, the Machine Learning as a Service (MLaaS) [71] paradigm allows to automatically train, test, and deploy ML models, handling the entire model lifecycle.

Nowadays, it is common for major cloud providers to enhance their products with functionalities enabling users to deploy ML projects [69], [81]. Cloud providers are generally a good solution to develop MLaaS platforms because they control a vast volume of data and provide a massive computational power, both instrumental to develop complex ML services for the large audience [69], [81].

However, while relying on MLaaS platforms is considered the most practical solution, not involving structural costs traditionally linked to in-premises infrastructures (i.e., ownership and maintenance of machines, in-house IT teams), it also comes at a steep price. For example, a standard instance with 2 vCPUs and 8 GiB RAM on AWS Sagemaker [1] costs about $0.10/h, on Azure Machine Learning [3] costs $0.096/h and on Google AI Platform [9] up to $0.10/h.<fnref n="2" /> Considering how production-ready deployments require much higher computing resources, such costs will increase considerably. Bigger cloud instances with 64 vCPUs and 256 GiB RAM, capable of efficiently executing optimized ML tasks, cost $968'680/year (AWS), $807'321/year (Azure), and $856'728/year (Google).

<footnote n="1">¹The process of propagating input data through the ML model to produce an output.</footnote>

<footnote n="2">²Prices for the East US region as of May 2024.</footnote>
In addition, outsourcing a sensitive model to untrustworthy cloud providers comes with serious drawbacks, both from a technical and jurisdictional perspective [67], [76].

To sidestep the mentioned drawbacks, one can consider an on-premise deployment of an *MLaaS* service over its private cloud infrastructure, typically built on top of an open-source and easy-to-access ML serving framework. Several frameworks allow providers to design scalable and efficient ML inference services and expose them via public-facing APIs.

However, the systematic evaluation of different model-serving strategies remains an open challenge despite end-users increasing demand for real-time ML predictions.

This paper aims to fill this existing gap by benchmarking five state-of-the-art open-source model serving frameworks, namely TensorFlow-Serving [26], [66], TorchServe [31], MLServer [20], MLflow [19], [82] and BentoML [4].

The reason why we select these frameworks is to have a representative of both DL-specific solutions (*i.e.*, TensorFlow Serving and TorchServe), the *de-facto* baseline solutions in serving models built, respectively, with TensorFlow [35] and PyTorch [68], and general-purpose frameworks for ML deployment (*i.e.*, MLServer, MLflow and BentoML).

In this study we focus on the design and evaluation of different serving runtimes, packaging model code and artifacts into containers to build APIs optimized for model inference. We leave the discussion of end-to-end serving platforms based on Kubernetes [16], like KServe [14] and Kubeflow [15], to future extensions of this work.

To thoroughly evaluate such frameworks, we consider four different and diverse application scenarios: (i) malware detection, (ii) cryptocurrency prices forecasting, (iii) image classification, and (iv) sentiment analysis. These scenarios represent four realistic use cases from real-world business workloads [2], [12], [28], [33], where ML inference services are implemented. However, the flexible nature of our study makes its results generalizable to multiple business applications, in addition to these ones.

Moreover, these scenarios allow our study to offer a wide range of real-world ML cases: regressions for the price forecasting use-case, binary classification (for malware detection and sentiment analysis), and multiclass classification (on images), ensuring the robustness and generalizability of our findings.

*Our results show that TensorFlow Serving consistently outperforms all the other frameworks in serving deep learning (DL) models. The two DL-specific frameworks (TensorFlow Serving and TorchServe) display lower latencies than the remaining general-purpose frameworks.*

The main contributions of this paper are:

* The systematic analysis of five state-of-the-art open source model serving frameworks on four real-world application scenarios;

* The identification of the best platform among the considered ones in terms of model serving latencies;

* The assessment of the best performing approach to DL model serving between DL-specific frameworks (TensorFlow Serving and TorchServe) and general-purpose ML frameworks (MLflow, MLServer, BentoML).

We follow an open science approach, releasing code and pre-trained models to the research community at https://zenodo.org/records/11499446.

**Roadmap.** §II describes the details of the model serving frameworks that we consider in this paper. §III provides an overview of the considered application scenarios. §IV characterizes our datasets. §V discusses the experimental evaluation, while in §V-A we present our results. We cover related work in §VI, before concluding with our future work in §VII.

## II. MODEL SERVING FRAMEWORKS

In this section, we provide insights on the selected model serving frameworks, e.g., TensorFlow Serving, TorchServe, MLServer, MLflow, and BentoML.

Table I provides a comprehensive overview of these serving frameworks, highlighting their CPU/GPU support capability and the pool of ML/DL libraries they handle. We further detail such frameworks in the rest of this section.

**TensorFlow Serving** [26] is a flexible, high-performance model-serving system for ML and deep learning (DL) models, typically designed for production environments. TensorFlow Serving provides a native integration with TensorFlow DL models.

Pretrained models are serialized in a specific format (*i.e.*, *SavedModel*), that contains a complete TensorFlow program, comprising of the trained parameters, the variables and the computation. Models can be saved and loaded in the *SavedModel* format using either a low-level *tf_saved_model* API or an higher-level *tf_keras.Model* API.

Once the model has been stored in the *SavedModel* and correctly loaded in the registry<fnref n="3" />, TensorFlow Serving will accept inference requests, typically over REST or gRPC interfaces.

**TorchServe** [31] is a tool that streamlines the deployment of PyTorch models in production environments, offering scalability, flexibility, and performance optimization. A PyTorch model (`.pth/pt`) differs from a TensorFlow *SavedModel*; while the first offers modular saving options (TorchScript and state dictionary), the latter provides an all-in-one encapsulation. Moreover, PyTorch models can be deployed using TorchScript [30], whereas *SavedModel* supports a wider range of deployment environments (e.g., TensorFlow Serving, TensorFlow.js [27], TensorFlow Lite [25]). Being primarily focused on serving PyTorch models, it does not provide out-of-the-box support for other ML frameworks. Its popularity among practitioners and ML enthusiasts is high, and popular model-hosting sites, such as HuggingFace<fnref n="4" /> and their Hugging

<footnote n="3">³A centralized repository that stores versioned models, metadata, and lineage information.</footnote>

<footnote n="4">⁴https://huggingface.co/</footnote>
<table><caption>TABLE I: Overview of the model serving frameworks (supported ML/DL libraries and model formats are shortlisted from the official docs).</caption><thead><tr><th>Framework</th><th>ML</th><th>DL</th><th>CPU</th><th>GPU</th><th>Supported (ML)</th><th>Supported (DL)</th><th>Model Formats</th></tr></thead><tbody><tr><td>TF-Serving</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td><td>-</td><td>TensorFlow, HuggingFace, Keras</td><td>SavedModel, TFLite, TF.js</td></tr><tr><td>TorchServe</td><td>✗</td><td>✓</td><td>✓</td><td>✓</td><td>-</td><td>PyTorch, HuggingFace</td><td>MAR, PyTorch, ONNX</td></tr><tr><td>MLServer</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Scikit-Learn, XGBoost, LightGBM, CatBoost</td><td>TensorFlow, PyTorch, HuggingFace, Keras</td><td>SaveModel, PyTorch, Keras, ONNX, Pickle, Joblib</td></tr><tr><td>MLflow</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Scikit-Learn, XGBoost, LightGBM, CatBoost</td><td>TensorFlow, PyTorch, HuggingFace, Keras</td><td>SaveModel, PyTorch, Keras, ONNX, Pickle, Joblib</td></tr><tr><td>BentoML</td><td>✓</td><td>✓</td><td>✓</td><td>✓</td><td>Scikit-Learn, XGBoost, LightGBM, CatBoost</td><td>TensorFlow, PyTorch, HuggingFace, Keras</td><td>SaveModel, PyTorch, Keras, ONNX, Pickle, Joblib</td></tr></tbody></table>

FaceTransformers, which are typically developed in PyTorch, can also be served using TorchServe [10].

To serve a model with TorchServe, it must be archived as a MAR file using the Torch Model Archiver [18]. TorchServe can automatically scale the total number of backend workers serving the model to the number equal to the available vCPUs or GPUs on the underlying computing node.

TorchServe also allows it to serve multiple models simultaneously within a single server instance. This can be instrumental in cases where a user requests more than one prediction at a time from different models.

Finally, TorchServe allows users to define custom handlers for preprocessing and postprocessing model inputs and outputs, allowing them to define flexible services that can adapt to specific requirements and applications.

MLServer [20] is a scalable and flexible framework optimized for serving ML models in production environments. MLServer provides out-of-the-box support for several ML frameworks (e.g., Scikit-learn [24], XGBoost [34], LightGBM [17], CatBoost [6]), as well as for DL frameworks like TensorFlow and PyTorch. This flexibility enables users to deploy a large set of ML models regardless of the framework they were originally trained in.

MLServer is highly scalable and capable of handling many inference requests concurrently. Moreover, it allows users to serve models in custom environments specifically designed to match the runtime requirements of different models.

MLServer implements a RESTful API endpoint for inference to serve predictions at scale using the models stored in the registry.

MLflow [19] is a comprehensive platform for managing the entire ML pipeline, from model training to experimentation, tracking/logging and deployment. Being general-purpose, it allows to serve ML and DL models trained in various frameworks, like TensorFlow, PyTorch, Scikit-learn etc.

MLflow provides tracking and logging features for the deployed models, enabling real-time model performance analysis. Moreover, it allows you to define custom pre- and post-processing logic to be integrated into the model inference service.

MLflow relies on Flask [8] to serve the inference endpoint, but integration with MLServer or other alternatives is possible.

BentoML [4] is a unified model-serving library for building scalable AI applications with Python. It includes tools for serving optimization, model packaging, and production deployment. Like MLServer and MLflow, BentoML is a general-purpose framework that supports a wide range of ML libraries, including TensorFlow, PyTorch, Scikit-learn, XGBoost, LightGBM, and CatBoost.

BentoML allows the packaging of pre-trained models and all the necessary requirements into a comprehensive, ready-to-deploy format named Bento. A Bento includes all the libraries, configurations, and pre-and post-processing logic needed for the model to produce inferences at scale correctly.

### A. Models: formats and conversions

As described before, TorchServe is not natively intended to handle deploying models not designed in PyTorch. Therefore, since our models were developed using TensorFlow (in SavedModel), we needed to convert them into PyTorch format (`.pth/pt`).

However, this task was challenging since the two formats proved to be very different both in structure and purpose (for a more comprehensive overview, refer to section II).

One common conversion method combines TensorFlow's support for exporting models to the ONNX format [22] and PyTorch's ability to import ONNX models. The ONNX (Open Neural Network Exchange) model format is an open-source interchange<fnref n="5" /> format for representing DL models. It aims to provide a standard way to allow models to be trained in one framework and deployed in another. It enables interoperability between different DL frameworks, just like TensorFlow and PyTorch: DL models trained in one framework can be exported to ONNX and imported into another framework for inference (or further training).

Undoubtedly, the cross-framework conversion of a model might generate a potential threat to validity. However, given the current state of the art, this approach is the only feasible way to allow a comparison between the two most commonly used DL frameworks, TensorFlow and PyTorch, and hereby enrich the quality of our study.

In our work, we leveraged two Python libraries written on top of ONNX: TF2ONNX [29], which allows us to convert TensorFlow models to ONNX, and ONNX2Torch [23], an ONNX to PyTorch converter.

## III. MOTIVATING SCENARIOS

We describe the four scenarios we consider best to conduct our experimental evaluation of the various model-serving frameworks. These scenarios capture different application contexts: (i) malware detection, (ii) cryptocurrency price forecasting, (iii) image classification, and (iv) sentiment analysis.

**Malware detection** on Android APKs (Android application packages) is the analysis of the file features to determine whether it contains malicious code or behavior. Nowadays,

<footnote n="5">⁵A file format designed to facilitate the exchange of models between different platforms, providing a standardized way to represent such models.</footnote>
it is common to adopt ML and DL algorithms to perform this classification task [37], [49], [58], [73].

Android malware detection approaches can be mostly divided into static and dynamic analyses. The static analysis extracts the features from an APK without running them into a device or Android emulator. Instead, the dynamic analysis extracts the features by running them on an Android emulator or device [49], [58].

In our evaluation, we rely on a static analysis approach. Our choice is motivated by the fact that, on malware detection tasks using ML techniques, dynamic analysis proved to be less effective, thus leading to a minor recognition accuracy if compared with a static approach [52].

The first step in the malware detection process is to extract relevant features from the APK file. These features can include permissions requested by the application, API calls, code structure, metadata, resource files, etc. To extract features from the original APKs, we leveraged Drebin [37], a lightweight static analyzer for Android applications. Then, we generated a vectorial representation of each application with values in {0, 1}, where 1 means that a specific feature is present in the APK and 0 otherwise.

Therefore, we used a simple Deep Neural Network (DNN) with two hidden layers of 200 units and a final sigmoid layer to recognize a malicious/benign APK from the previously generated vectorial representation. We trained our DNN model on a subsample of the CICMalDroid 2020 dataset [60], containing the 10% of the observations and following the same distribution between malicious and benign examples of the original data.

**Cryptocurrency price forecasting** is the analysis of co-movements and correlation patterns in the cryptocurrency market to forecast the price trends for a given coin. History has shown the extreme volatility of such trading prices. Reasons for such instability include the lack of adequate regulation, the inherent speculative nature of cryptocurrencies, and the lack of a governmental or institutional guarantor. However, recent studies [40], [46], [53] demonstrated the possibility of forecasting these particular time series using ML and DL models.

In our scenario, we leveraged the price series trends of a set of 8 highly-correlated coins (i.e., ADA, BNB, DOGE, ETH, LTC, SOL, XLM, XRP) to forecast the hourly price trends of Bitcoin (BTC) over a user-specified number of time steps.

We used a DNN with two hidden layers of 200 units and a final ReLu layer to forecast the BTC price using the correlated feature coins' series as feature variables to achieve this target. To estimate the feature coins' series over the desired time frame, we adopted a Moving Average (MA) over a sliding window of the last 24 hours' observations from the validation set. We trained our DNN model on a dataset gathered from Binance [5] (a popular cryptocurrency exchange) containing the closing prices of the analyzed cryptocurrencies between within 19 days, i.e., 01-01-2024 00:00:00 and 19-04-2024 23:00:00.

**Image classification** is a widely analyzed task in ML. It consists of categorizing images into predefined classes or

<table><caption>TABLE II: Overview of the motivating scenarios with datasets and models used.</caption><thead><tr><th></th><th></th></tr></thead><tbody><tr><td><strong>UC1: Malware Analysis</strong></td><td></td></tr><tr><td>Dataset</td><td>CICMalDroid 2020 [60] (10%)</td></tr><tr><td>Model</td><td>DNN + Drebin [37]</td></tr><tr><td><strong>UC2: Cryptocurrency</strong></td><td></td></tr><tr><td>Dataset</td><td>Binance [5]</td></tr><tr><td>Model</td><td>DNN + MA</td></tr><tr><td><strong>UC3: Image classification</strong></td><td></td></tr><tr><td>Dataset</td><td>ImageNet 2012 [72]</td></tr><tr><td>Model</td><td>MobileNetV2 [74]</td></tr><tr><td><strong>UC4: Sentiment analysis</strong></td><td></td></tr><tr><td>Dataset</td><td>IMDb 50K [59]</td></tr><tr><td>Model</td><td>Embedding + CNN [57]</td></tr></tbody></table>

labels. Crucial applications of image recognition include, for example, medical diagnosis [61] and object detection [70].

In our scenario, as a classification model, we used MobileNetV2 [74], i.e., a Convolutional Neural Network (CNN) based on an inverted residual structure with connections between the bottleneck layers. The model has been trained on the ImageNet 2012 dataset [72], first presented in the ImageNet Large Scale Visual Recognition Challenge 2012 (ILSVRC 2012) [11]. The dataset spans 1000 object classes and contains 1 281 167 training images, 50 000 validation images, and 100 000 test images.

The pre-trained model, containing the loaded ImageNet weights, was retrieved from Keras Applications [13]. It can recognize image input data of shape (224, 224, 3).

**Sentiment Analysis**, or opinion mining, is a natural language processing (NLP) application that involves the analysis of opinions, attitudes, emotions, and sentiments expressed in textual data (and, more recently, in other types of media). Common real-world applications of sentiment analysis involve the study of opinions on social media [48] and the analysis of customer feedback from reviews [77].

In this scenario, we used a Convolutional Neural Network (CNN) with an initial Embedding layer, two convolutional layers, and a final sigmoid layer to recognize a negative/positive sentiment from a tokenized<fnref n="6" /> input sentence.

To tokenize the input data, we cleaned it by removing punctuations and stopwords and applying a lemmatization<fnref n="7" />.

We trained our model on the IMDb 50K dataset [59], containing 50 000 movie reviews for natural language processing and text analytics applications.

Table II provides a comprehensive overview of these scenarios, highlighting the combination of dataset and models used in each.

## IV. DATASETS

To execute our experimental evaluation, we rely on four real-world datasets. Each is used in the considered scenarios (§III). The datasets are: CICMalDroid 2020 [60], Binance [5], ImageNet [72] and IMdb 50K [59].

**CICMalDroid 2020.** This dataset is an Android malware dataset with more than 17 341 samples, collected from several

<footnote n="6">⁶Tokenizers convert raw string inputs into integer inputs suitable for the Embedding layer.</footnote>

<footnote n="7">⁷The process of reducing words to their base, known as the lemma.</footnote>
sources, including VirusTotal [32] or Contagio [7], from
December 2017 to December 2018.

The APKs dataset spans five categories: adware, Banking malware, SMS malware, Riskware, and Benign. More specifically, Adware refers to advertising material that hides inside apps infected by malware. Banking malware is designed to access the user's online banking accounts. SMS malware exploits the SMS service to conduct attacks. Riskware is a legitimate program that can cause damage if used maliciously, and Benign is a nonmalicious, safe application.

We gathered ≈ 10% of the total samples (1727), following
the original distribution of the original data between Adware
(8.8%), Banking malware (14.5%), SMS malware (27.9%),
Riskware (25.2%) and Benign samples (23.6%).

In terms of sample size, the largest benign APK in our dataset is 47MB, while the smallest is 48KB; for malicious samples, instead, the biggest APK size is 41MB while the smallest is 8KB.

**Binance.** In this dataset, we gathered time series from Binance, a popular crypto coin exchange. The time series provides high resolution data points (i.e., one sampling per hour). It includes the closing price<fnref n="8" /> records for 9 top-traded cryptocurrencies (e.g., ADA, BNB, BTC, DOGE, ETH, LTC, SOL, XLM, XRP). The entire dataset consists of 23 544 observations from 01-01-2024 00:00:00 to 19-04-2024 23:00:00. Each coin's time series includes 2616 steps.

**ImageNet.** We use ImageNet [72], a well-known dataset and image database organized according to the WordNet [63] hierarchy. WordNet is a large lexical database for the English language, combining a dictionary and thesaurus. In WordNet, nouns, verbs, adjectives, and adverbs are grouped into cognitive synonym rings (synsets). In the ImageNet database, each image is annotated using WordNet synsets, making it one of the largest resources available for training DL models in computer vision tasks. While the original dataset contains over 14 million images, its most popular version, e.g., ILSVRC (ImageNet Large Scale Visual Recognition Challenge) 2012 [11], spans 1000 object classes and contains 1 281 167 training images, 50 000 validation images and 100 000 test images.

**IMDb 50K** is a popular dataset collected from IMDb (Internet Movie Database) [59]. This dataset, commonly used for binary sentiment classification (positive/negative), contains 25 000 highly polar movie reviews for training and 25 000 for testing purposes.

V. EVALUATION

In this section, we provide a detailed overview of the
experimental evaluation of the selected model-serving frame-
works (TensorFlow Serving, TorchServe, MLServer, MLflow,
and BentoML) in each of the four considered scenarios. We
aim to evaluate different strategies of the DL model serving
under realistic circumstances and assess which framework is
most efficient in terms of observed latencies. Specifically,

our experimental evaluation intends to answer the following
questions:

* Q1: Are there significative differences in the observed latencies for the considered model serving platforms?

* Q2:Can significative differences be observed in the latencies for different sizes of the input payload?

* Q3: Which model serving framework displays the most efficient performance on DL pipelines in terms of low latencies?

* Q4: Are DL-specific frameworks (TensorFlow Serving and TorchServe) capable of outperforming the general-purpose frameworks (MLflow, MLServer, and BentoML) on DL deployment tasks?

We support experimental reproducibility and release our
code, datasets, and instructions to reproduce our experiments
at https://zenodo.org/records/11499446.

**Experimental setup.** We use Ubuntu 22.04.4 LTS, 40 CPUs Intel(R) Xeon(R) Gold 5215 clocked at 2.50GHz, Linux kernel 5.15.0-105-generic, and 128 GB RAM.

We leverage Python v3.10.12, TensorFlow with TensorFlow
ModelServer v2.15.0, PyTorch v2.2.2 with TorchServe 0.10.0,
BentoML v1.2.12, MLServer v1.5.0 and MLflow v2.12.1.

We trained our models in TensorFlow. To deploy them on
TorchServe, which natively supports only PyTorch pretrained
formats (`.pth`), we converted the original models to Py-
Torch via ONNX [22]. We leveraged ONNX v1.16.0, with
ONNX2Torch [23] v1.5.14 and TF2ONNX [29] v1.16.1.

**Data pre-processing.** We aim to simulate a realistic interaction between a user and an inference platform for each of the considered scenarios. Therefore, we use real-world testing data, to which we apply a range of preprocessing techniques to transform the raw input into a format that can be "understood" and managed by the inference model. A comprehensive description of the input preprocessing techniques applied in each scenario is summarized in Table III.

1) **Scenario 1:** since the context of this scenario is to perform a malware detection on Android applications, the raw input data is an APK file (e.g., an archive compressed using the classical). Using Drebin, we extract features from a given APK and confront them with the 30 050 unique features gathered from the training set. Then, we generate a new vectorial representation for the APK of size 30 050, where each data point is either a 0 or 1, depending on whether the specific feature is present in the application. Therefore, the preprocessed output is a tensor<fnref n="9" /> of shape (1, 30050).

2) **Scenario 2:** in this scenario, the aim is to forecast a number *n* of future hourly prices for Bitcoin, given the time series of highly-correlated *altcoins*.<fnref n="10" /> As stated in §II, we forecast these series by applying a Moving Average (MA) over the last 24 prices in the validation set for each *altcoin*. The user-provided input is the number *n* of future steps. The resulting prediction from the DNN is generated on a

<footnote n="9">⁹A mathematical object that generalizes scalars, vectors, and matrices to higher dimensions.</footnote>

<footnote n="10">¹⁰Altcoins are all cryptocurrencies other than Bitcoin.</footnote>

<footnote n="8">⁸The last price at which the cryptocoin traded during the hour.</footnote>
preprocessed input tensor of shape (n, 8), where 8 are the
MA prices for the feature altcoins used in the forecasting
procedure (one among ADA, BNB, DOGE, ETH, LTC,
SOL, XLM or XRP).

3) **Scenario 3:** in the image classification scenario, clients provide an input image in JPEG format that can be of any resolution and shape. To make it ready for the MobileNetV2 inference, we reshape it in an image 224x224x3, that corresponds to a tensor of shape (1, 224, 224, 3).<fnref n="11" />

4) **Scenario 4:** finally, in this scenario the user provides an input sentence/opinion that the DL model classifies as either positive or negative. To make the data inference-ready, we apply tokenization and lemmatization and remove stopwords from the original sentence. Then the text is converted to a sequence of integers, where each integer represents a specific word in the vocabulary. Therefore, the resulting preprocessed input is a tensor of shape (n, 1), namely a list containing a sequence of the n tokens extracted from the input data.

To assess the performance of the five serving frameworks in
each scenario, we rely on oha [21], an HTTP load generator
built on Rust. On the client side, we set each run’s duration
to 1 minute, with a constant throughput of 200 requests/s. On
the server side, we set the number of parallel threads to 40,
equal to the number of CPUs available in the server hosting
the inference service.

We evaluated the latency distribution of each response
for three different payload sizes: small, medium and big.
Therefore, we ran a total of twelve experiments, three for each
scenario. In scenario 1, the size of the payload corresponds
to the dimension, in bytes, of the original APK (8.2KB,
1.4MB, and 45MB, respectively). Scenario 2 corresponds to
the number *n* of future steps to predict (namely, 1, 12, and
23). In scenario 3, it is the size, in bytes, of the original input
image (342KB, 1.4MB, and 18MB). Finally, in scenario 4 it
corresponds to the length of the original input sequence (7, 51
and 178).

We further analyze and discuss these results in the remain-
der of this section, highlighting the deviation between runs for
each payload size considered.

A. *Analysis of the results*

**Malware detection.** The average number of processed requests in one run (among the two DL-specific frameworks for the three experiments) was 75150, while for the three general-purpose frameworks was 14202.

A1/A4: In this scenario, the two DL-specific scenarios were able to significantly outperform MLflow, MLServer and BentoML.

The minimum average latency (among the two frameworks
for the three payloads) was 0.0372s, that is 68.37% lower than
the one observed for the three general-purpose frameworks

(0.1176s); on the other hand, the maximum average latency
was 0.0895s, 72.04% lower than the benchmark general-
purpose platforms (0.3201s).

TensorFlow Serving showed minimum inference times of, respectively, 0.0215s (small payload), 0.0260s (medium payload) and 0.0242s (high payload), and with maximum latencies (at the 99th percentile) of 0.0431s, 0.0445s, 0.0432s.

TorchServe showed minimum inference times of, respec-
tively, 0.0136s (small payload), 0.0127s (medium payload)
and 0.0135s (high payload), and with maximum latencies of
0.1434s, 0.1346s, 0.1283s.

A3: While TorchServe displayed the lower minimum latencies
compared to TensorFlow Serving, the latter proved to be more
stable among all the percentiles.

A2: No significative differences were detected when changing the payload sizes.

The only exception was BentoML, that showed an increase
in the maximum latency for the medium-sized APK compared
to the small and big payloads. This shift was only detectable
for the 99<sup>th</sup> percentile, and we believe it was likely due to
outliers. <sup>12</sup> The results are shown in Fig. 1.

**Cryptocurrency price forecasting.** The average number of processed requests in one run (among the two DL-specific frameworks for the three experiments) was 528794, while for the three general-purpose frameworks was 20605.

A1/A4: In this scenario, the two DL-specific scenarios clearly
outperform MLflow, MLServer and BentoML.

The minimum average latency (among the two frameworks
for the three payloads) was 0.0039s, that is 95.47% lower than
the one observed for the three general-purpose frameworks
(0.0860s); on the other hand, the maximum average latency
was 0.0676s, 69.95% lower than the benchmark general-
purpose platforms (0.2250s).

TensorFlow Serving showed minimum inference times of, respectively, 0.0008s (small payload), 0.0015s (medium payload) and 0.0016s (high payload), and with maximum latencies (at the 99th percentile) of 0.0035s, 0.0042s, 0.0042s.

TorchServe showed minimum inference times of, respec-
tively, 0.0093s (small payload), 0.0049s (medium payload)
and 0.0054s (high payload), and with maximum latencies of
0.1137s, 0.1262s, 0.1538s.

A3: In this case, TensorFlow Serving showed both lower
latencies overall and stability among all the percentiles than
TorchServe.

A2: No significative differences where detected when chang-
ing the payload sizes.

An exception is represented by BentoML and TorchServe,
that showed an increase in the maximum latency for the

<footnote n="11">¹¹The first dimension (i.e., 1) is the batch size, that for a single input is unitary.</footnote>

<footnote n="12">¹²According to [47], the variability of response time that leads to high tail latency in a service (especially at the 99th percentile) can arise for many reasons, including: shared resources, daemons, maintenance activities, queueing, power limits, garbage collection and energy management.</footnote>
<table><caption>TABLE III: Overview of original input and preprocessed one for each scenario.</caption><thead><tr><th></th><th>Scenario 1</th><th>Scenario 2</th><th>Scenario 3</th><th>Scenario 4</th></tr></thead><tbody><tr><th>Original Input</th><td>APK file</td><td>Number of steps</td><td>JPEG image</td><td>Text</td></tr><tr><th>Preprocessed Input</th><td>Tensor (1, 30050)</td><td>Tensor (n, 8)</td><td>Tensor (1, 224, 224, 3)</td><td>Tensor (n, 1)</td></tr></tbody></table>

<figure><img src="image_2.png" /><figcaption>Fig. 1: Stacked percentile chart of the average inference time for the serving frameworks in Scenario 1.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Fig. 2: Stacked percentile chart of the average inference time for the serving frameworks in Scenario 2.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3: Stacked percentile chart of the average inference time for the serving frameworks in Scenario 3.</figcaption></figure>

medium-sized input (BentoML) and for the big-sized input (TorchServe) compared to the other payloads, likely due to outliers.<fnref n="13" /> The results are shown in Fig. 2.

**Image classification.** The average number of processed requests in one run (among the two DL-specific frameworks for the three experiments) was 8778, while for the three general-purpose frameworks was 2526.

A1/A4: The two DL-specific scenarios outperform MLflow, MLServer and BentoML.

The minimum average latency (among the two frameworks for the three payloads) was 0.1427s, that is 70.2% lower than the one observed for the three general-purpose frameworks (0.4788s); on the other hand, the maximum average latency was 0.4828s, 80.06% lower than the benchmark general-purpose platforms (2.4213s).

TensorFlow Serving showed minimum inference times of, respectively, 0.0706s (small payload), 0.0756s (medium payload) and 0.0712s (high payload), and with maximum latencies (at the 99th percentile) of 0.4250s, 0.4260s, 0.4444s.

TorchServe showed minimum inference times of, respectively, 0.2065s (small payload), 0.2188s (medium payload) and 0.2140s (high payload), and with maximum latencies of 0.5357s, 0.5384s, 0.5272s.

A3: In this case, TensorFlow Serving showed lower latencies overall than TorchServe. Both frameworks displayed a general instability among all the percentiles.

A2: No significant differences were detected when changing the payload sizes.

An exception is represented by BentoML and MLflow, that showed an increase in the maximum latency for the small-sized input (BentoML) and for the medium-sized input (both BentoML and MLflow) compared to the other payloads. As for

<footnote n="13">¹³See note 12.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 4: Stacked percentile chart of the average inference time for the serving frameworks in Scenario 4.</figcaption></figure>

the previous scenarios, also in this case the shift is detectable only for the 99th percentile, most likely due to the presence of outliers.<fnref n="14" /> The results are shown in Fig. 3.

**Sentiment Analysis.** The average number of processed requests in one run (among the two DL-specific frameworks for the three experiments) was 545141, while for the three general-purpose frameworks was 19162.

**A1/A4:** As in the other scenarios, the DL-specific platforms outperform MLflow, MLServer and BentoML.

The minimum average latency (among the two frameworks for the three payloads) was 0.0042s, that is 95.44% lower than the one observed for the three general-purpose frameworks (0.0921s); on the other hand, the maximum average latency was 0.1231s, 53.19% lower than the benchmark general-purpose platforms (0.2630s).

TensorFlow Serving showed minimum inference times of, respectively, 0.0013s (small payload), 0.0015s (medium payload) and 0.0023s (high payload), and with maximum latencies (at the 99th percentile) of 0.0031s, 0.0038s, 0.0046s.

TorchServe showed minimum inference times of, respectively, 0.0032s (small payload), 0.0087s (medium payload) and 0.0085s (high payload), and with maximum latencies of 0.2558s, 0.3177s, 0.1541s.

**A3:** In this case, TensorFlow Serving showed both lower latencies overall and stability among all the percentiles than TorchServe.

**A2:** No significative differences where detected when changing the payload sizes.

An exception is represented by BentoML and TorchServe, that showed an increase in the maximum latency for the small and medium-sized input compared to the other payloads. The results are shown in Fig. 4.

## B. Lessons learned

In all the considered scenarios, TensorFlow Serving results clearly to be the best choice, outperforming every other framework.

A possible reason for this behavior is that the four models have been developed in TensorFlow. In fact, TensorFlow Serving is specifically designed and optimized for serving TensorFlow models, being able to leverage the framework's

full capabilities and providing native support for SavedModels formats.

The second-best choice results to be TorchServe. Also in this case, it is safe to assume that this is due to the framework being specifically developed to serve PyTorch models. As stated in section V, in fact, in order to evaluate this framework we needed to convert the original TensorFlow model in a PyTorch pretrained format (`.pth`). This resulting model contains the same weights and parameters of the original one, and therefore provides the same predictions where evaluated.

In the comparison between TensorFlow Serving and TorchServe, we can notice in the second a more unstable behavior, that reflects in outlier high peaks in latency as displayed in Fig. 1, Fig. 2 and Fig. 4, despite a general low-latency trend shown in the remaining quartiles.

Overall, the two DL-specific frameworks (TensorFlow Serving and TorchServe) are able to considerably outperform the three general-purpose ML frameworks (BentoML, MLFlow and MLServer) in all the considered scenarios.

Therefore, we can conclude that when serving DL pretrained models, TensorFlow Serving (for TensorFlow models) and TorchServe (for PyTorch models) are the most efficient solutions available open source to date.

## C. Macro-benchmark: turn-around inference time

In this section, we describe a macro-benchmark analysis conducted among the five serving frameworks in order to describe the turn-around inference time in three selected scenarios. Our aim was to simulate a full DL model serving pipeline, starting from a user request containing the raw, unprocessed input data, to the final postprocessed inference in human-readable format. We leverage Flask [8], a Python web framework designed to build lightweight applications, to put in place a local server able to, in order: (i) receive a user HTTP request, (ii) preprocess the input data, (iii) send an inference request to the corresponding ML serving API, and finally (iv) send back the resulting, post-processed predictions in human-readable format (e.g., labels or classes).

In the following paragraphs, we briefly describe the main results observed for each scenario.

**Malware detection.** As we can observe from Fig. 5, the average time to process a full request, for each APK size, was significantly high. More specifically, each framework suffered from a "cold start", due to an elevated time spent in the features

<footnote n="14">¹⁴See note 12.</footnote>
extraction from the input application. No noticeable differ-
ences could be observed between the single frameworks laten-
cies. TensorFlow Serving and TorchServe performed slightly
better than the other considered frameworks, mostly due to a
shorter average inference time (as we discussed in section V).
No strong differences in the turn-around time were detected for
the small and medium APK, while the larger input presented
significantly higher latencies due mostly to the massive pre-
processing involved.

**Cryptocurrency price forecasting.** From Fig. 6, we observe that the average time to process a full request, for each number of forecasting steps, was short. No noticeable differences could be observed between the single frameworks latencies. Also here, TensorFlow Serving and TorchServe performed slightly better than the other considered frameworks. No noticeable differences could be detected for the different input sizes, that did not effect the request turn-around time. This was mostly due to the "lightweight" nature of the input, that did not involve complex operations of pre and post processing.

**Image classification.** From Fig. 7, we observe a similar behavior to the first scenario. Also in this case, in fact, the input images in JPEG format could require a significant pre-processing time, that reflected in a "cold start" for all the frameworks. No noticeable differences could be observed between the single frameworks latencies, with the only exception of MLServer. In this case, we could detect some differences in the turn-around time with respect to the input sizes, with the larger JPEG image that displayed very high latencies due to the massive pre-processing involved for it.

VI. RELATED WORK

The study of model serving strategies in ML development
and MLOps has recently attracted the interest of academia.

In [51], the authors present the first performance evaluation
study of model serving integration tools in stream processing
frameworks. They leveraged Apache Flink as a representative
stream processing system and evaluated DL serving pipelines
for the image classification task. The study results indicated
superior throughput for pipelines that use embedded libraries
to serve pre-trained models. Moreover, in the comparison
between TorchServe and TensorFlow Serving, the latter consis-
tently exhibited better performance, a behavior also confirmed
in our paper.

Other recent works mainly focused on efficient and scalable
serving techniques for large language models (LLMs). For
example, [62] introduced SpecInfer, a system that accelerates
generative LLM serving with tree-based speculative inference
and verification. It leverages small speculative models to pre-
dict the LLM's outputs and organizes the resulting predictions
as a token tree, whose nodes each represent a candidate token
sequence. It was able to reduce the end-to-end latency while
preserving model quality significantly.

In [78], the authors propose Déjà Vu, a system targeted to
reduce Distributed LLM serving inner costs. It implements ef-
ficient prompt-token disaggregation to reduce pipeline bubbles,

micro batch swapping for efficient GPU memory management,
and state replication for fault tolerance.

Finally, [80] introduced the first real-world trace dataset of LLM serving workloads, detailing user, system, and LLM behaviors. Moreover, by analyzing these traces, the authors developed a benchmark suite to enable the performance evaluation of different serving systems.

VII. CONCLUSION

Model serving strategies are crucial in deploying ML mod-
els. They ensure that the models can be accessed by end-
users and utilized efficiently in real-world applications. In
this paper, we provided a comparison between five state-of-
the-art ML model serving frameworks (TensorFlow-Serving,
TorchServe, MLServer, MLflow, and BentoML) on four real-
world applicative scenarios (malware detection, cryptocoin
prices forecasting, image classification, and sentiment analy-
sis). More specifically, we analyzed different strategies of deep
learning (DL) model serving under realistic circumstances
and assessed which framework is most efficient regarding
latencies. Our study demonstrated that TensorFlow Serving
can outperform every other framework in all the considered
scenarios, mostly due to its architecture and structure, which
are defined to efficiently serve TensorFlow models. Moreover,
the two DL-specific frameworks (TensorFlow Serving and
TorchServe) achieved significantly lower latencies than the
three general-purpose ML frameworks (BentoML, MLFlow,
and MLServer), proving to be the two most efficient open-
source solutions for serving DL models.

Finally, we aim to extend our work to real-world applica-
tions where other cost measures, different from latency, might
be impactful. One is surely confidential ML, where models
have to be served and deployed in a manner that ensures
data privacy and model security. This is especially crucial
in domains like healthcare, finance, and defense, where the
confidentiality of data and models is paramount. Moreover,
we plan to further enrich our analysis by considering also
models trained using frameworks other than TensorFlow, like
PyTorch, JAX [41] and Caffe [54].

REFERENCES

[1] Amazon sagemaker website. https://aws.amazon.com/sagemaker/pricing/ (accessed July 11, 2024).
[2] Asticaai website. https://astica.ai/vision/ (accessed July 11, 2024).
[3] Azure machine learning website. https://azure.microsoft.com/en-us/pricing/details/machine-learning/ (accessed July 11, 2024).
[4] Bentoml documentation. https://docs.bentoml.com/en/latest/ (accessed July 11, 2024).
[5] Binance documentation. https://www.binance.com/ (accessed July 11, 2024).
[6] Catboost documentation. https://catboost.ai/ (accessed July 11, 2024).
[7] Contagio security blog. https://contagiodump.blogspot.com/ (accessed July 11, 2024).
[8] Flask documentation. https://flask.palletsprojects.com/en/3.0.x/ (accessed July 11, 2024).
[9] Google cloud ai website. https://cloud.google.com/ai-platform/pricing (accessed July 11, 2024).
[10] Huggingface and torchserve integration. https://github.com/pytorch/serve/blob/master/examples/Huggingface_Transformers/Transformer_handler_generalized.py (accessed July 11, 2024).
<figure><img src="image_6.png" /><figcaption>Fig. 5: Cumulative distribution function of the request turn-around time for the serving frameworks in Scenario 1.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Fig. 6: Cumulative distribution function of the request turn-around time for the serving frameworks in Scenario 2.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 7: Cumulative distribution function of the request turn-around time for the serving frameworks in Scenario 3.</figcaption></figure>

[11] Ilsvrc 2012 website. https://www.image-net.org/challenges/LSVRC/2012/ (accessed July 11, 2024).

[12] Kaspersky cloud ml for android. https://usa.kaspersky.com/about/press-releases/2019.cloud-ml-for-android (accessed July 11, 2024).

[13] Keras applications documentation. https://keras.io/api/applications/ (accessed July 11, 2024).

[14] Kserve documentation. https://kserve.github.io/website/latest/ (accessed July 11, 2024).

[15] Kubeflow documentation. https://www.kubeflow.org/ (accessed July 11, 2024).

[16] Kubernetes documentation. https://kubernetes.io/docs/home/ (accessed July 11, 2024).

[17] Lightgbm documentation. https://lightgbm.readthedocs.io/en/stable/ (accessed July 11, 2024).

[18] Mar file specifics. https://pytorch.org/serve/FAQs.html#what-is-a-mar-file (accessed July 11, 2024).

[19] Mlflow documentation. https://mlflow.org/docs/latest/index.html (accessed July 11, 2024).

[20] Mlserver documentation. https://mlserver.readthedocs.io/en/stable/ (accessed July 11, 2024).

[21] Oha documentation. https://github.com/hatoo/oha (accessed July 11, 2024).

[22] Onnx documentation. https://onnx.ai/onnx/index.html (accessed July 11, 2024).

[23] Onnx2torch documentation. https://github.com/ENOT-AutoDL/onnx2torch (accessed July 11, 2024).

[24] Scikit-learn documentation. https://scikit-learn.org/stable/ (accessed July 11, 2024).

[25] Tensorflow lite documentation. https://www.tensorflow.org/lite (accessed July 11, 2024).

[26] Tensorflow serving documentation. https://www.tensorflow.org/tfx/guide/serving (accessed July 11, 2024).

[27] Tensorflow.js documentation. https://www.tensorflow.org/js (accessed July 11, 2024).

[28] Text2data website. https://text2data.com/Demo (accessed July 11, 2024).

[29] Tf2onnx documentation. https://github.com/onnx/tensorflow-onnx (accessed July 11, 2024).

[30] Torchscript documentation. https://pytorch.org/docs/stable/jit.html (accessed July 11, 2024).

[31] Torchserve documentation. https://pytorch.org/serve/ (accessed July 11, 2024).

[32] Virustotal website. https://www.virustotal.com/ (accessed July 11, 2024).

[33] Walletinvestor website. https://walletinvestor.com/forecast (accessed July 11, 2024).

[34] Xgboost documentation. https://xgboost.readthedocs.io/en/stable/ (accessed July 11, 2024).

[35] Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. Software available from tensorflow.org.

[36] Ryo Akita, Akira Yoshihara, Takashi Matsubara, and Kuniaki Uehara. Deep learning for stock prediction using numerical and textual information.
tion. In 2016 IEEE/ACIS 15th International Conference on Computer and Information Science (ICIS), pages 1–6, 2016.

[37] Dan Arp, Michael Spreitzenbarth, Malte Hubner, Hugo Gascon, and Konrad Rieck. Drebin: Effective and explainable detection of android malware in your pocket. In *Network and Distributed System Security Symposium*, 2014.

[38] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: a framework for self-supervised learning of speech representations. In *Proceedings of the 34th International Conference on Neural Information Processing Systems*, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc.

[39] Alexey Bochkovskyi, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. *CoRR*, abs/2004.10934, 2020.

[40] Ahmed Bouteska, Mohammad Zoynul Abedin, Petr Hajek, and Kunpeng Yuan. Cryptocurrency price forecasting – a comparative analysis of ensemble learning and deep learning methods. *International Review of Financial Analysis*, 92:103055, 2024.

[41] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018.

[42] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, *Computer Vision – ECCV 2020*, pages 213–229, Cham, 2020. Springer International Publishing.

[43] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In *2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pages 4960–4964, 2016.

[44] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder–decoder approaches. In *Proceedings of SST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation*, pages 103–111, Doha, Qatar, oct 2014. Association for Computational Linguistics.

[45] Daniel Crankshaw, Xin Wang, Giulio Zhou, Michael J. Franklin, Joseph E. Gonzalez, and Ion Stoica. Clipper: a low-latency online prediction serving system. In *Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation*, NSDI'17, page 613–627, USA, 2017. USENIX Association.

[46] Pasquale De Rosa, Pascal Felber, and Valerio Schiavoni. Practical forecasting of cryptocurrencies timeseries using correlation patterns. In *Proceedings of the 17th ACM International Conference on Distributed and Event-Based Systems*, DEBS '23, page 80–90, New York, NY, USA, 2023. Association for Computing Machinery.

[47] Jeffrey Dean and Luiz André Barroso. The tail at scale. *Commun. ACM*, 56(2):74–80, feb 2013.

[48] Zulfadzli Drus and Haliyana Khalid. Sentiment analysis in social media and its application: Systematic literature review. *Procedia Computer Science*, 161:707–714, 2019. The Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia.

[49] Omar N. Elayan and Ahmad M. Mustafa. Android malware detection using deep learning. *Procedia Computer Science*, 184:847–852, 2021. The 12th International Conference on Ambient Systems, Networks and Technologies (ANT) / The 4th International Conference on Emerging Data and Industry 4.0 (EDI40) / Affiliated Workshops.

[50] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. *Neural Comput.*, 9(8):1735–1780, nov 1997.

[51] Sonia Horchidan, Emmanouil Kritharakis, Vasiliki Kalavri, and Paris Carbone. Evaluating model serving strategies over streaming data. In *Proceedings of the Sixth Workshop on Data Management for End-To-End Machine Learning*, DEEM '22, New York, NY, USA, 2022. Association for Computing Machinery.

[52] Muhammad Ijaz, Muhammad Hanif Durad, and Maliha Ismail. Static and dynamic malware analysis using machine learning. In *2019 16th International Bhurban Conference on Applied Sciences and Technology (IBCAST)*, pages 687–691, 2019.

[53] Patrick Jaquart, Sven Köpke, and Christof Weinhardt. Machine learning for cryptocurrency market prediction and trading. *The Journal of Finance and Data Science*, 8:331–352, 2022.

[54] Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe:

Convolutional architecture for fast feature embedding. *arXiv preprint arXiv:1408.5093*, 2014.

[55] V. Roshan Joseph. Optimal ratio for data splitting. *Statistical Analysis and Data Mining: The ASA Data Science Journal*, 15(4):531–538, April 2022.

[56] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. *Commun. ACM*, 60(6):84–90, may 2017.

[57] Yann LeCun and Yoshua Bengio. Convolutional networks for images, speech, and time series. In Michael A. Arbib, editor, *Handbook of Brain Theory and Neural Networks*, page 3361. MIT Press, 1995.

[58] Kaijun Liu, Shengwei Xu, Guoai Xu, Miao Zhang, Dawei Sun, and Haifeng Liu. A review of android malware detection approaches based on machine learning. *IEEE Access*, 8:124579–124607, 2020.

[59] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In *Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies*, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.

[60] Samaneh Mahdavifar, Andi Fitriah Abdul Kadir, Rasool Fatemi, Dima Alhadidi, and Ali A. Ghorbani. Dynamic android malware category classification using semi-supervised deep learning. In *2020 IEEE Intl Conf on Dependable, Autonomic and Secure Computing, Intl Conf on Pervasive Intelligence and Computing, Intl Conf on Cloud and Big Data Computing, Intl Conf on Cyber Science and Technology Congress (DASC/PiCom/CBDCom/CyberSciTech)*, pages 515–522, 2020.

[61] Pawan Kumar Mall, Pradeep Kumar Singh, Swapnita Srivastav, Vipul Narayan, Marcin Paprzycki, Tatiana Jaworska, and Maria Ganzha. A comprehensive review of deep neural networks for medical image processing: Recent developments and future opportunities. *Healthcare Analytics*, 4:100216, 2023.

[62] Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In *Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3, ASPLOS '24*, page 932–949, New York, NY, USA, 2024. Association for Computing Machinery.

[63] George A. Miller. WordNet: A lexical database for English. In *Human Language Technology: Proceedings of a Workshop held at Plainsboro, New Jersey, March 8-11, 1994*, 1994.

[64] Latrisha N. Mintarya, Jeta N.M. Halim, Callista Angie, Said Achmad, and Aditya Kurniawan. Machine learning approaches in stock market prediction: A systematic literature review. *Procedia Computer Science*, 216:96–102, 2023. 7th International Conference on Computer Science and Computational Intelligence 2022.

[65] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. In *Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2*, NIPS'14, page 2204–2212, Cambridge, MA, USA, 2014. MIT Press.

[66] Christopher Olston, Noah Fiedel, Kiril Gorovoy, Jeremiah Harmsen, Li Lao, Fangwei Li, Vinu Rajashekhar, Sukriti Ramesh, and Jordan Soyke. Tensorflow-serving: Flexible, high-performance ml serving. *arXiv preprint arXiv:1712.06139*, 2017.

[67] Nicolas Papernot, Patrick McDaniel, Arunesh Sinha, and Michael P. Wellman. Sok: Security and privacy in machine learning. In *2018 IEEE European Symposium on Security and Privacy*, pages 399–414, 2018.

[68] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. *PyTorch: an imperative style, high-performance deep learning library*. Curran Associates Inc., Red Hook, NY, USA, 2019.

[69] Robert Philipp, Andreas Mladenow, Christine Strauss, and Alexander Völz. Machine learning as a service: Challenges in research and applications. In *Proceedings of the 22nd International Conference on Information Integration and Web-Based Applications & Services, iiWAS '20*, page 396–406, New York, NY, USA, 2021. Association for Computing Machinery.
[70] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 779–788, 2016.

[71] Mauro Ribeiro, Katarina Grolinger, and Miriam AM Capretz. Mlaas: Machine learning as a service. In *2015 IEEE 14th international conference on machine learning and applications (ICMLA)*, pages 896–902. IEEE, 2015.

[72] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. *International Journal of Computer Vision (IJCV)*, 115(3):211–252, 2015.

[73] Justin Sahs and Latifur Khan. A machine learning approach to android malware detection. In *2012 European Intelligence and Security Informatics Conference*, pages 141–147, 2012.

[74] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In *2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pages 4510–4520, 2018.

[75] Neha Sharma, Vibhor Jain, and Anju Mishra. An analysis of convolutional neural networks for image classification. *Procedia Computer Science*, 132:377–384, 2018. International Conference on Computational Intelligence and Data Science.

[76] Sagar Sharma and Keke Chen. Confidential machine learning on untrusted platforms: a survey. *Cybersecurity*, 4, 2020.

[77] T. K. Shivaprasad and Jyothi Shetty. Sentiment analysis of product reviews: A review. In *2017 International Conference on Inventive Communication and Computational Technologies (ICICCT)*, pages 298–301, 2017.

[78] Foteini Strati, Sara Mcallister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjàvu: Kv-cache streaming for fast, fault-tolerant generative llm serving, 2024.

[79] Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer. In *Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM '19*, page 1441–1450, New York, NY, USA, 2019. Association for Computing Machinery.

[80] Yuxin Wang, Yuhan Chen, Zeyu Li, Zhenheng Tang, Rui Guo, Xin Wang, Qiang Wang, Amelie Chi Zhou, and Xiaowen Chu. Towards efficient and reliable llm serving: A real-world workload study, 2024.

[81] Mazin Yousif. Intelligence in the cloud – we need a lot of it. *IEEE Cloud Computing*, 4(6):4–6, 2017.

[82] Matei Zaharia, Andrew Chen, Aaron Davidson, Ali Ghodsi, Sue Ann Hong, Andy Konwinski, Siddharth Murching, Tomas Nykodym, Paul Ogilvie, Mani Parkhe, et al. Accelerating the machine learning lifecycle with MLflow. *IEEE Data Eng. Bull.*, 41(4):39–45, 2018.

[83] Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. Deep learning based recommender system: A survey and new perspectives. *ACM Comput. Surv.*, 52(1), feb 2019.