Title: MLMODELSCOPE: A DISTRIBUTED PLATFORM FOR MODEL EVALUATION AND BENCHMARKING AT SCALE Anonymous

Abstract: Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.

1. INTRODUCTION:

The emergence of Machine Learning (ML) and Deep Learning (DL) within a wide array of application domains has ushered in a great deal of innovation in the form of new models and hardware/software (HW/SW) stacks (frameworks, libraries, compilers, and hardware accelerators) to support these models. Being able to evaluate and compare these innovations in a timely manner is critical for their adoption. These innovations are introduced at such a rapid pace (Dean et al., 2018; arXiv ML Papers Statistics) that researchers are hard-pressed to study and compare them. As a result, there is an urging need by both research and industry for a scalable model/HW/SW evaluation platform.

Evaluation platforms must maintain repeatability (the ability to reproduce a claim) and fairness (the ability to keep all variables constant and allow one to quantify and isolate the benefits of the target of interest). For ML/DL, repeatable and fair evaluation is challenging, since there is a tight coupling between model execution and the underlying HW/SW components. Model evaluation is a complex process where the model, dataset, evaluation method, and HW/SW stack must work in unison to maintain the accuracy and performance claims (e.g. latency, throughput, memory usage). To maintain repeatability, authors are encouraged to publish their code, containers, and write documentation which details the usage along with HW/SW requirements (Mitchell et al., 2019; Reproducibility Checklist; Dodge et al., 2019; Lipton & Steinhardt, 2019; Pineau et al., 2018) . Often, the documentation miss details which make the results not reproducible. To perform a fair evaluation, evaluators have to manually normalize the underlying stack and delineate the codes to characterize performance or accuracy. This is a daunting endeavor. As a consequence, repeatable and fair evaluation is a "pain-point" within the community (Gundersen et al., 2018; Plesser, 2018; Ghanta et al., 2018; Hutson, 2018; Li & Talwalkar, 2019; Tatman et al., 2018 ; Reproducibility in Machine Learning; ICLR Reproducibility Challenge). Thus, an evaluation platform design must have a standard way to specify, provision, and introspect evaluations to guarantee repeatability and fairness.

In this paper, we propose MLModelScope: a distributed design which consists of a specification and a runtime that enables repeatable, fair, and scalable evaluation and benchmarking. The proposed specification is a text-based and encapsulates the model evaluation by defining its pre-processing, inference, post-processing pipeline steps and required SW stack. The runtime system uses the evaluation specification along with user-defined HW constraints as input to provision the evaluation, perform benchmarking, and generate reports. More specifically, MLModelScope guarantees repeatable and fair evaluation by (1) defining a novel scheme to specify model evaluation which separates the entanglement of data/code/SW/HW; (2) defining common techniques to provision workflows with specified HW/SW stacks; and (3) providing a consistent benchmarking and reporting methodology. Through careful design, MLModelScope solves the design objectives while being framework/hardware agnostic, extensible, and customizable. In summary, this paper makes the following contributions: 1 we comprehensively discuss the complexity of model evaluation and describe prerequisites for a model evaluation platform. 2 We propose a model evaluation specification and an open-source, framework/hardware agnostic, extensible, and customizable distributed runtime design which consumes the specification to execute model evaluation and benchmarking at scale. 3 We implemented the design with support for Caffe, Caffe2, CNTK, MXNet, PyTorch, TensorFlow, TensorRT, and TFLite, running on ARM, Power, and x86 with CPU, GPU, and FPGA. 4 For ease of use, we equip MLModelScope with command line, library, and ready-made web interfaces which allows "push-button" model evaluation Model evaluation is complex. Researchers that publish and share DL models can attest to that but are sometimes unaware of the full scope of this complexity. To perform repeatable and fair evaluation, we need to be cognizant of the HW/SW stack and how it affects the accuracy and performance of a model. Figure 1 shows our classification of the HW/SW stack levels. Model level ( L1 ) evaluates a model by performing input pre-processing, model inference, and post-processing. The pre-processing stage transforms the user input into a form that the model expects. The model inference stage calls the framework's inference API on the processed input and produces an output. The post-processing stage transforms the model output to a form that can be viewed by a user or used to compute metrics.

Framework level ( L2 ) performs model inference by executing the layers in the model graph using a framework such as TensorFlow, MXNet, or PyTorch. Layer level ( L3 ) executes a sequence of ML library calls for layers such as convolution, normalization, or softmax. ML Library level ( L4 ) invokes a chain of system library calls for functions in ML libraries such as cuDNN (Chetlur et al., 2014) , MKL-DNN (MKL-DNN) or OpenBLAS (Xianyi et al., 2014) . And, last but not the least, at the hardware level ( L5 ), there are CPU/GPU instructions, disk, and network I/O events, and other low-level system operations through the entire model evaluation. All the HW/SW abstractions must work in unison to maintain the reported accuracy and performance claims. When things go awry, each level within the abstraction hierarchy can be suspect.

Currently, model authors distribute models by publishing documentation and ad hoc scripts to public repositories such as GitHub. Due to the lack of specification, authors may under-specify or omit key aspects of model evaluation. This inhibits, or makes it difficult, for others to repeat their evaluations or validate their claims. Thus all aspects of the model evaluation must be captured by a evaluation platform to guarantee repeatability. To highlight this, consider the model evaluation pipeline at L1 . While the model inference stage is relatively straight forward, the pre-and post-processing stages are surprisingly subtle and can easily introduce discrepancies in the results. Some of the discrepancies might be "silent errors" -where the evaluation is correct for the majority of the inputs but is incorrect for a small number of cases. In general, accuracy errors due to under-specifying preand post-processing are difficult to identify and even more difficult to debug. In Section 4.1, we show the effects of under-specifying different operations in pre-processing on image classification models.

The current practice of publishing models also causes a few challenges which must be addressed by a fair and scalable evaluation platform. First, any two ad hoc scripts do not adhere to a consistent evaluation API. The lack of a consistent API makes it difficult to evaluate models in parallel and, in turn, slows down the ability to quickly compare models across different HW/SW stacks. Second, ad hoc scripts tend to not clearly demarcate the stages of the model evaluation pipeline. This makes it hard to introspect and debug the evaluation. Furthermore, since an apple-to-apple comparison between models requires a fixed HW/SW stack, it is difficult to perform honest comparison between two shared models without modifying some ad hoc scripts. MLModelScope addresses these challenges through the careful design of a model evaluation specification and a distributed runtime as described in Section 3. Software Stack-MLModelScope uses docker containers to maintain the software stacks. MLMod-elScope provides ready-made containers for all popular frameworks, but users can use any container hosted on Docker Hub. Multiple containers can be specified within the manifest. The container used for evaluation is dependent on the executing hardware and whether the system has a GPU or not.

3. MLMODELSCOPE DESIGN:

Model Source-Model source contains links to the model graph (the graph_path field) and weights (the weights_path field). For frameworks which have one file to represent the graph and its weights, the weights field is omitted from the manifest. The model can be stored in the cloud, downloaded on demand, and is cached to the local file system.

Versioning-Models, frameworks, and datasets are all versioned within MLModelScope using a semantic versioning (Preston-Werner, 2019) scheme. The versioning of frameworks and datasets supports constraints, such as ^1.x (Listing 1, Line 8). This tells MLModelScope that the model works on any TensorFlow v1 framework. Listing 2: MLModelScope's evaluation manifest for Inception-v3.

Pre-/Post-Processing Functions-To perform input pre-processing and output post-processing, MLModelScope allows arbitrary Python functions to be placed within the manifest file. The pre-and postprocessing functions have the signature def fun(env, data) where env contains metadata of the evaluation request and data is a PyObject representation of the user request for preprocessing and the model's output for post-processing. Internally MLModelScope executes the Python code within a Python subinterpreter (Python Subinterpreter) in the launched container. To reduce data copy overhead parameters are passed by reference to the processing functions. The pre-and post-processing functions are flexible; i.e. users may import external Python modules or invoke external scripts. By allowing arbi-trary pre-and post-processing function executions, MLModelScope works with existing processing codes and is capable of supporting arbitrary modalities.

Built-in Pre-/Post-Processing Functions-As vision models are widely used and their pre-and post-processing operations are less diverse, MLModelScope allows for common pre-processing image operations (e.g. image decoding, resizing, and normalization) and post-processing operations (e.g. topK, IOU, mAP, etc.) to be used within the manifest without writing code. Internally, MLModelScope invokes built-in pre-and post-processing code to perform these operations. Listing 2 can be placed within the inputs block (Lines 19-22 in Listing 1) as the pre-processing steps for Inception-v3. The steps are executed in the order that is specified, since, as we show in Section 4, the order of operations can have a significant impact on achieved model accuracy. Users are not required to use this feature, but using this feature allows users to easily compare pre-or post-processing steps.

We use this mechanism during our evaluation in Section 4. Listing 3: MLModelScope's predictor RPC API consists of 3 functions which are specified using Protobuf.

3.2. THE MLMODELSCOPE RUNTIME:

The MLModelScope runtime consumes the model manifest to provision evaluations and perform benchmarking. Users evaluate a model by specifying its name, version, and framework along with the target hardware requirements. The MLModelScope runtime uses these user-provided constraints to query previous evaluations or schedule new ones. The runtime is distributed and is built as a set of extensible and customizable modular components (see Figure 2 ). Due to space limitations, we only highlight the key components of the runtime (See Appendix for a description of all components):

Framework Predictors-At the core of the software stack are the frameworks. To enable uniform evaluation and maximize code reuse, MLModelScope wraps each framework's C++ inference API to provide a uniform interface (called predictor API). The predictor API (shown in Listing 3) is minimal and performs model loading, unloading, and inference. So long as a program implements MLModelScope's predictor API, it can be plugged into the system. This means that MLModelScope's design allows for exotic hardware or framework support. For example, some hardware, such as FPGAs and ASICs, do not have a framework per se. These hardware are exposed to MLModelScope through a program which implements the predictor API. The ModelLoad API for FPGAs, for example, downloads a bitfile and load it onto the device.

The predictor API is linked against common code to perform container launching, manifest file handling, downloading of required assets, pre-and post-processing function execution, collecting of performance profiles, and publishing of results -we call this bundle an agent. These agents can be run on separate machines, can be run in parallel, and are managed by the MLModelScope orchestration layer. Agents can be run on remote systems behind firewalls to allow for model evaluation on remote hardware -this allows hardware providers to give model evaluators access to perform model evaluations without full unrestricted access to the hardware. MLModelScope does not require modifications to a framework and thus pre-compiled binary versions of frameworks (e.g. distributed through Python's pip) or customized versions of a framework work within MLModelScope.

Manifest and Predictor Registry-MLModelScope uses a distributed key-value registry (Escriva et al., 2012) to store the model manifests and running agent information. MLModelScope's orchestration layer leverages the registry to facilitate the discovery of models and routing of user requests across the distributed agents using the HW/SW constraints provided by the user. The registry is dynamic -i.e. both model manifests and agents can be added and removed at runtime.

Profilers and Tracers-To enable performance debugging, MLModelScope collects system, framework, and model level profiling information. This data is published into a tracing server (OpenTracing; Sigelman et al., 2010) where it gets aggregated and summarized. Through the trace, users get a holistic view of the performance of model evaluation and can identify bottlenecks. To minimize overhead, the profilers are only active when a user enables them as part of the evaluation request.

Web UI and Command Line Interface-Users interact with MLModelScope through its web UI or command-line interface by specifying model and hardware constraints. These constraints are used to query the database for previous evaluations or to schedule new ones. Users can integrate MLModelScope within their existing tools or pipelines by using its REST or RPC APIs. To illustrate the execution flow of a model evaluation, consider a user wanting to run Inception-v3 trained using ILSVRC 2012 on an Intel system with TensorFlow satisfying the "≥1.10.x and ≤1.13.0" version constraint. The user specifies these constraints using MLMod-elScope's UI and invokes the model evaluation. MLMod-elScope then finds one or more systems which satisfy the user's constraints, sets up the environment, and launches the model evaluation within a container. The results are then published to a database for subsequent analysis.

3.3. MLMODELSCOPE EVALUATION FLOW:

Figure 2 shows the evaluation flow of a user's request. 1 On system startup, each agent publishes the hardware it is running on to the registry. This information is made visible to the MLModelScope orchestration layer. 2 A user then uses MLModelScope's UI to request an evaluation by specifying the model, framework, and hardware constraints. 3 An API request is then performed to the remote API handler, which then 4 queries the registry to find an agent which satisfies the user's constraints. 5 The request is then forwarded to one (or all) of the agents capable of running the evaluation. The agents then provision the hardware and software environment and run the model. 6 The agents then collect and publish the results to a centralized evaluation database. 7 Finally, an evaluation summary is presented to the user.

4. EVALUATION:

We implemented the MLModelScope design as presented in Section 3 with support for popular frameworks (Caffe, Caffe2, CNTK, MXNet, PyTorch, TensorFlow, TensorRT, and TFLite) and tested it on common hardware (X86, PowerPC, and ARM CPUs as well as GPU and FPGA accelerators). We populated it with over 300 models covering a wide array of inference tasks such as image classification, object detection, segmentation, image enhancement, recommendation, etc. We considered three aspects of MLModelScope for our evaluation: the effects of under-specified pre-processing on model accuracy, model performance across systems, and the ability to introspect model evaluation to identify performance bottlenecks. To demonstrate MLModelScope's functionality, we installed it on multiple Amazon instances and performed the evaluation in parallel using highly cited image classification models.

Unless otherwise noted, all results use TensorFlow 1.13.0-rc2 compiled from source; CUDNN 7.4; GCC 6.4.0; Intel Core i7-7820X CPU with Ubuntu 18.04.1; NVIDIA TITAN V GPU with CUDA Driver 410.72; and CUDA Runtime 10.0.1 (Amazon p3.2xlarge Instance).

4.1. MODEL PRE-PROCESSING:

We use MLModelScope to compare models with different operations in the pre-processing stage. Specifically, we look at the impact of image decoding, cropping, resizing, normalization, and data type conversion on model accuracy. For all the experiments, the post-processing is a common operation which sorts the model's output to get the top K predictions. To perform the experiments, we create variants of the original Inception-v3 (Silberman & Guadarrama, 2018; Szegedy et al., 2016) pre-processing specification (shown in Listing 2). We maintain everything else as constant with the exception to the operation of interest and evaluate the manifests through MLModelScope's web UI.

Color Layout-Models are trained with decoded images that are in either RGB or BGR layout. For legacy reasons, OpenCV decodes images in BGR layout by default and, subsequently, both Caffe and Caffe2 use the BGR layout (caffebgr). Other frameworks (such as TensorFlow and PyTorch) use RGB layout. Intuitively, incorrect color layout only misclassifies images which are defined by their colors. Images which are not defined by their colors, however, would be correctly classified. Figure 3 shows the Top 5 classifications for the same image when changing the color layout.

Data Layout-Images are represented by: N (batch size), C (channels), H (height), and W (width). Models are trained using data in either NCHW or NHWC form. Figure 4 shows Inception-v3's (trained using NHWC layout) Top1 result using different layouts for the same image. Decoding and Color Conversion-It is common to use JPEG as the image data serialization format (with ImageNet being stored as JPEG images). Model developers use library functions such as opencv.imread, PIL.Image.open, or tf.io.decode_jpeg to decode JPEG images. These functions may use different decoding algorithms and color conversion methods. For example, we find the YCrCb to RGB color conversion to not be consistent across the PIL and OpenCV libraries.

Figure 5 shows the results † of decoding an image using Python's PIL and compares it to decoding † To increase the contrast of the image differences on paper, we dilate the image (with radius 2) and adjust its pixel values to cover the range between 0 and 1. with OpenCV. As shown, edge pixels are not decoded consistently, even though these are critical pixels for inference tasks such as object detection.

Cropping and Resizing-Accuracy is sometimes reported for cropped datasets, and this is often overlooked when evaluating a model. For Inception-v3, for example, input images are 87.5% centercropped and then resized to 299 × 299. Figure 6 shows the effect of cropping on accuracy: (a) is the original image; (b) is the result of center cropping the image with 87.5% and then resizing; (c) is the result of just resizing; (d) and (f) shows the top-5 results for images (b) and (c). Intuitively, the effects of cropping are more pronounced for images where the marginal regions are meaningful (e.g. framed paintings).

Type Conversion and Normalization-After decoding, the image data is in bytes and is converted to FP32 (assuming an FP32 model). Mathematically, float to byte conversion is f loat2byte(x) = 255x, and byte to float conversion is byte2f loat(x) = x 255.0 and are equivalent. Because of programming semantics, however, the executed behavior of byte to float conversion is byte2f loat(x) =

x 255.0 . The input may also be normalized to have zero mean and unit variance ( pixel-mean stddev ). We find that the order of operations for type conversion and normalization matters. Figure 7 shows the image processing results using different order of operations for meanByte = stddevByte = 127.5 and meanF loat = stddevF loat = 0.5 where: (a) is the original image, (b) is the result of reading the image in bytes then normalizing it with both mean and standard deviation in bytes, byte2f loat( imgByte-meanByte stddevByte ), (c) is the result of reading an image in floats then normalizing it with both mean and standard deviation in FP32, byte2f loat(imgByte)-meanF loat Table 1 shows the effects of pre-processing operations ‡ on the top 1 and top 5 accuracy for the entire ImageNet (Deng et al., 2009) validation dataset. The experiments are run in parallel on 4 Amazon p3.2xlarge systems. We can see that the accuracy errors due to incorrect pre-processing might be hard to debug, since they might only affect a small subset of the inputs. For example, failure to centercrop the input results in 1.45% -7.5% top 1 accuracy difference, and 0.36% -4.22% top 5 accuracy difference.

4.2. HARDWARE EVALUATION:

We use MLModelScope to compare different hardware's achieved latency and throughput while fixing the model and software stack. We launch the same MLModelScope TensorFlow agent on 9 different Amazon EC2 systems recommended for DL (shown in Table 2 ). These systems are equipped with either GPUs or CPUs. We use MLModelScope's UI to run the evaluations in parallel across all 9 systems, and measure the achieved latency and throughput of the Inception-v3 model as the batch size is varied (shown in Figure 9 ). Using the measured latency and throughput, along with system pricing information, we calculate the cost/performance as "dollars per million images". We find that GPU instances in general are more cost-efficient than CPU instances for batched inference. We also observe that the g3s.xlarge is as cost efficient as the p3.2xlarge, because of the high price of the p3.2xlarge instance. We use MLModelScope to compare and introspect frameworks' performance by fixing the model and hard-ware. For illustration purpose, we show AlexNet, since it has less than 10 layers and fits within the paper. We use MLModelScope's TensorRT, MXNet, Caffe2, and Caffe agents and run them on the Amazon p3.2xlarge system. Figure 8 shows AlexNet's latency across frameworks. To understand the performance of each framework, we use MLModelScope's profiler to delve deep and capture each evaluation's layer and library performance information. Through the data, we observe that ML layers across frameworks are implemented differently and dispatched to different library functions. Take the first conv2 and the following relu layers for example. In TensorRT, these two layers are fused and are mapped into two trt_volta_scudnn_128x128_relu_small_nn_v1 kernels (Oyama et al., 2018) which take 1.95ms. In Caffe2, however, the layers are not fused and take 2.63ms. The sub-model profile information helps identify bottlenecks within the model inference. We can see that MLModelScope helps understand the performance across the HW/SW stack which is key to evaluating HW/SW stack choices.

4.3. FRAMEWORK EVALUATION AND INTROSPECTION:



5. RELATED WORK:

To encourage repeatability in ML/DL research, guidelines (Mitchell et al., 2019; Dodge et al., 2019; Li & Talwalkar, 2019; Lipton & Steinhardt, 2019; Pineau et al., 2018; Reproducibility Checklist) have been developed which authors are advised to follow. These guidelines are checklists of what is required to ease reproducibility and encourage model authors to publish code and write down the HW/SW constraints needed to repeat the evaluation. More often than not, model authors use notebooks (Ragan-Kelley et al., 2014), package managers (Fursin et al., 2018a; b) or containers (Kurtzer et al., 2017; Godlove, 2019) to publish their code or specify the SW requirements. These SW requirements are accompanied with a description of the usage, required HW stack, and are published to public repositories (e.g. on GithHub). Through its design, MLModelScope guarantees repeatable evaluations by codifying the model evaluation through the manifest and user-provided HW constraints.

Both industry and academia have developed consortiums to build benchmark suites that evaluate widely used models (MLPerf; MLMark; AI-Matrix; Gao et al., 2019; Li et al., 2019) . These benchmark suites provide separate (non-uniform) scripts that run each model. Each researcher then uses these scripts to perform evaluations on their target HW/SW stack. MLModelScope's model pipeline specification overlaps with the demarcation used by other benchmark suites (e.g. MLPerf seperates model evaluation into pre-processing, model inference, and post-processing). MLModelScope, as an evaluation platform, can incorporate models from benchmark suites so that they can benefit from the distributed evaluation, profiling, and experiment management capabilities. MLModelScope currently has models from benchmark suites such as MLPerf Inference and Alibaba's AI-Matrix built-in.

To allow for distributed evaluation, existing platforms utilize general distributed fabrics (Burns et al., 2016; Boehm et al., 2016; Hindman et al., 2011) to perform model serving (Kubeflow; Chard et al., 2019; Novella et al., 2018; Pachyderm; Zhou et al., 2019) or experimentation (Tsay et al., 2018; FAI-PEP) . MLModelScope differs in that it decouples the specification and provisioning of the model evaluation pipeline from the HW/SW stack to enable repeatable and fair evaluations. Moreover, it allows users to introspect the execution at sub-model granularity. To the best of the author's knowledge, no previous design addresses the confluence of repeatability, fairness, and introspection within scalable model evaluation at the same time.

6. CONCLUSION:

Everyday, an increasingly complex and diverse DL models as well as hardware/software (HW/SW) solutions are proposed -be it algorithms, frameworks, libraries, compilers, or hardware. Both industry and research are hard-pressed to quickly, thoroughly, consistently, and fairly evaluate these new innovations. This paper proposes MLModelScope, which is a specification along with a distributed runtime design that is scalable, extensible, and easy-to-use. 

A SUPPLEMENTARY MATERIAL:

MLModelScope is a big system, and we had to selectively choose topics due to the space limitation. This supplementary materials section is used to provide details about MLModelScope that we were unable to cover in the paper's main body. Specifically, we discuss how MLModelScope: (a) incorporates the latest research and production models through model manifests by showing object detection and instance segmentation models. (b) Attracts users by providing a web interface and command line for scalable model evaluation. (c) Is built from a set of modular components which allows it to be easily customized and extended.

A 

A.2 WEBSITE WORKFLOW:

Although MLModelScope provides both command line and library interfaces, we find the website provides an intuitive flow for specifying and running experiments. Figure 10 shows the flow, and a video demonstrating it can be found at https://drive.google.com/open?id=1LOXZ7hs_ cy-i0-DVU-5FfHwdCd-1c53z. In figure 10 , users first arrive at 1 MLModelScope's landing page.

The landing page contains a description of the project along with links to how to setup and install MLModelScope. Users can try MLModelScope by 2 clicking the demo button, which then displays 3 the inference tasks exposed through the website. If a user 4 selects object detection, then 5 models that are available for object detection are displayed. A user can then 7 selects one or more models and 8 selects one or more systems to run the evaluation on. The input can be specified as a URL, data from disk, or dataset 8 and once complete the user can perform the evaluation 9 . This 10 will run the evaluation on the remote system and 11 display the evaluation results along with summary of the execution flow. In this section we describe each component in Figure 11 in detail. The runtime is designed to be extensible and customizable.

A.3.1 USER INTERFACE AND API:

MLModelScope can be used as an application or as a library. Users interact with MLModelScope application through its website, command line, or its API interface.

The website and command line interface allow users to evaluate and profile models without familiarity with the underlying frameworks or profiling tools. Users who wish to integrate MLModelScope within their existing tools or pipelines can use the REST or RPC APIs. They can also compile MLModelScope as a standalone shared library and use it within their C/C++, Python, or Java projects.

A.3.2 ML ARTIFACTS:

As discussed in the main body of the paper, replication of model accuracy and performance results is dependent on: the usage of specific HW/SW stack; the training dataset; and the pre/post-processing steps on the inputs and outputs. MLModelScope specifies these requirements via a model manifest file described in Section 3. The manifest tells MLModelScope the HW/SW stack to instantiate and how to evaluate the model.

Asset Versioning -Models, frameworks, and datasets are versioned using a semantic versioning scheme. The MLModelScope middleware layer uses this information for asset management and discovery. To request a model, for example, users specify model, framework, hardware, or dataset constraints. MLModelScope solves the constraint and returns the predictors (systems where the model is deployed) that satisfy the constraint. The model evaluation can then be run on one of (or, at the user request, all) the predictors.

Docker Containers -To maintain the SW stack, evaluations are performed within docker containers.

To facilitate user introspection of the SW stack, MLModelScope integrates with existing docker tools that allows querying images's SW environment and metadata.

Pre/Post-Processing Operations -MLModelScope provides the ability to perform common operations such as resizing, normalization, and scaling without writing code. It also allows users to specify code snippets for pre/post-processing within the manifest file which are run within a Python subsession. MLModelScope is able to support a wide variety of models for different input modalities.

Evaluation History -MLModelScope uses the manifest information as keys to store the evaluation results in a database. Users can view historical evaluations through the website or command line using query constraints similar to the ones mentioned above. MLModelScope summarizes and generates plots to aid in comparing the performance across experiment.

A.3.3 FRAMEWORK AND MODEL PREDICTORS:

A predictor is a thin abstraction layer that exposes a framework through a common API. A predictor is responsible for evaluating models (using the manifest file) and capturing the results along with the framework's profile information.A predictor publishes its HW/SW stack information to MLMod-elScope's registry at startup, can have multiple instantiations across the system, and is managed by MLModelScope's middleware.

A.3.4 MIDDLEWARE:

The middleware layer is composed of services and utilities for orchestrating, provisioning, aggregating, and monitoring the execution of predictors -acting as a conduit between the user-facing APIs and the internals of the system.

:

‡ We omit from Table1the data layout pitfall results, since, as expected, it results in very low accuracy.

Figures and Tables:

Figure 3: Top 5 predictions using Inception-v3 with RGB or BGR color layout.

Figure 4: Top 1 predictions using Inception-v3 with NCHW or NHWC data layout.Figure5: Image decoding difference between PIL and OpenCV.

Figure 4: Top 1 predictions using Inception-v3 with NCHW or NHWC data layout.Figure5: Image decoding difference between PIL and OpenCV.

Figure 6: Differences in the prediction results due to cropping using TensorFlow Inception-v3.

Figure 7: Differences due to order of operations using TensorFlow Inception-v3.

stddevF loat , and (d) is the difference between (b) and (c). The inference results of (b) and (c) are shown in (e) and (f).

Figure 9: Inference latency of Inception-v3 for (a) CPU and (b) GPU systems. The xaxis is the batch size, and the y-axis is latency in seconds for (a) and throughput in images/second for (b).

MLModelScope's model specification for Mask_RCNN_ResNet50_v2_Atrous_COCO MXNet model.

Figure 10: The MLModelScope website provides an intuitive interface to conduct experiments.

Figure 11: MLModelScope's runtime components.

We also add introspection capability in MLModelScope to analyze accuracy at different stages and capture latency and memory information at different levels of the HW/SW stack. 6 We showcase MLModelScope by running experiments which compare different model pipelines, hardware, and frameworks.

Table content: <table><tr><td>2 MODEL EVALUATION CHALLENGES</td><td>L1</td><td>Model</td><td colspan="2">Conv Pre-process</td><td/><td colspan="2">BN Model Inference</td><td>Relu</td><td>Post-process</td></tr><tr><td/><td>L2</td><td>Framework</td><td>Data</td><td/><td/><td/><td>Concat</td><td>Softmax</td></tr><tr><td/><td/><td/><td/><td>Conv</td><td/><td>Bias</td><td>Relu</td></tr><tr><td/><td>L3</td><td>Layer</td><td>Malloc</td><td>CUDNN</td><td/><td colspan="2">Transpose</td><td>CUDNN</td><td>Free</td></tr><tr><td/><td>L4</td><td>Library</td><td>Malloc</td><td colspan="2">Kernel</td><td colspan="2">Kernel</td><td>Free</td></tr><tr><td/><td>L5</td><td>Hardware</td><td colspan="2">L$ Access</td><td colspan="2">DRAM Access</td><td>SP Flops</td></tr><tr><td/><td colspan="7">Figure 1: Execution of a model evaluation at different levels of hardware and software abstractions on GPUs.</td></tr></table>

The effects of the pre-processing on the Top 1 and Top 5 accuracy for heavily cited models.

Table content: <table><tr><td/><td colspan="2">Expected</td><td colspan="2">Color Layout</td><td colspan="2">Cropping</td><td colspan="2">Type Conversion</td></tr><tr><td>Model Name</td><td>Top1</td><td>Top5</td><td>Top1</td><td>Top5</td><td>Top1</td><td>Top5</td><td>Top1</td><td>Top5</td></tr><tr><td colspan="9">Inception-V3 (Szegedy et al., 2016) MobileNet1.0 (Howard et al., 2017) ResNet50-V1 (He et al., 2016a) ResNet50-V2 (He et al., 2016b) VGG16 (Simonyan &amp; Zisserman, 2014) 73.23% 91.31% 59.17% 82.77% 71.71% 90.61% 73.24% 91.33% 78.41% 94.07% 67.44% 88.44% 78.27% 94.24% 78.41% 94.08% 73.27% 91.30% 59.22% 82.95% 71.26% 90.17% 73.27% 91.29% 77.38% 93.58% 63.21% 85.65% 75.87% 92.82% 77.40% 93.56% 77.15% 93.43% 63.35% 85.95% 75.71% 92.72% 77.13% 93.42% VGG19 (Simonyan &amp; Zisserman, 2014) 74.15% 91.77% 60.41% 83.57% 72.66% 90.99% 74.14% 91.75%</td></tr></table>

Amazon systems used for evaluation.

Table content: <table><tr><td>Instance</td><td>Hardware</td><td colspan="2">$/hr Cost/Perf.</td></tr><tr><td colspan="2">p2.xlarge g3s.xlarge Tesla M60 (Maxwell), 8GB Tesla K80 (Kepler), 12GB p3.2xlarge Tesla V100-SXM2 (Volta), 16GB c5.large 2 Intel Platinum 8124M, 4GB c5.xlarge 4 Intel Platinum 8124M, 8GB c5.2xlarge 8 Intel Platinum 8124M, 16GB c4.large 2 Intel Xeon E5-2666 v3, 3.75GB c4.xlarge 4 Intel Xeon E5-2666 v3, 7.5GB c4.xlarge 8 Intel Xeon E5-2666 v3, 15GB</td><td>0.9 0.75 3.06 0.085 0.17 0.34 0.1 0.199 0.398</td><td>2.39 1.45 1.49 2.76 2.88 3.19 5.09 5.95 5.94</td></tr></table>

Through MLModelScope, users can perform fair and repeatable comparisons across models, software stacks, and hardware. MLModelScope's careful design of the specification, runtime, and parallel evaluation flow reduces time-to-test for model evaluators. With MLModelScope, we evaluate a set of representative image classification models and present insights into how different pre-processing operations, hardware, and framework selection affect model accuracy and performance.

Table content: <table/>

.1 MLMODELSCOPE MODEL MANIFESTS Listing 4 shows the manifest of SSD_MobileNet_v1_COCO, an objection detection model, for TensorFlow. This model embeds the pre-processing operations in the model graph, and thus requires no normalization, cropping, or resizing. The major difference from a image classification model manifest is the task type (being object_detection) and the outputs. There are three output tensors for this model (boxes, probabilities, and classes). These output tensors are processed by MLModelScope to produce a single object detection feature array, which can then be visualized or used to calculate the metrics (e.g. mean average precision).Listing 5 shows the manifest of Mask_RCNN_ResNet50_v2_Atrous_COCO, an instance segmentation model, for MXNet. The major difference from the object detection model in Listing 4 is the task type (being instance_segmentation) and the outputs. Listing 5 shows four outputs for this model (boxes, probabilities, classes, and masks). These output tensors are processed by MLModelScope to produce a single instance segmentation feature array. Note that unlike TensorFlow, MXNet uses layer indices in place of layer names to get the tensor objects.

Table content: <table><tr><td>16</td><td>-...</td></tr><tr><td>17</td><td>license: Apache License, Version 2.0 # license of the model</td></tr><tr><td>18</td><td>inputs: # model inputs</td></tr><tr><td>19</td><td>-type: image # first input modality</td></tr><tr><td>20</td><td>element_type: uint8</td></tr><tr><td>21</td><td>layout: HWC</td></tr><tr><td>22</td><td>color_layout: RGB</td></tr><tr><td>23</td><td>outputs:</td></tr><tr><td>24</td><td>-type: box</td></tr><tr><td>25</td><td>element_type: float32</td></tr><tr><td>26</td><td>layer_name: 0</td></tr><tr><td>27</td><td>-type: probability</td></tr><tr><td>28</td><td>element_type: float32</td></tr><tr><td>29</td><td>layer_name: 1</td></tr><tr><td>30</td><td>-type: class</td></tr><tr><td>31</td><td>element_type: float32</td></tr><tr><td>32</td><td>layer_name: 2</td></tr><tr><td>33</td><td>features_url: https://.../labels.txt</td></tr><tr><td>34</td><td>-type: mask</td></tr><tr><td>35</td><td>element_type: float32</td></tr><tr><td>36</td><td>source: # specifies model graph and weights sources</td></tr><tr><td>37</td><td>base_url: http://.../mxnet/Mask_RCNN_ResNet50_v2_Atrous_COCO/</td></tr><tr><td>38</td><td>graph_path: model-symbol.json</td></tr><tr><td>39</td><td>weights_path: model-0000.params</td></tr><tr><td>40</td><td>attributes: # extra model attributes</td></tr><tr><td>41</td><td>training_dataset: COCO # dataset used to for training</td></tr><tr><td>42</td><td>manifest_author: ...</td></tr><tr><td>1</td><td>name: SSD_MobileNet_v1_COCO # name of your model</td></tr><tr><td>2</td><td>version: 1.0 # version information in semantic version format</td></tr><tr><td>3</td><td>task: object_detection # task type</td></tr><tr><td>4</td><td>framework:</td></tr><tr><td>5</td><td>name: TensorFlow # framework name</td></tr><tr><td>6</td><td>version: 1.12.x # framework version contraint</td></tr><tr><td>7</td><td>container: # containers used to perform model evaluation</td></tr><tr><td>8</td><td>amd64:</td></tr><tr><td>9</td><td>gpu: mlcn/tensorflow:amd64-cpu</td></tr><tr><td>10</td><td>cpu: mlcn/tensorflow:amd64-gpu</td></tr><tr><td>11</td><td>ppc64le:</td></tr><tr><td>12</td><td>cpu: mlcn/tensorflow:ppc64le-gpu</td></tr><tr><td>13</td><td>gpu: mlcn/tensorflow:ppc64le-gpu</td></tr><tr><td>14</td><td>description: ...</td></tr><tr><td>15</td><td>references: # references to papers / websites / etc.. describing the model</td></tr><tr><td>16</td><td>-...</td></tr><tr><td>17</td><td>license: Apache License, Version 2.0 # license of the model</td></tr><tr><td>18</td><td>inputs: # model inputs</td></tr><tr><td>19</td><td>-type: image # first input modality</td></tr><tr><td>20</td><td>element_type: uint8</td></tr><tr><td>21</td><td>layer_name: image_tensor</td></tr><tr><td>22</td><td>layout: HWC</td></tr><tr><td>23</td><td>color_layout: RGB</td></tr><tr><td>24</td><td>outputs:</td></tr><tr><td>25</td><td>-type: box</td></tr><tr><td>26</td><td>element_type: float32</td></tr><tr><td>27</td><td>layer_name: detection_boxes</td></tr><tr><td>28</td><td>-type: probability</td></tr><tr><td>29</td><td>element_type: float32</td></tr><tr><td>30</td><td>layer_name: detection_scores</td></tr><tr><td>31</td><td>-type: class</td></tr><tr><td>32</td><td>element_type: float32</td></tr><tr><td>33</td><td>layer_name: detection_classes</td></tr><tr><td>34</td><td>features_url: https://.../labels.txt</td></tr><tr><td>35</td><td>source:</td></tr><tr><td>36</td><td>graph_path: https://.../ssd_mobilenet_v1_coco_2018_01_28.pb</td></tr><tr><td>37</td><td>attributes: # extra model attributes</td></tr><tr><td>38</td><td>training_dataset: COCO # dataset used to for training</td></tr><tr><td>39</td><td>manifest_author: ...</td></tr><tr><td/><td>Listing 4: MLModelScope's model specification for SSD_MobileNet_v1_COCO TensorFlow model.</td></tr><tr><td>1</td><td>name: Mask_RCNN_ResNet50_v2_Atrous_COCO # name of your model</td></tr><tr><td>2</td><td>version: 1.0 # version information in semantic version format</td></tr><tr><td>3</td><td>task: instance_segmentation</td></tr><tr><td>4</td><td>framework:</td></tr><tr><td>5</td><td>name: MXNet # framework for the model</td></tr><tr><td>6</td><td>version: 1.4.x # framework version contraint</td></tr><tr><td>7</td><td>container: # containers used to perform model evaluation</td></tr><tr><td>8</td><td>amd64:</td></tr><tr><td>9</td><td>gpu: mlcn/mxnet:amd64-cpu</td></tr><tr><td>10</td><td>cpu: mlcn/mxnet:amd64-gpu</td></tr><tr><td>11</td><td>ppc64le:</td></tr><tr><td>12</td><td>cpu: mlcn/mxnet:ppc64le-gpu</td></tr><tr><td>13</td><td>gpu: mlcn/mxnet:ppc64le-gpu</td></tr><tr><td>14</td><td>description: ...</td></tr><tr><td>15</td><td>references: # references to papers / websites / etc.. describing the model</td></tr></table>

Peer review (iclr2020):
The paper presents a unified approach to specify, evaluate and benchmark different ML methods. With the main goal of enforcing repeatability and faireness when testing different methods, authors propose an open source runtime on which 1) specify the model, 2) describe the workflow and 3) evaluate the benchmark of several ML algorithms and frameworks. The core of the work is the definition of the so-called *model evaluation manifest* which consists of a formatted collection of descriptive information where both hardware/software and framework versions are specified, along with the set of tasks to be carried on as well as the data sources to test the methods against. Once the manifest has been created, the desired hw/sw configuration is deployed on Amazon and the specified models are benchmarked. This benchmarking offers several insights on the evaluation of a given ML model, by stressing out the importance of aspects that can severely bias the final outcome of the model (e.g., pre-processing tasks, different hardware configurations or normalization of the data). To describe the workflow, authors use an image classifier on a given hardware as a running example, and play with different preprocessing methods to measure their impact on the final accuracy of the model. Some details are not well specified/clear in the work: 1) Data exploitation. There is the possibility of testing different methods on own datasets. Given that the deployment is run on Amazon instances, what are the requirements (e.g., data must be on S3 and so on). 2) The manifest can be injected with python scripts that, running in a container, perform the desired operations (preprocessing). It is stated that *parameters are passed by reference*. So if you pass a *mutable* object (*env*, I guess) you need to bind it to the outher scope. How this is accomplished? (globals?) Instead, if you pass an *immutable* object (*data*, I guess), you cannot rebind the outer reference nor mutate the object. So, what*s the meaning of *passing by reference*? 3) Privacy and anonymity. When performing debugging, system, framework and model level profiling information are collected on a tracing server. Is this server part of the platform?