MLModelScope: A Distributed Platform for ML Model Evaluation and Benchmarking at Scale
Authors: Cheng Li
Abdul Dakkak
Jinjun Xiong
Wen-mei Hwu
Venue:  (2020)
Keywords: Evaluation
Scalable
Repeatable
Fair
System

Abstract: Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major "pain point" for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking.  We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.

Under review as a conference paper at ICLR 2019
MLMODELSCOPE: A DISTRIBUTED PLATFORM FOR MODEL EVALUATION AND BENCHMARKING AT SCALE
Anonymous authors Paper under double-bling review
ABSTRACT
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard- hard-pressed analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning MLIDL evaluation, a major "pain point" for the community. This paper proposes MI ModelSScop,, an open-source, framework/hardwarr agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demon- strate ModoeeSScppe s capabilities we perform parallel evaluation and show how subtle changes model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.
INTRODUCTION
The emergence of Machine Learning (ML) and Deep Learning (DL) within a wide array of application domains has ushered in a great deal of innovation in the form of new models and hardware/software (HW/SW) stacks (frameworks, libraries, compilers, and hardware accelerators) to support these models. Being able evaluate and compare these innovations in timely manner critical for their adoption. These innovations are introduced at such rapid pace (Dean et al., 2018; arXiv ML Papers Statistics) that researchers are hard- pressed to study and compare them. As result, there an urging need by both research and industry for scalable model/HW/SW evaluation platform.
Evaluation platforms must maintain repeatability (the ability to reproduce claim) and fairness (the ability to keep all variables constant and allow one to quantify and isolate the benefits of the target of interest). For ML/DL, repeatable and fair evaluation is challenging, since there is tight coupling between model execution and the underlying HW/SW components. Model evaluation is complex process where the model, dataset, evaluation method, and HW/SW stack must work in unison to maintain the accuracy and performance claims (e.g. latency, throughput, memory usage). To maintain repeatability, authors are encouraged to publish their code, containers, and write documentation which details the usage along with HW/SW requirements (Mitchell et al., 2019; Reproducibility Checklist; Dodge et al., 2019; Lipton & Steinhardt, 2019; Pineau et al., 2018). Often, the documentation miss details which make the results not reproducible. To perform fair evaluation, evaluators have to manually normalize the underlying stack and delineate the codes to characterize performance or accuracy. This is a daunting endeavor. As consequence, repeatable and fair evaluation is a "pain-point within the community (Gundersen et al., 2018; Plesser, 2018; Ghanta et al., 2018; Hutson, 2018; Li & Talwalkar, 2019; Tatman et al., 2018: Reproducibility in Machine Learning; ICLR Reproducibility Challenge). Thus, an evaluation platform design must have standard way to specify, provision, and introspect evaluations to guarantee repeatability and fairness.
In this paper, we propose MLModelScope: distributed design which consists of specification and a runtime that enables repeatable, fair, and scalable evaluation and benchmarking. The proposed specification is text-based and encapsulates the model evaluation by defining its pre-processing, inference, post-processing pipeline steps and required SW stack. The runtime system uses the evaluation specification along with user-defined HW constraints input to provision the evalua- tion, perform benchmarking, and generate reports. More specifically, MLModelScope guarantees repeatable and fair evaluation by (1) defining novel scheme specify model evaluation which

---

Under review as a conference paper at ICLR 2019
separates the entanglement of data/coode/S/HH, (2) defining common techniques to provision workflows with specified HW/SW stacks; and (3) providing a consistent benchmarking and reporting methodology. Through careful design, MLModelScope solves the design objectives while being framework/hardware agnostic, extensible, and customizable.
In summary, this paper makes the following contributions: ① we comprehensively discuss the complexity of model evaluation and describe prerequisites for a model evaluation platform. 2 We propose a model evaluation specification and an open-source, framework/hardwwar agnostic, extensible, and customizable distributed runtime design which consumes the specification to execute model evaluation and benchmarking at scale. 3 We implemented the design with support for Caffe, Caffe2, CNTK, MXNet, PyTorch, TensorFlow. TensorRT, and TFLite, running on ARM, Power, and x86 with CPU, GPU, and FPGA. 4 For ease of use, we equip MI MoodeSccope with command line, library, and ready-made web interfaces which allows "push- We also add introspection capability in MiMooelScoope to analyze aceuracy at different stages and capture latency and memory information at different levels of the HW/SW stack. b We showcase MLModelScope by running experiments which compare different model pipelines, hardware, and frameworks.
model evaluation". S
Model
Life
procem
2 MODEL EVALUATION CHALLENGES
Coon
Model evaluation is complex. Researchers that publish and share DL models can attest to that but are sometimes unaware of the full scope of this complexity. To perform repeatable and fair evaluation, we need to be cognizant of the HW/SW stack and how it affects the accuracy and performance of a model. Figure shows our classification of the HW/SW stack levels. Model level (LT) evaluates a model by performing input pre- pre-processing, model in- ference, and procesroc.ss.gg. The pre- pre-proeessing stage transforms the user input into a form that the model expects. The model inference stage calls the framework's 's inference API on the processed input and produces an output. The post-processing stage transforms the model output to form that can be viewed by a user or used to compute metrics. Framework level ) performs model inference by executing the layers in the model graph using framework such as TensorFlow, MXNet, or Py PyToorc. Layer level
Framework
Layer
Notl od
Ouonel
Tursen
Ubrorr
Mollin
Rereal
Tourell
I Hordware
SP Hopa
Figure I: Exccution of model evaluation at different levels of hardware and software abstractions on GPUs.
(L3) executes a sequence of ML library calls for layers such as convolution, normalization, or softmax. MI. Library level (L4) invokes a chain of system library calls for functions in ML libraries such as cuDNN(Chetlur et al., 2014), MKL-DNN (MKL-DNN) or OpenBLAS (Xianyi et al., 2014). And, last but not the least, at the hardware level 55 there are CPU/GPU instructions, disk, and network I/O events, and other lew-level system operations through the entire model evaluation. All the HW/SW abstractions must work in unison to maintain the reported accuracy and performance claims. When things go awry, each level within the abstraction hierarchy can be suspect.
Currently, model authors distribute models by publishing documentation and ad hoc scripts public repositories such as GitHub. Due to the lack of specification, authors may under-specify or omit key aspects of model evaluation. This inhibits, or makes it difficult, for others repeat their evaluations or validate their claims. Thus all aspects of the model evaluation must be captured by evaluation platform to guarantee repeatability. To highlight this, consider the model evaluation pipeline at L7. While the model inference stage is relatively straight forward, the pre- and post-processing stages are surprisingly subtle and can easily introduce discrepancies in the results. Some of the discrepancies might be "silent errors" - where the evaluation is correct for the majority of the inputs but is incorrect for a small number of cases. In general, accuracy errors due to under-specifying pre- and post-processing are difficult to identify and even more difficull to debug. In Section 1, we show the effects of under-specifying different operations in pre-processing on image classification models.
The current practice of publishing models also causes few challenges which must be addressed by a fair and scalable evaluation platform. First, any two ad hoc scripts do not adhere to a consistent evaluation API. The lack of consistent API makes it difficult to evaluate models in parallel and, in turn, slows down the ability to quickly compare models across different HW/SW stacks. Second, ad
A video demo of web UT is http://dhheeggggg....oooooo///ee
2

---

Under review as a conference paper at ICLR 2019
hoc scripts tend to not clearly demarcate the stages of the model evaluation pipeline. This makes it hard to introspect and debug the evaluation. Furthermore, since an cople-to-aplle comparison between models requires fixed HW/SW stack, it is difficult to perform honest comparison between two shared models without modifying some ad hoe scripts. M.ModeeSScppe addresses these challenges through the careful design of model evaluation specification and a distributed runtime as described in Section 3.
name: Incept ionny3 nodel name version: 1.C,0 4 sarantic vezaion of nodal taak: classi ficaiiiationn zodel nodality
3 MLMODELSCOPPF DESIGN
description:
Eranework: i Haanewek inforration
name: "eneorf
franswork oeretion
We propose MLModelScope, an open-souree, frame- work/hardware agnostic, extensible and customizable distributed system design to perform model evaluation and benchmarking at scale. MI.ModelScope consists of a model evaluation specification and distributed runtime.
container: a containers used For techinecture
arm64: minx/tersart www.eaooooooooooooo..... 111:1-13-0000 303-0-0m6---0 opu
wnd64:
spu: mine/t flossl 13-0_and66 mine/tweanre Floxt1 0aadded
ppo64lo:
apu: ains/tetsoztion poresszzioooo 1o0:1-13-- 1rt---ppppdeeeep
gpu: mins/tersoz foot:o-1---1---- 133300_pp060l----- opa
TF EMABLE NIROSRAD NONFUSED 0
inpute:
scdel inguts
type: Image
ingut aodality
element_tyy styenn__type:: float32
3.1 MODEL EVALUATION MANIFEST
pre-procassing:
def prn procesxing (env, inpota):
... fe.g. irpo-t ogenov
preprod inguta
All models in MLModelScope are described using a model specification (called manifest). The manifest is a text file and includes the information needed to run a model. The manifest specifies information such as the model pre- and post-processing steps, its model sources (graph and weight), and its software stack. The hard- ware details are not present in the manifest, but are user-provided options when performing the evaluation. By decoupling the hardware specification from the manifest, a manifest can work across hardware.
outputa: bodel outpate
type: probabl
outgut modality
nane:
element
#loxt:32
p pww.essiiggigggoooo (env, inpate)
post-processing Lnn::
post,
... 5.g. os txco "Rdarrpp -/postprac..")
postproe
sourew: sodel aource
graph_path: htsps://.... so.a://f../....pp.. www.eptiee......o...
training_datased datasddttttttt dataset used for teaining
2012
version: 1.0.0
Listing I: Example evaluation manifest.
An example manifest is shown in Listing and contains model name, version, and type of task (Lines 1-3); framework name and version constraints (Lines 6-8); containers to use for evaluation (Lines 9-16); model inputs (Lines 19-22); pre-processing function (Lines 23-26); model outputs (Lines 27-30) post-processing function (Lines 31 31-34); model resources (Lines 35-36), and other metadata attributes (Lines 37-39). The key components of the manifest are:
Software Meoo--MMMMdeelSoop uses docker containers to maintain the software stacks. MLMod- elScope provides ready-made containers for all popular frameworks, but users can use any container hosted on Docker Hub. Multiple containers can be specified within the manifest. The container used for evaluation is dependent on the executing hardware and whether the system has a GPU or not.
Model Source-Model source contains links to the model graph (the graph_path field) and weights (the weights_pathl field). For frameworks which have one file to represent the graph and its weights, the weights field is omitted from the manifest. The model can be stored in the cloud, downloaded on demand, and is cached to the local file system.
Cersioning-Modess frameworks, and datasets are all versioned within MLModelScope using semantic versioning (Preston-Werner,, 2019) scheme. The versioning of frameworks and datasets supports constraints, such as ~ x (Listing 1, Line 8). This tells MLModelScope that the model works on any TensorFlow v1 framework.
type: image input modalsty
Pre-/Post-Proceesing Functions-Tl To perform input pre-processing and output post-processing, MLModelScope allows arbitrary Python functions to be placed within the manifest file. The pre- and post- processing functions have the signature def fun (env, data) where env contains metadata of the evaluation request and data is a PyOb ject representation of the user request for pre- processing and the model's output for post-processing. Internally MLModelScope executes the Python code within a Python sub- interpreter (Python Subinterpreter) in the launched container. To reduce data copy overhead parameters are passed by reference the processing functions. The pre- and post-procsssing functions are flexible; i.e. users may import external Python modules or invoke external scripts. By allowing arbi-
layes_nans: data
element type: "loat 32
stags: .pre -.re-proowes processing st-ps
decode:
element type: int8
data dayau NEWC
color layout: RGB
crop:
method: centar percentage: 87.5
dimensions: [3, 299, 299] nethod: bilinear ratio: true mean: 127,2, 127.5, 127.51
is
16
15
rescale: 127.5
Listing 2: MI ModelSeopp''s 's eval- uation manifest for Inception-v3.
3

---

Under review as a conference paper at ICLR 2019
trary pre- and post-processiing function executions, coMo works with existing processing codes and is capable of supporting arbitrary modalities.
Built-in Ctte/Posst-Prcesssing Functions-As vision models are widely used and their pre- and post-processing operations are less diverse, MLModelScope allows for common pre-processing image operations (e.g. image decoding, resizing, and normalization) and post-procsssing operations (e.g. topK, IOU, mAP, etc.) to be used within the manifest without writing code. Internally, MLModelScope invokes built-in pre- and post-processing code to perform these operations. Listing can be placed within the inputs block (Lines 19-22 in Listing I) as the pre-processing steps for Inception-v3. The steps are executed in the order that is specified, since, as we show in Section the order of operations can have a significant impact on achieved model accuracy. Users are not required to use this feature, but using this feature allows users to easily compare pre- or post-processing steps. We use this mechanism during our evaluation in Section 4.
77 Opene a predictor.
Egc Modeltoad [Opesstequaat] retuena [Medeltandle) [)
3.2 THE MLMODELSCOPE RUNTIME
Foc ModelOnload (Model8andle) zetuzas (CleseResponse)] ()
77 perforn sodal inderance, on usez data,
Egc Predict ctiquuct)) setuins (Proddictiesseepoossss ()
The MLModelScope runtime consumes the model manifest to provision evaluations and 3 functions which are specified using Protobuf.
Listing 3: ModeeSooops''s predictor RPC API consists of
perform benchmarking. Users evaluate a model by specifying its name, version, and framework along with the target hardware requirements. The MLModelScope runtime uses these user-provided constraints to query previous evaluations or schedule new ones. The runtime is distributed and is built as a set of extensible and customizable modular components (see Figure 2). Due to space limitations, we only highlight the key components of the runtime (See Appendix for a description of all components):
Framework Predictors-Ai At the core of the software stack are the frameworks. To enable uniform evaluation and maximize code reuse, MLModelScope wraps each framework's s C++ inference API to provide a uniform interface (called predictor API). The predictor API (shown in Listing 3) is minimal and performs model loading, unloading, and inference. So long as a program implements MLModeelSopp's 's predictor API, it can be plugged into the system. This means that MoModeSScopp'ss design allows for exotic hardware or framework support. For example, some hardware, such as FPGAs and ASICs, do not have a framework per se. These hardware are exposed to MoModeSScopee through program which implements the predictor API. The ModelLoad API for FPGAs, for example, downloads bitfile and load it onto the device.
The predictor API is linked against common code to perform container launching, manifest file handling, downloading of required assets, pre- and post-processing function execution, collecting of performance profiles, and publishing of results - we call this bundle an agent. These agents can be run on separate machines, can be run in parallel, and are managed by the MLModelScope orchestration layer. Agents can be run on remote systems behind firewalls to allow for model evaluation on remote hardware this allows hardware providers to give model evaluators access to perform model evaluations without full unrestricted access to the hardware. MLModelScope does require modifications to framework and thus pre-compiled binary versions of frameworks (e.g. distributed through Python's 's pip) or customized versions of a framework work within MoModelScope.
Manifest and Predictor et al., 2012) to store the model manifests and running agent information. MLModelScope's orches- tration layer leverages the registry to facilitate the discovery of models and routing of user requests across the distributed agents using the HW/SW constraints provided by the user. The registry is dynamic- - i.e. both model manifests and agents can be added and removed at runtime.
MMMoeelSccpee uses a distributed key-value registry (Escriva
Profilers and Tracers- To enable performance debugging, MI.ModelScope collects system, frame- work, and model level profiling information. This data is published into a tracing server (OpenTracing: Sigelman et al., 2010) where it gets aggregated and summarized. Through the trace, users get holistic view of the performance of model evaluation and can identify bottlenecks. To minimize overhead, the profilers are only active when user enables them as part of the evaluation request.
Web UI and Command Line Interface Usses interact with MLModelScope through its web UI or command-line interface by specifying model and hardware constraints. These constraints are used to query the database for previous evaluations or to schedule new ones. Users can integrate MI ModelScoppe within their existing tools pipelines by using its REST or RPC APIs.
4

---

Under review as a conference paper at ICLR 2019
3.3 MLMODELSCOPE EVALUATION FLOW
Web indoce
Coread
Agpeptiss
To illustrate the execution flow of a model evaluation, consider a user wanting to run Inception-v3 trained using ILSVRC 2012 on an Intel system with TensorFlow sat- isfying the "21.10.> and $1.13. O" version con- straint. The user specifies these constraints using MLMod- elScope's 's UI and invokes the model evaluation. MLMod- elScope then finds one or more systems which satisfy the user's 's constraints, sets up the environment, and launches the model evaluation within a container. The results are then published to a database for subsequent analysis.
lencl Afliee
Freark Drakne
Barked Bepty
Podilee Vnllen
Shmpplorr PPCA
IPOA Agert
Agen
Wargar
Hochare
Figure 2: MLModelScope's distributed run- time enables scalable evaluation across mod- els, frameworks, and systems.
Figure shows the evaluation flow of user's request. On system startup, each agent publishes the hardware it is running on to the registry. This information is made visible to the MLModelScope orchestration layer. @ A user then uses MLModelScopee' 's UI to request an evaluation by specifying the model, framework, and hardware constraints. An API request is then performed to the remote API handler, which then A queries the registry to find an agent which satisfies the user's constraints. 2 The request is then forwarded to one (or all) of the agents capable of running the evaluation. The agents then provision the hardware and software environment and run the model. @ The agents then collect and publish the results centralized evaluation database. Finally, an evaluation summary is presented to the user.
4 EVALUATION
We implemented the MLModelScope design as presented in Section 3 with support for popular frameworks (Caffe, Caffe2, CNTK, MXNet, PyTorch, TensorFlow, TensorRT, and TFLite) and tested on common hardware (X86, PowerPC, and ARM CPUs as well as GPU and FPGA accelerators). We populated it with over 300 models covering a wide array of inference tasks such as image classification, object detection, segmentation, image enhancement, recommendation, etc. We considered three aspects of MLModelScope for our evaluation: the effects of under-specified pre-processing on model accuracy, model performance across systems, and the ability to introspect model evaluation to identify performance bottlenecks. To demonstrate MLModelScope's 's functionality, we installed it multiple Amazon instances and performed the evaluation in parallel using highly cited image classification models.
Unless otherwise noted, all results use TensorFlow 1. 13. 0-re2 compiled from source; CUDNN 7.4; GCC 6.4.0; Intel Core i7-7820X CPU with Ubuntu 18.04.1; NVIDIA TITAN V GPU with CUDA Driver 410.72; and CUDA Runtime 10.0.1 (Amazon p3. 2xlarge Instance).
4.1 MODEL PRE- PROCESSOE
We use MooodelScope compare models with different operations in the pre-processing stage. Specifically, we look at the impact of image decoding, cropping, resizing, normalization, and data type conversion on model accuracy. For all the experiments, the post-processing is a common operation which sorts the model's output to get the top K predictions. To perform the experiments, we create variants of the original Inception-v3 (Silberman & Guadarrama, 2018; Szegedy et al., 2016) rre-processing specification (shown in Listing 2). We maintain everything else as constant with the exception to the operation of interest and evaluate the manifests through MLModeeSSopp's web UI.
Color Layout-Modeds are trained with decoded images that are in either RGB or BGR layout. For legacy reasons, OpenCV decodes images BGR layout by default and, subsequently, both Caffe and Caffe2 use the BGR layout (caffebgr). Other frameworks (such as TensorFlow and Py PoTch)) use RGB layout. Intuitively, incorrect color layout only misclassifies images which are defined by their colors. Images which are not defined by their colors, however, would be correctly classified. Figure shows the Top classifications for the same image when changing the color layout.
Data Layout-
Images are represented by: N (batch size), C' (channels), H (height), and W (width). Models are trained using data in either NCHW or NHWC form. Figure 4 shows Inception-v3's (trained using NHWC layout) T result using different layouts for the same image.
lmygut-Images
5

---

Under review as a conference paper at ICLR 2019
PIL
OpenCV
A0A Coler syyaut
Laannel
NHWC 39.8067% platypus
NCHW 31.3215% mixing bowl
Figure 3: Top 5 predictions using Inception-v3 with RGB BGR color layout.
Figure 4: Top 1 predictions using Inception-v3 with NCEW NHRC data layout.
Figure Image decoding difference between PII. and OpenCV.
dleptarr 8.567284
/dbon bush deghar) 260006
sillnn dlephart 0040009
0..37166
Wamesdo Bragon) N.OecoARRY
Komada onggen)
0.066605674
Homnaaaa 00000444344
(a)
(b)
(c)
(d)
(e)
Figure 6: Differences in the prediction results due to cropping using TensorFlow Inception-v3.
Gillagg Sulaety 1..E
+0 00000000
IPI
(a]
(b)
(c)
[d]
(8)
Figure 7: Differences due to order of operations using TensorFlow low Inception-va.
1f ms
dropout
TensorRI
conv2trelu
softmax
conv
MXNet
conv2
activation
relu
pooling
Caffe2
conv2
Coffe
conv2
ims
2.5
1rt_ volloosccudnnnnnmmm 128x122 relu, relu_small_nn_yl _n_yl T volla_scuddnn 128x128xxx__pp relureeee___eeeeeeeee_ relu_small_nn _n__v
overhead
activation fw _d_ kernel Figure 8: Performance of AlexNet with batch size 64 across frameworks Amazon P3. MI Mode
wwdnn:winogrod__nnffee nonfused" volta_sgemm. ssomm__mmmm___mmm lttpeseeeeeepeeeem 128x64_nm _add_tensor_kernan
enables one understand and debug performance bottlenecks layer and sub-layer granularity. The axis top (0 35) is duration (m.) to evaluate each layer within the model, while axis the bottom (0" "3) is duration (ms) to evaluate the kemelss within the second convolution and Relu layers.
Decoding and Color Conversion It common to use JPEG as the image data serialization for- mat (with ImageNet being stored JPEG images). Model developers use library functions such as opencv. imread, PIL. Image open, or f.. io. decode _ipeeeippgggeeeeeee to decode JPEG images. These functions may different decoding algorithms and color conversion methods. For example, we find the YCrCb to RGB color conversion to not be consistent across the PIL and OpenCV libraries. Figure shows the results of decoding image using Python's 's PIL and compares decoding
TTo increase the contrast of the image differences on paper, dilate the image (with radius 2) and adjust its pixel values the range hetween and 1.

---

Under review as a conference paper at ICLR 2019
Expected
Color Luyout
Cropping
Type Conversion
Model Name
Top1
Top5
Top1
Top5
Top1
Tops
Top1
Top5
Inception-Ve V3 (Seegedy al., 2016)
78.41%
94.07%
67.44%
88.44%
78.27%
94.24%
78.41%
94.08%
MobileNett.0 (Howard et 2017)
73.27%
91.30%
69.
82.93%
71.26%
90.17%
73.27%
91.29%
ResNet50-V1 (He et al., 2016a)
77.38%
93.58%
63.21%
85.63%
76.87%
92.82%
77.40%
93.56%
ResNet50-V2 (He al., 2016b)
77.15%
93.43%
63.35%
85.95%
75.71%
92. 72%
77.13%
93.12%
VGG16 (Simonyan & Zisserman, 2014)
73.23%
91.31%
59.17%
82.77%
71.71%
90.615
73.24%
91.33%
VGG19 (Simonyan & Zisserman, 2014)
74.13%
91.77%
60.41%
83.57%
72.66%
90.99%
74.14%
91.75%
Table I: The effects of the pre-processing on the Top I and Top 5 accuracy for heavily cited models.
with OpenCV. As shown, edge pixels are not decoded consistently, even though these are critical pixels for inference tasks such as object detection.
sometimes reported for cropped datasets, and this is often overlooked when evaluating a model. For Inception-v3, for example, input images are 87.5% center- cropped and then resized to 299 x 299. Figure 6 shows the effect of cropping on accuracy: (a) is the original image: (b) is the result of center cropping the image with 87.5% and then resizing; (c) is the result of just resizing; (d) and (f) shows the top-5 results for images (b) and (c). Intuitively, the
Cropping and Resizing-Accuracy
effects of cropping are more pronounced for images where the marginal regions are meaningful (e.g.
framed paintings).
Type Conversion and Normalization Affte decoding, the image data is in bytes and is converted to FP32 (assuming an FP32 model). Mathematically, float to byte conversion is float2byte(x) = 255x, and byte float conversion is byle2, (1)) 255.0 and are equivalent. Because of programming semantics, however, the executed behavior of byte to float conversion is byte2floath
x
255.0
The input may also be normalized to have zero mean and unit variance (pizel - Serder ). We find that the order of operations for type conversion and normalization matters. Figure 7 shows the image processing results using different order of operations for mean Byte stddevByts = 127.5 and mean meanFloat reading the image in bytes then normalizing it with both mean and standard deviation in bytes, [foal imngyte- Blte ) (c) is the result of reading an image in floats then normalizing stddevByte
Feldere
stddev = 0.5 where: (a) is the original image, (b) the result of
it with both mean and standard deviation in FP32, byte2flout (img myenn- and (d) is the difference between (b) and (c). The inference results of (b) and (c) are shown in (e) and (f).
stddeu
ca..arge ceggg c5 2xdage
Table 1 shows the effects of pre-processing operations on the top and top 5 accuracy for the entire ImageNet (Deng et al., 2009) validation dataset. The experiments are run in parallel on 4 Amazon p3. 2xlarge systems. We can see that the accuracy errors due to incorrect pre-processing might be hard to debug, since they might only affect a small subset of the inputs. For example, failure center- crop the input results in 1.45% 7.5% top accuracy difference, and 0.36% 4.22% top accuracy difference.
2 calarge a c4.laare u 2darge
16
(6) CPU Latency
p2.xaarge gaa,dargg u p3 2xlaage
400
2000
4
16
32
64
128
256
(h) GPU Thronghout
4.2 HARDWARE EVALUATION
Figure 9: Inference latency of Inception-v3 for (a) CPU and (b) GPU systems. The x- axis is the batch size, and the y-axis is la- teney in seconds for (a) and throughput in images second for (b).
We use MoMooeelSoppe to compare different hardware's 's achieved latency and throughput while fixing the model and software stack. We launch the same MLModelScope TensorFlow agent on 9 different Amazon EC2 systems recommended for DL (shown in Table 2). These systems are equipped with either GPUs or CPUs. We use MLModelScope's UI to run the evaluations in parallel across all 9 systems, and measure the achieved latency and throughput of the Inception-v3 model as the batch size varied (shown Figure 9). Using the measured latency and throughput, along with system pricing information, we calculate the cost/performance as "dollars per million images". We find that GPU instances in general are more cost-efficient than CPU instances for batched inference. We also observe that the g3s. xlarge is as cost efficient as the p3 2xlarre, because of the high price of the p3. 2x1 a rge instance.
Imbance
Hardware
Shr
CedPPrrl
plaxlange
KMI (Kopkers
4.3 FRAMEWORK EVALUATION AND INTROSPECTION
p3. 2xclape
Tola V100 SXME (Vatua, 1608
SGB
0.25
3.06
1.45
1.49
lust Platanae $1243L 4GBB
0L085
276
cSuxtange
Harrmu 8130M
0.12
2.88
We use MIModeelcoppe to compare and introspect frameworks' performance by fixing the model and hard-
2 Ieae Xumn ES-2266 v3. 3/15
0.1
5.0
4 Innet Xewa ES 2006 v2. 7,5GB
0.199
535
8 Iinel Xeon ES 2060 v3. 1538
0.508
554
Table 2:
Amazon systems used for evaluation.
We omit from Table the data layout pitfall results, since, as expected, it results in very low accuracy.
7

---

Under review as a conference paper at ICLR 2019
ware. For illustration purpose, show AlexNet, since it has less than 10 layers and fits within the paper. We use MLModelScope's 's TensorRT, MXNet, Caffe2, and Caffe agents and run them on the Amazon p3 2xlarge system. Figure 8 shows AlexNet's s latency across frameworks. To understand the performance of each framework, we use MI.ModelScope's profiler to delve deep and capture each evaluation's s layer and library performance information. Through the data, observe that MI layers across frameworks are implemented differently and dispatched to different library functions. Take the first conv2 and the following relu layers for example. In TensorRT, these two layers are fused and are mapped into two trt_volta___ wwwla__scudnn___aeeeeeeeee (eaddnn_128xll_
reluu relu__maa__eeeeeeeeeeeeeeeeaaaae ____n___________________________ kernels (Oyama al., 2018) which take 1.95ms. In Caffe2, however, the layers are not fused and take 2. 2.63ms. The sub-model profile information helps identify bottlenecks within the model inference. We can see that MoModeSScope helps understand the performance across the HW/SW stack which is key to evaluating HW/SW stack choices.
5 RELATED WORK
To encourage repeatability in ML/DL research, guidelines (Mitchell al., 2019; Dodge et al., 2019; Li & Talwalkar, 2019; Lipton & Steinhardt, 2019; Pineau et al., 2018; Reproducibility Checklist) have been developed which authors are advised to follow. These guidelines are checklists of what is required to case reproducibility and encourage model authors to publish code and write down the HW/SW constraints needed to repeat the evaluation. More often than not, model authors use note- books (Ragan-Ko Kelley et al., 2014), package managers (Fursin et al., 201 8a;b) or containers (Kurtzer et al., 2017; Godlove, 2019) to publish their code or specify the SW requirements. These SW requirements are accompanied with a description of the usage, required HW stack, and are published to public repositories (e.g. on GithHub). Through its design, MLModelScope guarantees repeat- able evaluations by codifying the model evaluation through the manifest and user-provided HW constraints.
Both industry and academia have developed consortiums to build benchmark suites that evaluate widely used models (MLPerf; MLMark; AI-Matrix; Gao et al., 2019; Li et al., 2019). These benchmark suites provide separate (non-uniform) scripts that run each model. Each researcher then uses these scripts to perform evaluations on their target HW/SW stack. MLModelScope's model pipeline specification overlaps with the demarcation used by other benchmark suites (e.g. MLPerf seperates model evaluation into pre-processing, model inference, and post-processing). MLModelScopee as evaluation platform, can incorporate models from benchmark suites so that they can benefit from the distributed evaluation, profiling, and experiment management capabilities. MLModelScope currently has models from benchmark suites such as MLPerf Inference and Alibaba's Al-Matrix built-in.
To allow for distributed evaluation, existing platforms utilize general distributed fabrics (Burns et al., 2016; Boehm et al., 2016; Hindman et al., 201 1) to perform model serving (Kubeflow; Chard et al., 2019; Novella et al., 2018; Pachyderm; Zhou et al., 2019) or experimentation (Tsay et al., 2018: FAI-PEP). MLModelScope differs in that it decouples the specification and provisioning of the model evaluation pipeline from the HW/SW stack to enable repeatable and fair evaluations. Moreover, it allows users to introspect the execution at sub-model granularity. To the best of the author's knowledge, no previous design addresses the confluence of repeatability, fairness, and introspection within scalable model evaluation the same time.
CONCLUSION
Everyday, an increasingly complex and diverse DL models as well as hardwarc/software (HW/SW) solutions are proposed - be it algorithms, frameworks, libraries, compilers, or hardware. Both industry and research are hard-pressed to quickly, thoroughly, consistently, and fairly evaluate these new innovations. This paper proposes MLModeeSccppe, which is specification along with a distributed runtime design that is scalable, extensible, and easy-to-use. Through MLModelScope, users can perform fair and repeatable comparisons across models, software stacks, and hardware. MLModelScopee s careful design of the specification, runtime, and parallel evaluation flow reduces time-to-tess test for model evaluators. With MLModelScope, we evaluate a set of representative image classification models and present insights into how different pre-processing operations, hardware, and framework selection affect model accuracy and performance.
8

---

wttp::/w.w.ggg.eeeeeepeppp/////e Under review as a conference paper at ICLR 2019
REFERENCES
AI-Matrix. Al Matrix. https://aimatrix.a, 2018. Accessed: 2019-09-20.
arXiv MI. Papers Statistics. arXiv MI. Papers Statistics. http:://arii.orrllissaatt.M.L 2019. Accessed: 2019-09-20.
Matthias Boehm, Michael W Dusenberry, Deron Eriksson, Alexandre V Evfimievski, Faraz. Makari Manshadi, Niketan Pansare, Berthold Reinwald. Frederick R Reiss, Prithviraj Sen, Arvind C Surve, et al. Systemml: Declarative machine learning on spark. Proceedings of the VLDB Endowment, (13):1425-1436, 2016.
Brendan Burns, Brian Grant, David Oppenheimer, Eric Brewer, and John Wilkes, Borg, omega, and kubernetes. 2016.
caffebgr. Image Pre-Processing Accessed: 2019-09-20.
html, 2019.
Ryan Chard, Zhuozhao Li, Kyle Chard, Logan Ward, Yadu Babuji, Anna Woodard, Steven Tuecke, Ben Blaiszik, Michael Franklin, and Ian Foster. Dlhub: Model and data serving for science. In 2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pp. 283-292. IEEE, 2019.
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro, and Evan Shelhamer, cudnn: Efficient primitives for deep learning. arXiv preprint arXiv: 1410.0799, 2014.
Jeff Dean, David Patterson, and Cliff Young. A new golden age in computer architecture: Empowering the machine-learning revolution. IEEE Micro, 38(2):21-29, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Puttern Recognition, 2009. CVPR 2009. IEEE Conference on, pp. 248-255. IEEE, 2009.
Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. Show your work: Improved reporting of experimental results. In Proceedings of EMNLP, 2019.
Robert Escriva, Bernard Wong, and Emin Gün Sirer. Hyperdex: A distributed, searchable key-value store. In Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication, pp. 25-36. ACM, 2012.
FAI-PEP. Facebook Al Performance Evaluation Platform. https://iiinhh.oommraeeebo////A 2019. Accessed: 2019-09-20.
Grigori Fursin, Anton Lokhmotov, Dmitry Savenko, and Eben Upton. collective knowledge work- flow for collaborative research into multi-objective autotuning and machine learning techniques. 2018a.
arXiv preprint
Grigori Fursin, Anton Lokhmotov. Dmitry Savenko, and Eben Upton. A Collective Knowl- edge workflow for collaborative research into multi-objective autotuning and machine learn- ing techniques. January 2018b. URL rpi3-crowd-tuning- 2oaning-2017-interrctivve
Wanling Gao, Fei Tang, Lei Wang, Jianfeng Zhan, Chunxin Lan, Chunjie Luo, Yunyou Huang, Chen Zheng, Jiahui Dai, Zheng Cao, Daoyi Zheng, Haoning Tang, Kunlin Zhan, Biao Wang. Defei Kong, Tong Wu, Minghe Yu, Chongkang Tan, Huan Li, Xinhui Tian, Yatao Li, Junchao Shao, Zhenyu Wang, Xiaoyu Wang, and Hainan Ye. Aibench: An industry standard internet service ai benchmark suite, 2019.
Sindhu Ghanta, Lior Khermosh, Sriram Subramanian, Vinay Sridhar, Swaminathan Sundararaman, Dulcardo Arteaga, Qianmei Luo, Drew Roselli, Dhananjoy Das, and Nisha Talagala. A systems perspective reproducibility in production machine learning domain. 2018

---

Under review as a conference paper at ICLR 2019
David Godlove. Singularity: Simple, secure containers for compute-driven workloads. In Proceedings of the Practice and Experience in Advanced Research Computing on Rise of the Machines (Learn-
ing). PEARC 19, pp. 24:1-24:4, New York, NY, USA, 2019. ACM. ISBN 978-1-45035 1-4503-7227-5. doi: 10. 1445/3332186333321199 URL http:://oi..cc.orrg//////// htt..aam....010.111455/333188330 145/3332118..33321222
Odd Erik Gundersen, Yolanda Gil, and David W Aha. On reproducible ai: Towards reproducible research, open science, and digital scholarship in publications. Al Magazine, 39(3);56-68, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer. 2016b.
Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D Joseph, Randy H Katz, Scott Shenker, and Ion Stoica. Mesos: A platform for fine-grained resource sharing in the data center. In NSDI, volume 11, pp. 22-22, 2011.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: (74.0v/700004611,, 2017.
Matthew Hutson. Artificial intelligence faces reproducibility crisis, 2018
ICLR Reproducibility Challenge. ICLR. https//tppoooccbbillyyyllleeeeep 2019/ 2019.
Accessed: 2019-09-20.
Kubeflow. The Machine Learning Toolkit for Kubernetes. https///www.kbeeflowoorrg 2019. Accessed: 2019-09-20.
Gregory M Kurtzer, Vanessa Sochat, and Michael W Bauer. Singularity: Scientific containers for mobility of compute. PloS one, 12(5):9077 2017.
Cheng Li, Abdul Dakkak, Jinjun Xiong, Wei Wei, Lingjie Xu, and Wen mei Hwu. Across-stack profiling and characterization of machine learning models gpus, 2019.
Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search, 2019.
Zachary C Lipton and Jacob Steinhardt. Research for practice: troubling trends in machine-learning scholarship. Communications of the ACM, 62(6))45-53, 2019.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 220-229. ACM, 2019.
MKL-DNN. Deep Neural Network Library. hattsss/gihhbb.com/mmmcmmallll 2019, Accessed: 2019-09-20.
MLMark. EEMBC MLMark Machine Learning performance on embedded edge-device platforms. http://www..eebbb.ornnlmaaatt 2019. Accessed: 2019-09-20.
MLPerf. MLPerf. https://mlperf.orgg 2019. Accessed: 2019-09-20.
Jon Ander Novella, Payam Emami Khoonsari, Stephanie Herman, Daniel Whitenack, Marco Ca- puccini, Joachim Burman, Kim Kultima, and Ola Spjuth. Container-based bioinformatics with pachyderm. bioRxiv, pp. 299032, 2018.
OpenCV. OpenCV. https://openev.orrg/ 2019. Accessed: 2019-09-20.
OpenTracing. OpenTracing: Cloud native computing foundation. http://opentracingg io, 2019.
Accessed: 2019-09-20.
10

---

Under review as a conference paper at ICLR 2019
Yosuke Oyama, Tal Ben-Nun, Torsten Hoefler, and Satoshi Matsuoka. Accelerating deep learning frameworks with micro-batches. In 2018 IEEE International Conference on Cluster Computing (CLUSTER), pp. 402-412, IEEE, 2018.
Pachyderm. Pachyderm Scalable, Reproducible Data Science. https//www.ppccydemiio 2019. Accessed: 2019-09-20.
Joelle Pineau, Genevieve Fried, Rosemary Nan Ke, and Hugo Larochelle. Iclr 2018 reproducibility challenge. In /CML workshop on Reproducibility Machine Learning, 2018.
HE Plesser. Reproducibility vs. replicability: A brief history confused terminology. Frontiers in neuroinformatics, 11:76, 2018.
Tom Preston-Werner. Semantic versioning 2.0.0. https//wwww.semver.rrg. 2019.
Python Subinterpreter. Initialization, Finalization, and Threads. hwtp::/eseepooooooo..o.oooooooee hum/Hsub-iterppreer--supp interpreter-suppor, 2019. Accessed: 2019-09-20.
html#ssub
Min Kelley, F Perez, B Granger, T Kluyver, P Ivanov, J Frederic, and M Bussonnier. The jupyter/ipython architecture: a unified view computational research, from interactive exploration
to communication and publication. In AGU Fall Meeting Abstracts, 2014.
Reproducibility Checklist. The machine learning reproducibility checklist. http://www.cs.mggill.dd 2018. Accessed: 2019-09-20.
Oppineaa/ReproducciiittyyyyCyCC
Reproducibility in Machine Learning. Reproducibility in Machine Learning. https://iclr.co
2019. Accessed: 2019-09-20.
Benjamin H Sigelman, Luiz Andre Barroso, Mike Burrows, Pat Stephenson, Manoj Plakal, Donald Beaver, Saul Jaspan, and Chandan Shanbhag. Dapper, large-scale distributed systems tracing infrastructure. Technical report, Technical report, Google, Inc, 2010.
Nathan Silberman and Sergio Guadarrama. Tensorflow-slim image classification model library. URL: 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for targe-scale image recognition. arXiv preprint arXivv 2014.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-266, 2016.
Rachael Tatman, Jake VanderPlas, and Sohier Dane. A practical taxonomy of reproducibility for machine learning research. Reproducibility in Machine Learning Workshop at ICML, 2018.
Jason Tsay, Todd Mummert, Norman Bobroff, Alan Braz, Peter Westerink, and Martin Hirzel. Runway: machine learning model experiment management tool. SysMI. 2018
Zhang Xianyi, Wang Qian, and Zaheer Chothia. Openblas. URL: hup://xianyi. github. io/OpenBLAS, 2014.
Jinan Zhou, Andrey Velichkevich, Kirill Prosvirov, Anubhav Garg, Yuji Oshima, and Debo Dutta. Katib: A distributed general automl platform on kubernetes. In 2019 USENIX Conference on Operational Machine Learning (OpML 19), pp. 55-57, 2019.

---

Under review as a conference paper at ICLR 2019
A SUPPLEMENTARY MATERIAL
ModoeeSScope is a big system, and we had to selectively choose topics due to the space limitation. This supplementary materials section is used to provide details about MLModelScope that were unable to cover in the paper's main body. Specifically, we discuss how MLModelScope: (a) incorporates the latest research and production models through model manifests by showing object detection and instance segmentation models. (b) Attracts users by providing a web interface and command line for scalable model evaluation. (c) Is built from set of modular components which allows it to be easily customized and extended.
A.1 MLMODELSCOPE MODEL MANIFESTS
Listing shows the manifest of SSD Mobi LeweeeeeeeeeeCtee an objection detection model, for TensorFlow. This model embeds the pre-processing operations in the model graph, and thus requires no normalization, cropping, or resizing. The major difference from a image classification model manifest is the task type (being ob object see_ddeeee detection and the outputs. There three output tensors for this model (boxes, probabilities, and classes). These output tensors are processed by MLModelScopp to produce single object detection feature array, which then be visualized or used to calculate the metrics (e.g. mean average precision).
850 Wob.leeell.ll
intornation your model
version:
vercior.
task: aojjst__dttcctinee detection
franework:
name: JensezFlow t dramewozk cans
version: 1.12.x 2 rranevork version contraint
container: $ cuntainers used Lu pes fusi mudel evalcacion
and64:
spu: pllm/teneertiooveendd
cpu: nicn/tenecrt www.erflwwwweeeeeeee low
#pc641e:
cpu: nlcn/caasarioooigeeeeee
spu: nlom//caacctiioosseee.....
description:
papeza websites ecc.. deser nodel
license: Apache License, Version 2,0 # License of the model
18
inpute: model inputs
19
type: image first input modality
20
element type: uints
21
layer_ name: inage_censor
22
layout: BMC
23
color_iayout: RGB
24
outputs:
25
type: box
26
elament Eype: Float32
27
layer name: dotection_boxss
28
type: preasbility
type: float32
30
1ayer_name: detect.on_scores
31
slass elenent elament__tpe: Float32
32
33
layer name: delwelion
34
teatures_url: ptlpatt/..../aalls...
35 source:
56
graph path; httpes// /asd leb:lenet_vv wow.oooo__oooooooooooooo 2l__coo__200.__0_ 20180001000010000000000000000010 20100.01_28..0 28.pb sttributes: 1 extra nodal attrsbures
37
38
Basining_datteee 0000 i datasoz used to foz training
39
mrnifest_autbocs ...
Listing 4: MLModelScope's model specification for SSD @sD_MoblleNet__vvlee
TensorFlow model.
Listing shows the manifest Mask Becc 50aNN__ReNeet____ v2 Atusuu cOCO, an instance seg- mentation model, for MXNet. The major difference from the object detection model in Listing 4 is the task type (being instance instance_segmeetttaa ionn) and the outputs. Listing 5 shows four outputs for this model (boxes, probabilities, classes, and masks). These output tensors are processed by MoModelScope to produce single instance segmentation feature array. Note that unlike TensorFlow, MXNet uses layer indices in place of layer names get the tensor objects.
of your model version: 1.0 version information in senantic varsion. fornat
name:
task: iistanco_secmentatae
the
version:
versicn contraint
containes: i conteiners to pertosn model evaluation
and64;
qpu: mwcn/mmett:ani iand6--pd
10 cpu: nlcn/mxact :ttp://kccttdddd-pppppp
ppc641e:
12 opu: slen/maael sppuélle-gpo
13
gpu: slen/msaet spe 2po641e-ppp
description: ...
15
refarences: re references to papers xebsites eto.. describing the nodel
12

---

Under review as a conference paper at ICLR 2019
Ageche License,
License
Anguts:
19
type:
fizst imput modality
20
uints
21
layout: ENC
22
color :lyou__yout:: RGB
23
outputs:
24
type: box
25
elament type: Flowt32
26
layer nane:
27
type: preatblltt
28
eiement type: f1oat32
29
Layer_nane nane:
30
type: class
31
Ziement_type, 21oa:32
Layer
htlpst//...//aballls
34
type: mack
35
float32
56
source: specities model
weights
37
base_uri: http:// www.at//Saak_kCiii.
38
graph_path: medol-synbel..ee
38
weights path: path: model 00.00.parems
40
attributes: wxiia model aitzzbuues
41
beiiiing_datasee 00co
datases used Lo Zoz Leaining
42
manifest author: ...
Listing MLModelScope's model specification for Mask RCWN_ v2 Atrusuutrustteooeeee MXNet model.
Reproduce and Analyze Diverse Machine Learning Worklands
10
Figure The ModellSScope website provides an intuitive interface to conduct experiments.
A.2 WEBSITE WORKFLOW
Although MLModelScope provides both command line and library interfaces, we find the web- site provides intuitive flow for specifying and running experiments. Figure 10 shows the flow, and a video demonstrating it can be found at https://drive. htwpiddivee.gggl..oom www.eeeoooggg.oooooee http://cooooope//dd/i///-//
cy-i--DVU-SFHHwCCC 1c53z. figure 10. users first arrive 1 MLModelScope' 's landing page. The landing page contains description of the project along with links to how to setup and install MI ModelSSpppe Users can try MI ModelScoope by clicking the demo button, which then displays @ the inference tasks exposed through the website. If user 4 selects object detection, then 5 models that are available for object detection are displayed. A user can then selects one or more models and @ selects one more systems to run the evaluation on. The input can be specified as URL, data from disk, or dataset @ and once complete the user can perform the evaluation O. This 10 will run the evaluation on the remote system and 1 display the evaluation results along with summary of the execution flow.
13

---

Under review as a conference paper at ICLR 2019
A.3
MLMODELSCOPE' RUNTIME ARCHITECTURE
In this section we describe each component in Figure 11 in detail. The runtime is designed be extensible and customizable.
User Interface
Website
Command Line
API
REST
RPC
Ciibvrry
A.3.1 USER INTERFACE AND API
Trocer
Analyser
MLModelScope be used application library. Users interact with MLModelScope application through its website, command line, its API interface. The website and command line interface allow users to evaluate and profile models without familiarity with the underlying frameworks or profiling tools. Users who wish integrate MLModelScopp within their existing tools or pipelines can use the REST or RPC APIs. They can also compile coModelScope as a standalone shared library and use it within their C/C++, Python, or Java projects.
Registry
Moneger
Eval Database
ML Artfacts
Manifests
Repo
Caffe
Predictors
CNTK
Caffe2
Tensorflow
MXNet
TensorkT
PyTorch
Hardware
GPU
FPGA
Figure ponents.
11: MLModelScope's runtime
A.3.2 ML ARTIFACTS
As discussed in the main body of the paper, replication of model accuracy and performance results is dependent on: the usage of specific HW/SW stack; the training dataset; and the pre/post-procesinng steps on the inputs and outputs. MLModelScope specifies these requirements via a model manifest file described in Section 3. The manifest tells MLModelScope the HW/SW stack to instantiate and how to evaluate the model.
Asset Versioning - Models, frameworks, and datasets are versioned using semantic versioning scheme. The MiMoodelScope middleware layer uses this information for asset management and discovery. To request model, for example, users specify model, framework, hardware, dataset constraints. MLModelScope solves the constraint and returns the predictors (systems where the model is deployed) that satisfy the constraint. The model evaluation can then be run of (or, the user request, all) the predictors.
Docker Containers To maintain the SW stack, evaluations are performed within docker containers. To facilitate user introspection of the SW stack, MLModelScope integrates with existing docker tools that allows querying images's SW environment and metadata.
CooPost-Processing Operations - MLModelScope provides the ability to perform common op- erations such as resizing, normalization, and scaling without writing code. It also allows to specify code snippets for pre/post-processing within the manifest file which are run within a Python subsession. MLModelScope is able support wide variety of models for different input modalities.
Evaluation History- - MLModelScope uses the manifest information as keys to store the evaluation results database. Users can view historical evaluations through the website or command line using query constraints similar to the ones mentioned above. MLModelScope summarizes and generates plots to aid in comparing the performance across experiment.
A.3.3 FRAMEWORK AND MODEL PREDICTORS
predictor is thin abstraction layer that exposes a framework through a common API. A predictor responsible for evaluating models (using the manifest file) and capturing the results along with the framework's profile information.A predictor publishes its HW/SW stack information to Mod elScope's 's registry startup, can have multiple instantiations across the system, and is managed by MLModelScope's middleware.
A.3.4 MIDDLEWARE
The middleware layer is composed of services and utilities for orchestrating, provisioning, aggregating, and monitoring the execution predictors - acting as a conduit between the eser-facing APIs and the internals of the system.
14

---

Under review as a conference paper at ICLR 2019
Manifest and Predictor Registry MLModelScope uses distributed key-value database to store the registered model manifests and running predictors. MLModelScope leverages the registry to facilitate discovery of models, load balancing request across predictors, and to solve user constraint for selecting the predictor (using HW/SW stack information registered). The registry is dynamic both model manifests and predictors can be added deleted at runtime throughout the lifetime of the application.
Data Manager MLModelScope data manager is responsible for downloading the assets (dataset and models) required by the model's manifest file. Assets can be hosted within MLModelScope's assets repository, or hosted externally. For example, Listing [ines 35-36) the manifest uses model that's 's stored within the MLModelScope repository, the data manager downloads this model on demand during evaluation.
Within MoooodsScops's repository, datasets are stored in an efficient data format and are placed near compute on demand. The dataset manager exposes consistent API get values and iterate through the dataset.
Tracer - The MoMooeeSScpp tracer middleware that captures the stages of the model evaluation, leverages the predictor's is framework profiling capability, and interacts with hardware and system level profiling libraries to capture fine grained metrics. The profiles do no need to reflect the wall clock time, for example, users may integrate system simulator and publish the simulated time rather than wall-clock time.
MLModelScope publishes the tracing results asynchronously to a distributed server allowing users to view a single end-to-end time line containing the pipeline traces. Users can view the entire end-to-end time line and can "zoom" into specific component (shown in Figure 1) and traverse the profile at different abstraction levels. To reduce trace overhead, users control the granularity (AI component, framework, library, or hardware) of the traces captured.
MLModelScope leverages off-the-shelf tools to enable whole AI pipeline tracing. To enable the AI pipeline tracing, users inject reference to their tracer as part the model inference API request to MLModelScope. MLModelScope then propagates its profiles to the injected application tracer instead of the MLModelScope tracer - placing them within the application time line. This allows MLModelScope to integrate with existing application time lines and allows traces to span API requests.
A.3.5 FRAMEWORKS
At time of writing, MLModdelSoppe has built-in support for Caffe, Caffe2, CNTK, MXNet, PyTorch, Tensorflow, TFLite, and TensorRT. MLModelScope uses "vanilla" unmodified versions of the frameworks and uses facilities within the framework enable layer-level profiling - this allows MLModelScope to work with binary versions of the frameworks (version distributed through Python's 's pip. for example) and support customized or different versions of the framework with no code modifications. To avoid overhead introduced by scripting languages, MIModellSope'ss supported frameworks use the frameworks' C-level API directly - consequently the evaluation profile as close to the hardware as possible.
A.3.6 HARDWARE
MLModelScope has been tested on X86, PowerPC, and ARM CPUs as well as NVIDIA's Kepler, Maxwell, Pascal, and Volta GPUs. can leverage NVIDIA Volta's TensorCores, and can also perform inference with models deployed on FPGAs. During evaluation, users can select hardware constraints such as: whether to run on CPU or GPU, type of architecture, type of interconnect, and minimum memory requirements - which MLModelScope considers when selecting a system.
A.4 MLMODELSCOPE SOURCE CODE
This project is open-source and the code spans multiple (> 15) repositories on GitHub. Thus it is difficult to anonymize for the blind review process. We happy share the links to the source code with the PC members. The links will be included in the paper after the blind review process.
15


Reviews:
[{"review_id": "rklBR5qaKr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a unified approach to specify, evaluate and benchmark different ML methods.\nWith the main goal of enforcing repeatability and faireness when testing different methods, authors propose\nan open source runtime on which 1) specify the model, 2) describe the workflow and 3) evaluate the benchmark \nof several ML algorithms and frameworks.\nThe core of the work is the definition of the so-called \"model evaluation manifest\" which consists of a formatted\ncollection of descriptive information where both hardware/software and framework versions are specified, along with\nthe set of tasks to be carried on as well as the data sources to test the methods against.\nOnce the manifest has been created, the desired hw/sw configuration is deployed on Amazon and the specified models are benchmarked.\nThis benchmarking offers several insights on the evaluation of a given ML model, by stressing out the importance of aspects that can severely bias  the final outcome of the model (e.g., pre-processing tasks, different hardware configurations or normalization of the data).\nTo describe the workflow, authors use an image classifier on a given hardware as a running example, and play with different  preprocessing methods to measure their impact on the final accuracy of the model.\n\nSome details are not well specified/clear in the work:\n1) Data exploitation. There is the possibility of testing different methods on own datasets. Given that the deployment is run on Amazon instances, what are the requirements (e.g., data must be on S3 and so on). \n2) The manifest can be injected with python scripts that, running in a container, perform the desired operations (preprocessing). It is stated that \"parameters are passed by reference\". So if you pass a \"mutable\" object (\"env\", I guess) you need to bind it to the outher scope. How this is accomplished? (globals?)\nInstead, if you pass an \"immutable\" object (\"data\", I guess), you cannot rebind the outer reference nor mutate the object. So, what's the meaning of \"passing by reference\"?\n3) Privacy and anonymity. When performing debugging, system, framework and model level profiling information are collected on a tracing server. Is this server  part of the platform?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJggCzLbcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the question of model evaluation and reproducibility in machine learning research, specifically deep learning research, and designs and tests an extensive system for evaluating and comparing models. Users specify their evaluation parameters through a text file, and this is used by the runtime, which can be interfaced through a UI or the command line, in order to carry out the evaluation. Several experiments shed interesting light on various aspects of model, framework, and hardware performance.\n\nHypothetically, if I were to design a new model and wish to evaluate its performance relative to existing SotA models, I would potentially use this system to run all of the models, including my own. That would mean that I need to \"upload\" or otherwise integrate my model into this system, and it was unclear from my reading of the paper how easy such a process would be. Similarly, I would wish to maintain similar training and evaluation conditions for my model, e.g., the same pre and post-processing, and that would involve \"extracting\" those steps from the system for use during training. I would also like to understand whether or not this is feasible and easy given the system's design.\n\nIn section 3.1, the authors write \"The hardware details are not present in the manifest, but are user-provided options when performing the evaluation.\" An example of how this operates would be useful in the paper.\n\nAs far as experiments go, I'm not sure what the main takeaway is from section 4.1. To me, the takeaway that pre-processing is important and existing models are sensitive to pre-processing is not a new finding. The results from Table 1 could certainly be obtained without the use of the proposed system, and though there would be some scaffolding involved, I don't think that the coding would not be particularly difficult. Is the takeaway that the proposed system makes it easier or faster to evaluate the effects of different types of pre-processing? Wouldn't this be most interesting at training time?\n\nI find the experiments in sections 4.2 and 4.3 interesting. In section 4.2, I'm not sure if figure 9 includes enough information or description to conclude that \"GPU instances in general are more cost-efficient than CPU instances for batched inference\", and some more detail here would be useful.\n\nGenerally, I believe that the work is well-motivated and timely, the authors seem to have done a good job in citing related work (though admittedly I don't know much about this area), and the results are supportive of the claims of the system's usefulness.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1xlT9d5oH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "* Note: I highlight I did not assess the model design, which is the main contribution of the paper, and  did not know the background of prior work of system design to really assess the novelty of the work, my score is solely based on the experiments. \n I am an expert in machine learning/computer vision, so I could assess the experiments in terms of their validity and relevance from the machine learning/vision perspective, however, I may not be the best person to evaluate the design choices of the system. Therefore I choose the low experience for my background.\n\n* Paper summary: The paper proposes a framework to evaluate machine learning models in a hardware-agnostic way.\nTo evaluate the models using this framework, the user needs to specify the pre-processing, inference, and post-processing steps and the required software/hardware stack. The authors argue that this is important to consider the  HW/SW stack to allow a fair evaluation and reproducibility. Models are specified using a model specification called manifest.\n\n* The authors assume that SW/HW stack change the results of deep learning models a lot, and this is the main assumption in this work, however, normally in practice HW/SW stack wont change the results.\n\n* I found the experiments either not related to the point of the paper or being very trivial not helping to backing up the arguments of the paper. \n\n\n* In section 4.1, the authors consider different preprocessing operations and study their impact on the model performance, however, the fact that preprocessing impact the results is trivial in machine learning. In the same section, color layout and data layout, cropping and resizing, where the authors discussed about for instance how changing the data representation from NCHW or NHWC change the results, this is also trivial, because if you change the dimensions, you need to also change the model in a way that handles this change of dimension, therefore, this is clear that the results will change accordingly as well. Such experiments does not back up the main argument of the paper, which argues for fair evaluation between neural models, nor provides informative information to the reader.\n\nOn section 4.1, the experiment of type conversion and normalization, again this is mathematically clear that the order would change the results,  let's call imgByte=x, then by substituting given\nvalues for mean and standard evaluation, equation (b) is simplified to (b) = (x-127.5)/((127.5)*(255)) \nhowever simplification of (c) results in (x/255-0.5)/0.5 = (x-0.5*255)/(0.5*255)=(x-127.5)/(0.5*255) \nthe dominator of (b) and (c) are not equal, therefore, this is trivial that the results of these two \nthe expression would not be the same. The author posed it as a new finding, but this is trivial that mathematically\nthese two equations would not be equal. Again, this experiment does not add any value to the paper.\n\nIn section 4.2, in Figure 9, the authors show a plot of the CPU latency for different batch sizes and instances,\ntogether with GPU throughput for different batch sizes, i.e., images/seconds. The authors show latency for CPU\ninstances, versus throughput for GPU instance, since these two measures are not shown for both instances, this is \nnot supported from the text, how actually authors compare this two instance and draw the conclusion that which instance is more efficient since there is no value shown for CPU throughput. Apart from that, I don't see how this section and determining if GPU or CPU instances of  Amazon compute cloud is more cost-efficient is related to the point of this paper which is on reproducibility. Also please have a look at Amazon webpage:\nhttps://docs.aws.amazon.com/dlami/latest/devguide/gpu.html\nHere, they explicitly mention that \"A GPU instance is recommended for most deep learning purposes. Training new models will be faster on a GPU instance than a CPU instance. You can scale sub-linearly when you have multi-GPU instances or if you use distributed training across many instances with GPUs.\",  so having this experiment again neither back up the arguments in the paper, nor add value to the paper.\n\n* The major issue with this submission is that the experiments are not related to the arguments of the paper, and are not conveying any message towards backing up the arguments of the paper.\n\n* Another crucial problem is that to allow a fair comparison especially in neural models, as shown in several studies(see [1] as a sample), this is important to account for random seeds and study how it impacts the model performance, to allow a fair evaluation of the models this is important to consider this factor, fair evaluation of models is argued to be the main point of this paper, however, the authors does not consider this factor in the paper, nor study it in the experiments.\n\n[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers and Iryna Gurevych\n\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a platform for benchmarking, and in particular hardware-agnostic evaluation of machine learning models. This is an important problem as our field strives for more reproducibility.

This was a very confusing paper to discuss and review, since most of the reviewers (and myself) do not know much about the area. Two of the reviewers found the paper contributions sufficient to be (weakly) accepted. The third reviewer had many issues with the work and engaged in a lengthy debate with the authors, but there was strong disagreement regarding their understanding of the scope of the paper as a Tools/Systems submission.

Given the lack of consensus, I must recommend rejection at this time, but highly encourage the authors to take the feedback into account and resubmit to a future venue.
Decision: Reject