MLModelScope: A Distributed Platform for ML Model Evaluation and Benchmarking at Scale

1 INTRODUCTION . The emergence of Machine Learning ( ML ) and Deep Learning ( DL ) within a wide array of application domains has ushered in a great deal of innovation in the form of new models and hardware/software ( HW/SW ) stacks ( frameworks , libraries , compilers , and hardware accelerators ) to support these models . Being able to evaluate and compare these innovations in a timely manner is critical for their adoption . These innovations are introduced at such a rapid pace ( Dean et al. , 2018 ; arXiv ML Papers Statistics ) that researchers are hard-pressed to study and compare them . As a result , there is an urging need by both research and industry for a scalable model/HW/SW evaluation platform . Evaluation platforms must maintain repeatability ( the ability to reproduce a claim ) and fairness ( the ability to keep all variables constant and allow one to quantify and isolate the benefits of the target of interest ) . For ML/DL , repeatable and fair evaluation is challenging , since there is a tight coupling between model execution and the underlying HW/SW components . Model evaluation is a complex process where the model , dataset , evaluation method , and HW/SW stack must work in unison to maintain the accuracy and performance claims ( e.g . latency , throughput , memory usage ) . To maintain repeatability , authors are encouraged to publish their code , containers , and write documentation which details the usage along with HW/SW requirements ( Mitchell et al. , 2019 ; Reproducibility Checklist ; Dodge et al. , 2019 ; Lipton & Steinhardt , 2019 ; Pineau et al. , 2018 ) . Often , the documentation miss details which make the results not reproducible . To perform a fair evaluation , evaluators have to manually normalize the underlying stack and delineate the codes to characterize performance or accuracy . This is a daunting endeavor . As a consequence , repeatable and fair evaluation is a “ pain-point ” within the community ( Gundersen et al. , 2018 ; Plesser , 2018 ; Ghanta et al. , 2018 ; Hutson , 2018 ; Li & Talwalkar , 2019 ; Tatman et al. , 2018 ; Reproducibility in Machine Learning ; ICLR Reproducibility Challenge ) . Thus , an evaluation platform design must have a standard way to specify , provision , and introspect evaluations to guarantee repeatability and fairness . In this paper , we propose MLModelScope : a distributed design which consists of a specification and a runtime that enables repeatable , fair , and scalable evaluation and benchmarking . The proposed specification is a text-based and encapsulates the model evaluation by defining its pre-processing , inference , post-processing pipeline steps and required SW stack . The runtime system uses the evaluation specification along with user-defined HW constraints as input to provision the evaluation , perform benchmarking , and generate reports . More specifically , MLModelScope guarantees repeatable and fair evaluation by ( 1 ) defining a novel scheme to specify model evaluation which separates the entanglement of data/code/SW/HW ; ( 2 ) defining common techniques to provision workflows with specified HW/SW stacks ; and ( 3 ) providing a consistent benchmarking and reporting methodology . Through careful design , MLModelScope solves the design objectives while being framework/hardware agnostic , extensible , and customizable . In summary , this paper makes the following contributions : 1 we comprehensively discuss the complexity of model evaluation and describe prerequisites for a model evaluation platform . 2 We propose a model evaluation specification and an open-source , framework/hardware agnostic , extensible , and customizable distributed runtime design which consumes the specification to execute model evaluation and benchmarking at scale . 3 We implemented the design with support for Caffe , Caffe2 , CNTK , MXNet , PyTorch , TensorFlow , TensorRT , and TFLite , running on ARM , Power , and x86 with CPU , GPU , and FPGA . 4 For ease of use , we equip MLModelScope with command line , library , and ready-made web interfaces which allows “ push-button ” model evaluation∗ . 5 We also add introspection capability in MLModelScope to analyze accuracy at different stages and capture latency and memory information at different levels of the HW/SW stack . 6 We showcase MLModelScope by running experiments which compare different model pipelines , hardware , and frameworks . 2 MODEL EVALUATION CHALLENGES Pre-process Post-process Model Inference Conv Conv Bias Concat BN Data Softmax Relu Relu Malloc CUDNN Transpose FreeCUDNN L $ Access DRAM Access SP Flops Malloc Kernel Kernel Free Model Framework Layer Library Hardware L1 L2 L3 L4 L5 Figure 1 : Execution of a model evaluation at different levels of hardware and software abstractions on GPUs . Model evaluation is complex . Researchers that publish and share DL models can attest to that but are sometimes unaware of the full scope of this complexity . To perform repeatable and fair evaluation , we need to be cognizant of the HW/SW stack and how it affects the accuracy and performance of a model . Figure 1 shows our classification of the HW/SW stack levels . Model level ( L1 ) evaluates a model by performing input pre-processing , model inference , and post-processing . The pre-processing stage transforms the user input into a form that the model expects . The model inference stage calls the framework ’ s inference API on the processed input and produces an output . The post-processing stage transforms the model output to a form that can be viewed by a user or used to compute metrics . Framework level ( L2 ) performs model inference by executing the layers in the model graph using a framework such as TensorFlow , MXNet , or PyTorch . Layer level ( L3 ) executes a sequence of ML library calls for layers such as convolution , normalization , or softmax . ML Library level ( L4 ) invokes a chain of system library calls for functions in ML libraries such as cuDNN ( Chetlur et al. , 2014 ) , MKL-DNN ( MKL-DNN ) or OpenBLAS ( Xianyi et al. , 2014 ) . And , last but not the least , at the hardware level ( L5 ) , there are CPU/GPU instructions , disk , and network I/O events , and other low-level system operations through the entire model evaluation . All the HW/SW abstractions must work in unison to maintain the reported accuracy and performance claims . When things go awry , each level within the abstraction hierarchy can be suspect . Currently , model authors distribute models by publishing documentation and ad hoc scripts to public repositories such as GitHub . Due to the lack of specification , authors may under-specify or omit key aspects of model evaluation . This inhibits , or makes it difficult , for others to repeat their evaluations or validate their claims . Thus all aspects of the model evaluation must be captured by a evaluation platform to guarantee repeatability . To highlight this , consider the model evaluation pipeline at L1 . While the model inference stage is relatively straight forward , the pre- and post-processing stages are surprisingly subtle and can easily introduce discrepancies in the results . Some of the discrepancies might be “ silent errors ” — where the evaluation is correct for the majority of the inputs but is incorrect for a small number of cases . In general , accuracy errors due to under-specifying preand post-processing are difficult to identify and even more difficult to debug . In Section 4.1 , we show the effects of under-specifying different operations in pre-processing on image classification models . The current practice of publishing models also causes a few challenges which must be addressed by a fair and scalable evaluation platform . First , any two ad hoc scripts do not adhere to a consistent evaluation API . The lack of a consistent API makes it difficult to evaluate models in parallel and , in turn , slows down the ability to quickly compare models across different HW/SW stacks . Second , ad ∗A video demo of web UI is at https : //drive.google.com/open ? id=1LOXZ7hs_cy-i0-DVU-5FfHwdCd-1c53z . hoc scripts tend to not clearly demarcate the stages of the model evaluation pipeline . This makes it hard to introspect and debug the evaluation . Furthermore , since an apple-to-apple comparison between models requires a fixed HW/SW stack , it is difficult to perform honest comparison between two shared models without modifying some ad hoc scripts . MLModelScope addresses these challenges through the careful design of a model evaluation specification and a distributed runtime as described in Section 3 . 3 MLMODELSCOPE DESIGN 1 name : Inception-v3 # model name 2 version : 1.0.0 # semantic version of model 3 task : classification # model modality 4 license : MIT # model license 5 description : ... 6 framework : # framework information 7 name : TensorFlow 8 version : ^1.x # framework version constraint 9 container : # containers used for architecture 10 arm64 : mlms/tensorflow:1-13-0_arm64-cpu 11 amd64 : 12 cpu : mlms/tensorflow:1-13-0_amd64-cpu 13 gpu : mlms/tensorflow:1-13-0_amd64-gpu 14 ppc64le : 15 cpu : mlms/tensorflow:1-13-0_ppc64le-cpu 16 gpu : mlms/tensorflow:1-13-0_ppc64le-gpu 17 envvars : 18 - TF_ENABLE_WINOGRAD_NONFUSED : 0 19 inputs : # model inputs 20 - type : image # first input modality 21 layer_name : data 22 element_type : float32 23 pre-processing : | 24 def pre_processing ( env , inputs ) : 25 ... # e.g . import opencv as cv 26 return preproc_inputs 27 outputs : # model outputs 28 - type : probability # output modality 29 layer_name : prob 30 element_type : float32 31 post-processing : | 32 def post_processing ( env , inputs ) : 33 ... # e.g . os.exec ( `` Rscript ~/postproc.r '' ) 34 return postproc_inputs 35 source : # model source 36 graph_path : https : // ... /inception_v3.pb 37 training_dataset : # dataset used for training 38 name : ILSVRC 2012 39 version : 1.0.0 Listing 1 : Example evaluation manifest . We propose MLModelScope , an open-source , framework/hardware agnostic , extensible and customizable distributed system design to perform model evaluation and benchmarking at scale . MLModelScope consists of a model evaluation specification and a distributed runtime . 3.1 MODEL EVALUATION MANIFEST . All models in MLModelScope are described using a model specification ( called manifest ) . The manifest is a text file and includes the information needed to run a model . The manifest specifies information such as the model pre- and post-processing steps , its model sources ( graph and weight ) , and its software stack . The hardware details are not present in the manifest , but are user-provided options when performing the evaluation . By decoupling the hardware specification from the manifest , a manifest can work across hardware . An example manifest is shown in Listing 1 and contains model name , version , and type of task ( Lines 1–3 ) ; framework name and version constraints ( Lines 6–8 ) ; containers to use for evaluation ( Lines 9–16 ) ; model inputs ( Lines 19–22 ) ; pre-processing function ( Lines 23–26 ) ; model outputs ( Lines 27–30 ) ; post-processing function ( Lines 31–34 ) ; model resources ( Lines 35–36 ) ; and other metadata attributes ( Lines 37–39 ) . The key components of the manifest are : Software Stack−MLModelScope uses docker containers to maintain the software stacks . MLModelScope provides ready-made containers for all popular frameworks , but users can use any container hosted on Docker Hub . Multiple containers can be specified within the manifest . The container used for evaluation is dependent on the executing hardware and whether the system has a GPU or not . Model Source−Model source contains links to the model graph ( the graph_path field ) and weights ( the weights_path field ) . For frameworks which have one file to represent the graph and its weights , the weights field is omitted from the manifest . The model can be stored in the cloud , downloaded on demand , and is cached to the local file system . Versioning−Models , frameworks , and datasets are all versioned within MLModelScope using a semantic versioning ( Preston-Werner , 2019 ) scheme . The versioning of frameworks and datasets supports constraints , such as ^1.x ( Listing 1 , Line 8 ) . This tells MLModelScope that the model works on any TensorFlow v1 framework . 1 type : image # input modality 2 layer_name : data 3 element_type : float32 4 steps : # pre-processing steps 5 decode : 6 element_type : int8 7 data_layout : NHWC 8 color_layout : RGB 9 crop : 10 method : center 11 percentage : 87.5 12 resize : 13 dimensions : [ 3 , 299 , 299 ] 14 method : bilinear 15 keep_aspect_ratio : true 16 mean : [ 127.5 , 127.5 , 127.5 ] 17 rescale : 127.5 Listing 2 : MLModelScope ’ s evaluation manifest for Inception-v3 . Pre-/Post-Processing Functions−To perform input pre-processing and output post-processing , MLModelScope allows arbitrary Python functions to be placed within the manifest file . The pre- and postprocessing functions have the signature def fun ( env , data ) where env contains metadata of the evaluation request and data is a PyObject representation of the user request for preprocessing and the model ’ s output for post-processing . Internally MLModelScope executes the Python code within a Python subinterpreter ( Python Subinterpreter ) in the launched container . To reduce data copy overhead parameters are passed by reference to the processing functions . The pre- and post-processing functions are flexible ; i.e . users may import external Python modules or invoke external scripts . By allowing arbi- trary pre- and post-processing function executions , MLModelScope works with existing processing codes and is capable of supporting arbitrary modalities . Built-in Pre-/Post-Processing Functions−As vision models are widely used and their pre- and post-processing operations are less diverse , MLModelScope allows for common pre-processing image operations ( e.g . image decoding , resizing , and normalization ) and post-processing operations ( e.g . topK , IOU , mAP , etc . ) to be used within the manifest without writing code . Internally , MLModelScope invokes built-in pre- and post-processing code to perform these operations . Listing 2 can be placed within the inputs block ( Lines 19–22 in Listing 1 ) as the pre-processing steps for Inception-v3 . The steps are executed in the order that is specified , since , as we show in Section 4 , the order of operations can have a significant impact on achieved model accuracy . Users are not required to use this feature , but using this feature allows users to easily compare pre- or post-processing steps . We use this mechanism during our evaluation in Section 4 . 3.2 THE MLMODELSCOPE RUNTIME 1 // Opens a predictor . 2 rpc ModelLoad ( OpenRequest ) returns ( ModelHandle ) { } 3 // Close an open predictor . 4 rpc ModelUnload ( ModelHandle ) returns ( CloseResponse ) { } 5 // Perform model inference on user data . 6 rpc Predict ( PredictRequest ) returns ( PredictionResponse ) { } Listing 3 : MLModelScope ’ s predictor RPC API consists of 3 functions which are specified using Protobuf . The MLModelScope runtime consumes the model manifest to provision evaluations and perform benchmarking . Users evaluate a model by specifying its name , version , and framework along with the target hardware requirements . The MLModelScope runtime uses these user-provided constraints to query previous evaluations or schedule new ones . The runtime is distributed and is built as a set of extensible and customizable modular components ( see Figure 2 ) . Due to space limitations , we only highlight the key components of the runtime ( See Appendix for a description of all components ) : Framework Predictors−At the core of the software stack are the frameworks . To enable uniform evaluation and maximize code reuse , MLModelScope wraps each framework ’ s C++ inference API to provide a uniform interface ( called predictor API ) . The predictor API ( shown in Listing 3 ) is minimal and performs model loading , unloading , and inference . So long as a program implements MLModelScope ’ s predictor API , it can be plugged into the system . This means that MLModelScope ’ s design allows for exotic hardware or framework support . For example , some hardware , such as FPGAs and ASICs , do not have a framework per se . These hardware are exposed to MLModelScope through a program which implements the predictor API . The ModelLoad API for FPGAs , for example , downloads a bitfile and load it onto the device . The predictor API is linked against common code to perform container launching , manifest file handling , downloading of required assets , pre- and post-processing function execution , collecting of performance profiles , and publishing of results — we call this bundle an agent . These agents can be run on separate machines , can be run in parallel , and are managed by the MLModelScope orchestration layer . Agents can be run on remote systems behind firewalls to allow for model evaluation on remote hardware — this allows hardware providers to give model evaluators access to perform model evaluations without full unrestricted access to the hardware . MLModelScope does not require modifications to a framework and thus pre-compiled binary versions of frameworks ( e.g . distributed through Python ’ s pip ) or customized versions of a framework work within MLModelScope . Manifest and Predictor Registry−MLModelScope uses a distributed key-value registry ( Escriva et al. , 2012 ) to store the model manifests and running agent information . MLModelScope ’ s orchestration layer leverages the registry to facilitate the discovery of models and routing of user requests across the distributed agents using the HW/SW constraints provided by the user . The registry is dynamic — i.e . both model manifests and agents can be added and removed at runtime . Profilers and Tracers−To enable performance debugging , MLModelScope collects system , framework , and model level profiling information . This data is published into a tracing server ( OpenTracing ; Sigelman et al. , 2010 ) where it gets aggregated and summarized . Through the trace , users get a holistic view of the performance of model evaluation and can identify bottlenecks . To minimize overhead , the profilers are only active when a user enables them as part of the evaluation request . Web UI and Command Line Interface−Users interact with MLModelScope through its web UI or command-line interface by specifying model and hardware constraints . These constraints are used to query the database for previous evaluations or to schedule new ones . Users can integrate MLModelScope within their existing tools or pipelines by using its REST or RPC APIs . 3.3 MLMODELSCOPE EVALUATION FLOW To illustrate the execution flow of a model evaluation , consider a user wanting to run Inception-v3 trained using ILSVRC 2012 on an Intel system with TensorFlow satisfying the `` ≥1.10.x and ≤1.13.0 '' version constraint . The user specifies these constraints using MLModelScope ’ s UI and invokes the model evaluation . MLModelScope then finds one or more systems which satisfy the user ’ s constraints , sets up the environment , and launches the model evaluation within a container . The results are then published to a database for subsequent analysis . Figure 2 shows the evaluation flow of a user ’ s request . 1 On system startup , each agent publishes the hardware it is running on to the registry . This information is made visible to the MLModelScope orchestration layer . 2 A user then uses MLModelScope ’ s UI to request an evaluation by specifying the model , framework , and hardware constraints . 3 An API request is then performed to the remote API handler , which then 4 queries the registry to find an agent which satisfies the user ’ s constraints . 5 The request is then forwarded to one ( or all ) of the agents capable of running the evaluation . The agents then provision the hardware and software environment and run the model . 6 The agents then collect and publish the results to a centralized evaluation database . 7 Finally , an evaluation summary is presented to the user .

Summary: This paper studies the question of model evaluation and reproducibility in machine learning research, specifically deep learning research, and designs and tests an extensive system for evaluating and comparing models. Users specify their evaluation parameters through a text file, and this is used by the runtime, which can be interfaced through a UI or the command line, in order to carry out the evaluation. Several experiments shed interesting light on various aspects of model, framework, and hardware performance.