Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: * Note: I highlight I did not assess the model design, which is the main contribution of the paper, and  did not know the background of prior work of system design to really assess the novelty of the work, my score is solely based on the experiments. 
 I am an expert in machine learning/computer vision, so I could assess the experiments in terms of their validity and relevance from the machine learning/vision perspective, however, I may not be the best person to evaluate the design choices of the system. Therefore I choose the low experience for my background.

* Paper summary: The paper proposes a framework to evaluate machine learning models in a hardware-agnostic way.
To evaluate the models using this framework, the user needs to specify the pre-processing, inference, and post-processing steps and the required software/hardware stack. The authors argue that this is important to consider the  HW/SW stack to allow a fair evaluation and reproducibility. Models are specified using a model specification called manifest.

* The authors assume that SW/HW stack change the results of deep learning models a lot, and this is the main assumption in this work, however, normally in practice HW/SW stack wont change the results.

* I found the experiments either not related to the point of the paper or being very trivial not helping to backing up the arguments of the paper. 


* In section 4.1, the authors consider different preprocessing operations and study their impact on the model performance, however, the fact that preprocessing impact the results is trivial in machine learning. In the same section, color layout and data layout, cropping and resizing, where the authors discussed about for instance how changing the data representation from NCHW or NHWC change the results, this is also trivial, because if you change the dimensions, you need to also change the model in a way that handles this change of dimension, therefore, this is clear that the results will change accordingly as well. Such experiments does not back up the main argument of the paper, which argues for fair evaluation between neural models, nor provides informative information to the reader.

On section 4.1, the experiment of type conversion and normalization, again this is mathematically clear that the order would change the results,  let's call imgByte=x, then by substituting given
values for mean and standard evaluation, equation (b) is simplified to (b) = (x-127.5)/((127.5)*(255)) 
however simplification of (c) results in (x/255-0.5)/0.5 = (x-0.5*255)/(0.5*255)=(x-127.5)/(0.5*255) 
the dominator of (b) and (c) are not equal, therefore, this is trivial that the results of these two 
the expression would not be the same. The author posed it as a new finding, but this is trivial that mathematically
these two equations would not be equal. Again, this experiment does not add any value to the paper.

In section 4.2, in Figure 9, the authors show a plot of the CPU latency for different batch sizes and instances,
together with GPU throughput for different batch sizes, i.e., images/seconds. The authors show latency for CPU
instances, versus throughput for GPU instance, since these two measures are not shown for both instances, this is 
not supported from the text, how actually authors compare this two instance and draw the conclusion that which instance is more efficient since there is no value shown for CPU throughput. Apart from that, I don't see how this section and determining if GPU or CPU instances of  Amazon compute cloud is more cost-efficient is related to the point of this paper which is on reproducibility. Also please have a look at Amazon webpage:
https://docs.aws.amazon.com/dlami/latest/devguide/gpu.html
Here, they explicitly mention that "A GPU instance is recommended for most deep learning purposes. Training new models will be faster on a GPU instance than a CPU instance. You can scale sub-linearly when you have multi-GPU instances or if you use distributed training across many instances with GPUs.",  so having this experiment again neither back up the arguments in the paper, nor add value to the paper.

* The major issue with this submission is that the experiments are not related to the arguments of the paper, and are not conveying any message towards backing up the arguments of the paper.

* Another crucial problem is that to allow a fair comparison especially in neural models, as shown in several studies(see [1] as a sample), this is important to account for random seeds and study how it impacts the model performance, to allow a fair evaluation of the models this is important to consider this factor, fair evaluation of models is argued to be the main point of this paper, however, the authors does not consider this factor in the paper, nor study it in the experiments.

[1] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks, Nils Reimers and Iryna Gurevych




review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper studies the question of model evaluation and reproducibility in machine learning research, specifically deep learning research, and designs and tests an extensive system for evaluating and comparing models. Users specify their evaluation parameters through a text file, and this is used by the runtime, which can be interfaced through a UI or the command line, in order to carry out the evaluation. Several experiments shed interesting light on various aspects of model, framework, and hardware performance.

Hypothetically, if I were to design a new model and wish to evaluate its performance relative to existing SotA models, I would potentially use this system to run all of the models, including my own. That would mean that I need to "upload" or otherwise integrate my model into this system, and it was unclear from my reading of the paper how easy such a process would be. Similarly, I would wish to maintain similar training and evaluation conditions for my model, e.g., the same pre and post-processing, and that would involve "extracting" those steps from the system for use during training. I would also like to understand whether or not this is feasible and easy given the system's design.

In section 3.1, the authors write "The hardware details are not present in the manifest, but are user-provided options when performing the evaluation." An example of how this operates would be useful in the paper.

As far as experiments go, I'm not sure what the main takeaway is from section 4.1. To me, the takeaway that pre-processing is important and existing models are sensitive to pre-processing is not a new finding. The results from Table 1 could certainly be obtained without the use of the proposed system, and though there would be some scaffolding involved, I don't think that the coding would not be particularly difficult. Is the takeaway that the proposed system makes it easier or faster to evaluate the effects of different types of pre-processing? Wouldn't this be most interesting at training time?

I find the experiments in sections 4.2 and 4.3 interesting. In section 4.2, I'm not sure if figure 9 includes enough information or description to conclude that "GPU instances in general are more cost-efficient than CPU instances for batched inference", and some more detail here would be useful.

Generally, I believe that the work is well-motivated and timely, the authors seem to have done a good job in citing related work (though admittedly I don't know much about this area), and the results are supportive of the claims of the system's usefulness.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper presents a unified approach to specify, evaluate and benchmark different ML methods.
With the main goal of enforcing repeatability and faireness when testing different methods, authors propose
an open source runtime on which 1) specify the model, 2) describe the workflow and 3) evaluate the benchmark 
of several ML algorithms and frameworks.
The core of the work is the definition of the so-called "model evaluation manifest" which consists of a formatted
collection of descriptive information where both hardware/software and framework versions are specified, along with
the set of tasks to be carried on as well as the data sources to test the methods against.
Once the manifest has been created, the desired hw/sw configuration is deployed on Amazon and the specified models are benchmarked.
This benchmarking offers several insights on the evaluation of a given ML model, by stressing out the importance of aspects that can severely bias  the final outcome of the model (e.g., pre-processing tasks, different hardware configurations or normalization of the data).
To describe the workflow, authors use an image classifier on a given hardware as a running example, and play with different  preprocessing methods to measure their impact on the final accuracy of the model.

Some details are not well specified/clear in the work:
1) Data exploitation. There is the possibility of testing different methods on own datasets. Given that the deployment is run on Amazon instances, what are the requirements (e.g., data must be on S3 and so on). 
2) The manifest can be injected with python scripts that, running in a container, perform the desired operations (preprocessing). It is stated that "parameters are passed by reference". So if you pass a "mutable" object ("env", I guess) you need to bind it to the outher scope. How this is accomplished? (globals?)
Instead, if you pass an "immutable" object ("data", I guess), you cannot rebind the outer reference nor mutate the object. So, what's the meaning of "passing by reference"?
3) Privacy and anonymity. When performing debugging, system, framework and model level profiling information are collected on a tracing server. Is this server  part of the platform?
review_title: Official Blind Review #1

Meta-review: The paper proposes a platform for benchmarking, and in particular hardware-agnostic evaluation of machine learning models. This is an important problem as our field strives for more reproducibility.

This was a very confusing paper to discuss and review, since most of the reviewers (and myself) do not know much about the area. Two of the reviewers found the paper contributions sufficient to be (weakly) accepted. The third reviewer had many issues with the work and engaged in a lengthy debate with the authors, but there was strong disagreement regarding their understanding of the scope of the paper as a Tools/Systems submission.

Given the lack of consensus, I must recommend rejection at this time, but highly encourage the authors to take the feedback into account and resubmit to a future venue.
Decision: Reject