Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: - analysis of how APIs change over time
- comparison of several commercial APIs over three years 
- the opportunity to analyze the changes in ML models predictions
strengths: - comparison of several commercial APIs could be used as a tool to choose the best API for our use case
- access to historical predictions of APIs
weaknesses: - the biggest concern: analysis of the commercial APIs based on standard datasets (a very high overfitting probability) proposes a limited use case for such results; it would be much better to propose a new dataset for such analysis; otherwise, it could be an analysis of which team added `this` dataset to the training data 
- is a year a proper interval for the benchmark APIs?
- commercial APIs are compared, but there is a lack of some standard benchmarks; hence it is sometimes hard to compare with SOTA models 
correctness: why do we evaluate commercial APIs only based on standard datasets? 
clarity: the paper is written well 
relation to prior work: it is hard to find previous related work 
additional feedback: 1. Did you consider the comparison of the commercial APIs with open-source SOTA models? How does the difference look alike?
2. Why did you choose only standard datasets for the benchmark? 
3. Do you plan to add more datasets?
4. How do you want to mitigate the potential overfitting of commercial models?
review_title: Is it worth benchmarking a commercial ML API based on a standard dataset?

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The dataset is a collection of query, output pairs gathered at different points in time from commercially available ML APIs from different providers across a range of tasks. Besides the predictions, the dataset also includes the corresponding confidence scores as returned by each API.

The authors perform an example analysis regarding model shifts using the dataset collected.

---

### Revision Aug 19, 2022
The authors addressed most of the weaknesses pointed out by the reviewers. To my ethics concern however, I received a fallacious argumentum ad populum: "All of these are public benchmark datasets that are widely used in the machine learning community". I therefore lowered my score from 9 to 8.
strengths: Accessibility: The dataset is easily accessible from the provided URL. A Python helper package is provided as well to make it easier to access slices of the dataset.

Significance: I agree with the authors that there’s a need for understanding the quality and shift of commercial ML APIs due to an increase in adoption of these products. The dataset contains 3 consecutive years of data, which is more than datasets of 1 or 2 years as referenced in related work.

Contribution: The authors consider different tasks and modalities when constructing the dataset. They also include example analyses regarding distribution shifts for different APIs.
weaknesses: Relevance: The datapoints are collected only once per year. Any analysis done using this dataset could be outdated by a year in the worst case.
correctness: The datasets used to query the commercial ML APIs are known to the research community.
clarity: The paper is clear and it is easy to understand how the dataset was constructed.
relation to prior work: The authors compare to other datasets which only consider 1 or 2 consecutive years, as compared to this work where 3 consecutive years are considered. They also include downstream analyses in the related work.
additional feedback: 1. How were the dates for data collection chosen? Is it possible to increase the frequency of data collection?
2. Will the dataset continue to be updated in future years?
review_title: Historical dataset of ML API queries and corresponding outputs from different providers

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The main proposal in the paper is a dataset of predictions of commercial machine learning apis. The tasks chosen are classification and structured prediction tasks with 3-4 datasets per task. The paper also promises to update this dataset of predictions from year to year. The paper also contains a analysis of the predictions in terms of overlap of predictions, changes in accuracy over time, changes in confidence and changes in performance by groups such as accents, race, and gender. 


strengths: - The paper presents a valuable resource which would hard to gather for many other groups.
- The future applications which this dataset would allow seem important and currently relevant - eg. applications such as the ones related to MLaaS, ML pipeline monitoring (Sec 2), and the fairness centric analysis conducted in Sec 4.1.
- The coverage of tasks included in the dataset is good and the presented analysis is interesting.

weaknesses: - It seems important for the dataset to be updated over a longer period of time, its unclear what plan is in place to ensure that this happens. 

correctness: The construction of the dataset seems sound.

clarity: The paper is well written.
relation to prior work: Prior work seems appropriately discussed.

additional feedback: - Lines 115-122: These lines describe how all of the raw outputs are converted into a consistent format in this work. What additional information is present in the outputs from the different APIs? If there is substantial metadata it would be nice to release the additional output information in the hope that it will allow future work by others not envisioned in this paper.
- Is it possible to find/make educated guesses about the specific models (or kinds of models) or training datasets for the models whose outputs are included in this dataset? This might be useful additional information.
- In a project like this it would be meaningful to solicit datasets that the broader community would like to see predictions. If possible, it may be nice to have a system for community input on which datasets should be included in the dataset.
- Line 141: "PO=1" -> "PO=0"?
- Line 159: Consider using parenthesis to make the GD(t) equation easier to read.

**Post review responses:** Dear authors, thank you for the significant additions to the paper and your responses. I have raised my score appropriately.
review_title: Interesting dataset + analysis; Likely to be a useful resource

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: In summary, the submission is about HAPI, a longitudinal "large-scale dataset" of ML predictions by cloud ML providers (APIs, or MLAAS - ML As A Service). Specifically, it consists of ~1.7M instances (each instance consisting of input, API output and metadata incl. confidence scores)  across "diverse tasks including image tagging, speech recognition and text mining from 2020 to 2022" - specifically the 6 areas of "spoken command recognition (SCR), sentiment analysis (SA), and facial emotion recognition (FER), and structured predictions including multi-label image classification (MIC), scene text recognition (STR), and named entity recognition (NER)". This is potentially beneficial for the community to study the evolution/change of such 'black boxed' APIs -- from accuracy, to algorithms employed, to mitigation strategies towards ethical issues, etc.

- - -

### Revision 20220819
In light of the authors' willingness for an R&R to incorporate suggested changes from myself and other reviewers, and that a proportion of the key issues have been addressed, I have revised my rating accordingly. This note documents this change in the interest of transparency.
Score was `5` --> revised to `6`

### Revision 20220831
Considering the newest version of the paper, score was `6` --> revised to `7`

strengths: Strengths:
1. Significance - the contribution is highly significant given today's trends in using MLAAS for cost-effective processing of large collections of data. 
2. Relevance - The ML community can gain awareness about issues (from accuracy to ethics) about the 'black box' MLAASes by observing their [emergent] behaviour over time - and such a dataset as HAPI will be beneficial in tracking these changes.
3. Accessibility and Accountability - The compilation of such a dataset is documented in Section A, specifically e.g. A.4, of the Supplementary Material. It is based on existing state-of-the art datasets - see [Ethics] and [Relation To Prior Work] below.
4. Ethical and Social Implications - The HAPI dataset might help in research on e.g. fairness and bias of MLAASes by uncovering the general 'shift' in trends and emergent behaviour over time. See [Ethics] below as well.
weaknesses: Weaknesses:
1. Significance - there is no issue regarding significance of this contribution.
2. Relevance - there is a potential issue with the choice of the MLAASes surveyed in the dataset, which might lead to arguments against using the dataset. e.g. Google APIs are overrepresented in the dataset (across the "6 areas" as alluded to in the [Summary And Contributions]). See detailed commentary in [Correctness] below.
3. Accessibility and Accountability - per (2), there is an overrepresentation of Google APIs in the study, with an underrepresentation of Amazon APIs. See [Correctness].
4. Ethical and Social Implications - closely related to (2) and (3) - to convince readers of its impartiality, the selection criteria needs to be made clear - why the particular distribution of cloud providers? This has impacts in terms of, say, social implications - if a ML practitioner needs to compare the top MLAAS providers, and seeks to use HAPI, there will be missing data as the top 3 cloud providers are represented differently in the dataset.
correctness: There are no issues in-the-main with regards to the assemblage of test datasets (chosen based on extant state-of-the-art), metrics adopted (e.g., S4.1), nor the standardisation of output instances (S3) as "the output formats of different ML APIs are often different".

However, the main issue which needs clarification, per [Weaknesses], point (3) -
This disparity of the surveyed MLAAS 'brands' are at odds with, e.g., popularity of Cloud MLAAS provide ratings (e.g., by survey - https://www.kdnuggets.com/2021/01/cloud-computing-data-science-ml-trends-2020-2022-battle-giants.html; or by market share - https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers/). 

Hence, this raises an interesting point - what is the selection criteria for the surveyed MLAASes - cost? funding? In fact, the selection criteria for the MLAASes are not documented in the main nor supplementary materials manuscript. This needs to be made explicitly clear to avoid any potential conflict of interest - see also [Ethics] below. Moreover, the funding for this project (which involves monetary considerations in using the MLAASes) needs to be disclosed.
clarity: The paper is generally well-written, albeit with some issues hampering readability.

1. There are typos through the paper. Examples include but are not limited to "Table 3: Datesets " (p.4), "Octomber 2020" (p. 3 in Table 2).

2. The tables/figure(s) are not clearly understandable at first glance.

(a) "Table 2: Prices of ML services used for each task at their evaluation time" - it is not clear what the triples mean, until the caption makes it clear (it is across X years)

(b) Table 3 - instead of using acronyms, there is enough space on the page to include a brief description of "SCR" etc (usage of acronyms are fine, but if the datasets themselves are also abbreviated, this will hamper readability).

(c) Figure 2 - which is a table but with conditional formatting - it is not clear how the table is laid-out at first glance as it is, in effect, a multidimensional table (A1 ... C3 containing 9 cells; each cell containing a mini-table).  The units could also be added to the table itself (% vs points).
Other issues about the layout: "Row 1, 2 and 3 represent spoken command recognition, sentiment analysis, and facial emotion recognition, respectively." - shouldn't this be the first set A1/A2/A3 (5 rows incl header), ... etc, for there are 5 rows to a "set"? Also the labels "spoken command recognition" etc could appear next to a band. There are no row labels for the final "horizontal set" (i.e. C1/C2/C3)


relation to prior work: The authors have claimed "HAPI is the first large-scale ML API dataset", which the reviewer accepts *prima facie*.
However, the authors need to discuss, at least in the Supplementary Material, the background information on the hows/whats of the original datasets used in constructing HAPI. 

e.g. in the Supplementary Material, "for SCR, four datasets were used: DIGIT [4], AMNIST [ 23], CMD [ 57 ], and FLUENT [ 49 ]. The sampling rate is 8 kHz, 48462 kHz, 16 kHz, and 16 kHz, respectively. " --> why these four? justifications including brief overviews about each sets strengths and weaknesses, including any foreseen limitations (see also [Documentation] below).
additional feedback: The main thing this reviewer is concerned about is the somewhat arbitrary (with apologies) choice of MLAAS per category without explaining why, as well as the limitations this could bring about - providing this explanation including Funding will alleviate any potential doubts on conflicts-of-interests etc.
review_title: Review of Chen & Jin & Eyuboglu & Re & Zaharia & Zou (2022) - updated 20220819

Meta-review: metareview: This paper proposes a dataset of commercial ML API performance through time, which is an interesting topic given the wide usage of such ML APIs, and their evolving performance (and blindspots) isn't necessarily always considered. The initial reviews had high variance: There were some very high scores, but also recommendations for rejection. There were some very valid concerns raised by the reviewers: whether or not the use of popular pre-existing standard benchmarks increases the risk of overfitted performance, the coverage of APIs and datasets considered in API, whether yearly measurements is too sparse, and if the currently collected data represents a long enough period of time (unfortunately, I don't think it'd be possible, but it'd be particularly interesting to see earlier history). These are considerations that I encourage the authors to consider moving forward, as it appears there are plans for further expansion.

After some active discussion and revisions from the authors, however, the reviewers have converged toward recommending acceptance. Given the importance of MLAAS today and likely the future, datasets such as HAPI can have wide applicability for measuring performance, weaknesses, bias, fairness, and other model characteristics. As such, I recommend acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept