Venue: AutoML 2023 Conference

Reviews:

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: - The authors propose a solution that offers a speed-up during inference time by using multi-stage inference, switching between a simpler model and a more complex model such as XGBoost based on a certain fraction of the total points. The authors use LR by considering the application at only a subspace at a time, where, only the most accurate subspaces are combined for the first-stage inference.
overall_review: **Questions:**

- Line 123: Creating $b^n$ subsets

    If I understand correctly, for each numerical feature we have b bins, for a binary categorical feature we have 2 bins, while for categorical features we have n_unique_cat bins, so $b^n$ is not the correct term, since for a categorical feature we can have either fewer or more bins, making the term neither a minimum, neither a maximum constraint.

- Could the authors describe the number of features and bins that were discovered for the considered datasets in Table 1?

- Is there any reason for why standard deviations were not provided for Cases 1-4 in Table 1?

- "Other datasets are a representative subset of the 20+ public datasets from [31] that we used for evaluation."

    What was the criteria for filtering the datasets and only including a subset?

- I would like the authors to add a few more datasets to their experiment. In particular, I would request 3 interesting public datasets from the AutoML Benchmark[1], KDDCup09_appetency, jasmine, Amazon_employee_access, and Higgs from [31] (binary classification problems that fit the experimental framework of the work).


[1] Gijsbers, P., LeDell, E., Thomas, J., Poirier, S., Bischl, B., & Vanschoren, J. (2019). An open source AutoML benchmark. arXiv preprint arXiv:1907.00909.

[2] The work.

**Corrections:**

**Line 38: decision trees still outperform on tabular datasets**

I would correct decision trees to gradient boosted decision trees since by themselves decision trees are weak learners.

**Line 35: ML competitions with tabular data have been dominated not by deep learning models, but by gradient boosting models**

 I believe ML competitions quite a few times include an ensembling of DNNs and GBDTs for the top solutions.

**Line 47, Deep learning tends to lose out to XGBoost on structured data and this trend is stronger when training data is limited in size.**

The related work section seems to be lacking, since there have been several works that argue the opposite, that deep learning outperforms tabular data and the works do not mention any trends that deep learning does not perform well when training data is limited [1][2]. From personal experience, I have not observed the trend either. 

Kadra, A., Lindauer, M., Hutter, F., & Grabocka, J. (2021). Well-tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34, 23928-23941.

Gorishniy, Y., Rubachev, I., Khrulkov, V., & Babenko, A. (2021). Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34, 18932-18943.
review_summary: Pros: 

- The paper proposes an interesting solution on how to handle multi-stage inference and provides a motivating example (Figure 1) on the usage of LR by putting the instances into diverse bins.
- The paper is clear.

Cons:

- The method proposed by the authors affects a specific niche, that is an organization that needs to perform a large number of predictions continuously on an already trained model. I do not believe the proposed method affects the majority of researchers in the field of AutoML, since the time taken even without using the multi-stage inference is in the range of milliseconds. Additionally, the latency reduction comes with a performance deterioration. Moreover, preparing the method has a significant overhead on the practitioner since one needs to detect the feature importances (the authors also suggest XGBoost), then organize the data into bins, run LRwithBins and detect the correct hyperparameters based on the hyperparameter n for the number of important features and b number of bins via HPO. Lastly, the practitioner would need to detect the threshold that is also dataset specific for where the performance degrades from using LRBins and XGBoost.

- There is a larger space overhead that would be required for every dataset depending on the number of features and number of bins.
actions_required_to_increase_overall_recommendation: Addressing the questions and corrections.
review_title: null

final_score: rating: 4: Weak Reject: For instance, a paper with minor technical flaws, limited impact, and/or weak evaluation.
review_content: summary_of_contributions: In many machine learning applications, real-time inference can still present a bottleneck. Hence, it is common to separate ML code into services that are queried by product code using APIs. While this approach simplifies product code by abstracting away ML internals and clarifies software architecture, it also introduces network latency and additional CPU overhead.

In this paper, the authors propose simplifying inference algorithms and embedding them directly into the product code, thereby reducing network communication. Their approach can optimize over half of the inputs for public datasets and a high-performance real-time platform that deals with tabular data. By using AutoML for both training and inference, they achieved a reduction in inference latency, CPU resources, and network communication between the application front-end and the ML back-end.

overall_review: Overall Review

Pro : 

The empirical evaluation of the proposed method is based on experiments conducted on both public and company-internal datasets from a high-performance ML platform that serves millions of real-time decisions per second. 

The evaluation shows that the proposed method, LRwBin, achieves competitive performance. In particular, the standalone performance of LRwBin demonstrates the efficiency of the approach.

Code for this project is publicly available

Cons 

Although the proposed system is mainly based on previously existing techniques, it still makes a valuable contribution to the field by combining these techniques in a novel way. 

Some parts of the paper lack technical explanations and details, which could make it difficult for readers to fully understand the system. Moreover, while the AutoML part is mentioned, it could have been better highlighted as an important aspect of the proposed approach.

Additionally, the paper discusses the tradeoff between ML performance and efficiency of inference, but this aspect could have been given more attention and elaboration, especially since it is an important consideration for practical applications.

review_summary: The paper proposes a multi-stage inference approach that includes a simplified first stage that can be embedded into product code to reduce network communication and lower CPU overhead with negligible loss in ML performance. The proposed approach relies on AutoML for the optimal choice of hyperparameters, the composition of combined bins, and choosing the separation threshold between stages of inference. The approach improves the average resource efficiency of inference, which also improves energy efficiency, and is compatible with hardware acceleration. This work could lead to significant enhancements in resource efficiency and latency.

This is a technical paper that presents a system together with an extensive empirical evaluation. The outcomes may be important in terms of required resources and are relevant for handling real-time inference in large applications context. 

The AutoML contribution is limited since AutoML is only used as a tool in the system. Moreover, this part lacks details. 

actions_required_to_increase_overall_recommendation: Better highlight AutoML part. 
Try to better highlight the contribution within the global system.
review_title: null

final_score: rating: 8: Accept: Technically sound paper with major impact and strong evaluation, with perhaps some minor flaws.
review_content: summary_of_contributions: The paper discusses the challenges in developing robust high-performance machine learning (ML) systems and proposes multi-stage inference as a solution to the bottleneck of real-time inference. The authors argue that decision tree models, such as XGBoost, outperform deep learning models on structured data, and that a simple first-stage model embedded in the product code can reduce latency and CPU resource usage. The authors validate their proposed multi-stage inference on diverse public tabular datasets and in a high-performance production system, showing minor declines in accuracy and ROC AUC but significant latency and CPU resource consumption gains.
overall_review: The paper offers several strengths:

-The paper describes a well-defined problem of creating multi-stage inference systems and proposes a practical solution to automate the process using AutoML.
-The proposed solution appears to be logically consistent with the stated problem and the experiments conducted to evaluate the approach are well-designed, producing results that support the authors' claims.
-The work is very well written and is easy to read.
-The multi-stage method proposed in the paper is a highlight, as it reduces the number of calls to the model's API, saving time and resources.

The authors provide a clear description of the infrastructure experiment setting, which is important for understanding the results and contributions of the work.

There are some concerns with the paper, including:

-As noted in the Clarity section, comparison between XGBoost and a hybrid model is confusing, and it is unclear if the hybrid model is the same as LRwBins or something different. Clarifying this is important to relate and improve traceability between tables 1 and 2.
-More information about the experimental setup could further clarify the contributions of the work. Specifically, information about the infrastructure for the remote model and product code.

review_summary: The paper propose a practical solution to automate the process of creating multi-stage inference systems for real-time inference problem. The experiments conducted to evaluate the proposed approach are well-designed, and the results support the authors' claims. The paper is well-written and easy to read. While there are some minor details that could be improved for clarity, overall, the work presents a valuable contribution to the field of AutoML and Multi-Stage inference.
actions_required_to_increase_overall_recommendation: NA
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak impact, and/or weak evaluation.
review_content: summary_of_contributions: The paper presents an approach to speed up inference (on average) in certain deployed applications, through multi-stage inference.  The core proposal is that instead of using remote calls to offload inference to servers, the customer facing or deployed component can perform inference on part of the data. 
The approach is meant only for tabular data. The approach achieves this through the proposed Logistic Regression with Bins (LRwBins) algorithm. The authors present a detailed description of the approach, and some example figures to illustrtae the same.  The autyhors also provide a section on implementation. Empirical evaluation is performed with both open source and proiprietary data sets. Additionally, figures are presented to show comparative performance degradation. Furthermore, authors present some additional details on limitations as well as future directions.
overall_review: The paper is interesting, but is lacking in several aspects. The narrative clarity is poor and information is not well presented which makes it difficult to understand the details. The technical aspects while sound, authors do not present edge cases or generalization study. The empirical evaluation uses only a few datasets. While some of them are sizable, the applicability of this approach in a broader scenario and different data characteristics is not clear. 
On the positive side, the authors provide adequate references. The analysis performed is also good, especially to compare loss of accuracy. The case of RPC causing delays in inference which is a valid problem the authors are proposing a solution for.
review_summary: I would recommend the paper not be accepted in the current format. 
actions_required_to_increase_overall_recommendation: 1. Improve clarity
2. Improve empirical evaluation
review_title: null

Decision: Accept(Workshop)