Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: Sequential classification problems, where there is an existing hierarchy among the classes with one class transforming into the next set of classes, are commonly observed in many clinical settings of disease progression. Generally, the state-of-the-art methods in these settings treat all the stages as a multi-class problem without considering the hierarchy. This paper proposes a model unfolding strategy that trains a series of models of increasing complexity for each stage and progressively moves a data sample to next stages only when the first stage is detected. The proposed model leads to a reduction in both memory as well as inference time costs instead of running a single complex classifier. A Sepsis-Septic Shock prediction task and subcategory detection in CIFAR-100 dataset is used to evaluate the proposed method and baselines.
strengths_and_weaknesses: Overall this paper studies an important area of model decomposition. The key concern in the paper is experimental results. Although the model unfolding idea is interesting, the baselines used are not explained adequately. Further analysis based on the algorithm's robustness and ease of use need to be done.
- An important component of the proposed methodology is the measured confidence criterion of a model. Accurate estimation of the uncertainty of a deep learning model has been an extensive area of research. This paper lacks the comparison with other uncertainty-quantification methods like MC-dropout, Bayesian approximations, etc., which are more state-of-the-art. Additionally, the algorithms depend on the reliability of the individual stage classifiers, and thus under-trained parts classifiers can lead to faulty signals for gating operations.
- In addition, the baselines are not adequately explained, which makes it very hard to judge the comparisons properly. For example, the terms LSTM-C, LSTM-A, Multi-class LSTM (Rec.) has been used without proper explanations.
- The writing in the paper lacks clarity which makes it difficult to understand. A few examples are listed below:
1. Line 156, the process of calculating the model's confidence is absent.
2. The optimization of Objective II (Sec. 3.2.3) is not explained properly. The paper is not self-contained, with most of the discussion deferred to the Appendix.
3. Experimental settings are also not explained in the paper (Sec. 4.1). What are the features?
4. The Baseline section (line 280) is poorly written, with the baselines not matching the terms being used in the tables and figures.
5. Table 4.1.2 (line 314) is absent in the paper.
6. Several other typos need to be fixed.
questions: Please refer to above section.
limitations: Not applicable
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors propose a cost-aware 2D prediction pipeline UnfoldML to solve multi-stage classification problem under the limited resources. In order to better find the optimal trade-off between model performance and cost, the author define a search space and three gate parameters (ICK_0, IDK, ICK_1) to navigate the trade-off. In addition, two learning algorithms (hard-gating traning algorithm and soft-gating training algorithm) are proposed to solve this optimization problem with search space. Finally, two experiments on two different real-world tasks show that the proposed method can achieve good performance as well as reducing the spatio-temporal cost compared to other baseline models.

strengths_and_weaknesses: Compared to previous work, the authors defined a cost-performance search space and propose two algorithms to solve this optimization problem. The authors did lots of experiments with different models and evaluate the model performance using AUC curve instead of accuracy (can be biased). 


The proposed hard-gating training algorithm is unstable. Its performance is expected to vary with the upper bound maxA.

The format of this paper is not good. Equations and figures are too small, and there are lots of typos. For example, line 240: Hard-Gating (3); line 306: Eq. ??; line 308: figure 3; line 314: Table 4.1.2. 

Didn't find Appendix section
questions: Does the cheaper algorithm DKD have the similar prediction with the expensive MoE algorithm on gate parameters?

How long will it take to search an optimal cost-AUC tradeoff in the defined search space?

According to Table 2, The proposed method UnfoldML-S (Rec.) performs worse than the baseline Multu-class LSTM (Rec.), because both have similar AUC but the proposed method has more cost and worse early prediction hours.
limitations: For the search sapce, the vertical space is not well-designed. According to the experiments, at each stage, the authors only train models with different predefined setting of architectures and different combination of predefined feature sets.

There is no proof, experiment or data to validate that the proposed (hard-gating or soft-gating) algorithms find the optimal policy in the defined search space. It may be sub-optimal.

The authors only did experiments on two-stage classification task, which is simple on multi-stage classification.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to unfold a monolithic multi-class classifier to a series of single-stage classifiers for reducing deployment cost. The proposed method is a dynamic 2D projection pipeline which makes ICK predictions on sequential multi-stage classification tasks. The topic is interesting and this manuscript shows promising results on two real-world tasks.
strengths_and_weaknesses: Strengths:
- The topic of unfolding a multi-class classifier to a series of single-stage calssifiers is interesting
- Experiment on real-world dataset with a relatively large number of samples

Weakness:
- Methodological novelty is unclear
- Lack of experimental comparison with recent methods
- There is no theoretical analysis of the proposed method

questions: More detailed comments are as follows:
- What exactly "spatio-temporal cost" means?
- Would a lower-cost model interference must result in accuracy degradation?
- Two training algorithms (i.e., hard-gating and soft-gating), what is the relationship beween them? What is the method novelty here?
- Comparison methods in Tables are relatively out-of-date. Are there any newer methods for comparison on this reserch topic?
- In Sec. 4.2, it is claimed "our second experiment is a computer vision task demonstrating the generalizability of UnfoldML". The claimed generalizability is in terms of what? How is it validated?

Minor comments:
- Typo in Line 306, missing Eq. number.
- Tables should add standard deviations of the results.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a cost-aware prediction pipeline that "unfolds" a single multi-class classifier into a series of single-stage classifiers, reducing its deployment cost. To be more specific, three classes of gate functions are learned to maximize the system-wise accuracy while minimizing the prediction cost. 
strengths_and_weaknesses: Strengths
  - the spatial-temporal cost of inference is reduced by orders of magnitude by the method. 
  - the proposed method navigates a more complete cost-AUC trade-off space.

Weakness
  - The experiments focus mainly on two-stage tasks (such as cifar100, first on coarse classes then fine classes), it would be interesting to demonstrate the effectiveness on more than 2 stage tasks. 
  - Some explanations with a simple example (such as classifying CIFAR-100) to illustrate the method would be easier to understand. 
  - The method focuses on the "subcategory-of-interest" task, would it be generalized to classification on all classes? 

questions: The method focuses on the "subcategory-of-interest" task, would it be generalized to classification on all classes? 

limitations: The author discussed the limitation but not negative societal impacts of the work.
review_title: null

Meta-review: metareview: The paper presents a method for transforming a multi-class multimodal classifier into a multi-stage classifier, increasing the efficiency at runtime and only using the necessary modalities for prediction. Most reviewers agree that this is an important problem for the community and has a high potential for impact. Reviewer f34b raised a question about uncertainty quantification, which the authors clarified successfully. The authors also introduced an appendix for policy selection and explanation on model predictions at the reviewer's request -- as these are tangential to the paper, the effort is appreciated. The authors also answered the questions about the optimization problem asked by reviewer cok9 - the reviewer did not comment on the author response, however the answers seem pertinent.
An remaining issue is the lack of theoretical analysis of the work, though the experiments do support the authors' claims about the cost reduction. Reviewers 5CL8 and qEBg issued positive comments w.r.t the paper's strengths, specifically the real world experiments, the reduction in cost and the exploration of the cost-AUC trade-off space.
confidence: Less certain
Decision: Accept