UnfoldML: Cost-Aware and Uncertainty-Based Dynamic 2D Prediction for Multi-Stage Classification
Authors: Yanbo Xu
Alind Khare
Glenn Matlin
Monish Ramadoss
Rishikesan Kamaleswaran
Chao Zhang
Alexey Tumanov
Venue: NeurIPS 2022 Accept (2022)
Keywords: Cost-aware
Prediction pipeline
Multi-stage cascade
Uncertainty
ML for health
Systems for ML

Abstract: Machine Learning (ML) research has focused on maximizing the accuracy of predictive tasks. ML models, however, are increasingly more complex, resource intensive, and costlier to deploy in resource-constrained environments. These issues are exacerbated for prediction tasks with sequential classification on progressively transitioned stages with “happens-before” relation between them.We argue that it is possible to “unfold” a monolithic single multi-class classifier, typically trained for all stages using all data, into a series of single-stage classifiers. Each single- stage classifier can be cascaded gradually from cheaper to more expensive binary classifiers that are trained using only the necessary data modalities or features required for that stage. UnfoldML is a cost-aware and uncertainty-based dynamic 2D prediction pipeline for multi-stage classification that enables (1) navigation of the accuracy/cost tradeoff space, (2) reducing the spatio-temporal cost of inference by orders of magnitude, and (3) early prediction on proceeding stages. UnfoldML achieves orders of magnitude better cost in clinical settings, while detecting multi- stage disease development in real time. It achieves within 0.1% accuracy from the highest-performing multi-class baseline, while saving close to 20X on spatio- temporal cost of inference and earlier (3.5hrs) disease onset prediction. We also show that UnfoldML generalizes to image classification, where it can predict different level of labels (from coarse to fine) given different level of abstractions of a image, saving close to 5X cost with as little as 0.4% accuracy reduction.

UnfoldML: Cost-Aware and Uncertainty-Based Dynamic 2D Prediction for Multi-Stage Classification
Yanbo Xu--x, Alind Khare",; Glenn Matlin', Monish Ramadoss', Rishikesan Kamaleswaran', Chao Zhang', Alexey Tumanoy! GGeorgia Institute of Technology, Emory University Atlanta, GA
Abstract
Machine Learning (ML) research has focused on maximizing the accuracy of predictive tasks. ML models, however, are increasingly more complex, resource intensive, and costlier to deploy in resource-constrained environments. These issues are exacerbated for multi-stage prediction where stages transition in progression with "happens-before relationship. We argue that it is possible to "unfold" monolithic single multi-class classifier, typically trained for all stages using all data, into series of single-stage classifiers. Each single-stage classifier can be then cascaded gradually from cheaper to more expensive classifiers, which are trained using only the necessary data modalities features required for that stage. Hence, we propose Unf o oldML, a cost-aware and uncertainty-basee dynamic 2D prediction pipeline for multi-stage classification that enables (1) navigation of the accuracy/cost tradeoff space, (2) reduction in spatio-temporal cost of inference, and (3) early prediction on proceeding stages. UnfoldML achieves orders of magnitude better cost in clinical seltings, while delecting multi-stage discase development in multi-
real time. It achieves within 0.1% accuracy from the nighest-performine class baseline, while saving close to 20X on spatio-temporal cost of inference and earlier (3.5hrs) disease onset prediction. We also show that UnfoldML generalizes to image classification, where it can predict different level of labels (from coarse to fine) given different level of abstractions of image, saving close 5X cost with as little as 0.4% accuracy reduction.
Introduction
Machine Learning (ML) research has mostly focused improving prediction accuracy for classifica- tion tasks, such as image classification (Foret et al., 2020; Xie et al., 2017), disease risk prediction (Feng al., 2020; Xu et al., 2018), pedestrian detection (Zhang et al., 2018; Cai et al., 2015), The understandable drive for high accuracy has often resulted in deeper, complex neural networks, which can incur high memory (spatial cost) and high latency (temporal cost) at inference time. However, the deployment of ML applications must be cost-aware. Run time environment like mobile devices or bedside-patient monitors are commonly resource constrained, and applications that can be offloaded to cloud computing always aim for reduced cloud bill. In this paper, we focus on developing a pipeline that can balance between high prediction accuracy and low spatio-temporal cost in deploying ML classification models.
We consider scenario of deploying MI classifiers in multi-stage classification task where predicted class can progressively transform to a next stage of classes, characterized as "happens- before" relationship between classes. This task is commonly observed in many real-world applications. For instance in clinical settings, disease progression is often identified by a series of stage transitions
Authors contributed equally to this research.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

Stage-1 1
Stage-2
Stage-3
peotttttii yrch
when lacking contidance (IDK)
Absttact Inget
Detailed Input
More Input
Ieput
pnodittitoiit peats Por His) V-1
Casciées uparrid conther
Beck Stage
JCK, closs
IDK dass:
ICK, dass
carsine
Usaomin
High canamyy
Coufidently trassition query to nest stage (ICK,). or early exit the pipeline (ICK)
Rotam
Cascado
Pristtiae
apgraade
and early
models
pondiction
(a) The 2D query propagation mechanism in UnfoldMA
(b) IDK and ICK classes
Figure 1: 2D uncertainty-baet based propagation OnfoldML: Queries that are confidently low risk will return ICKo and be monitored by cheaper models; queries that are hard to predict will return IDK K and be advanced to costlier but more confident models; queries that are in confidently high risk will return IC' and be transited to next stage.
(Sperling et al., 2011; Singer el al., 2016). Delection on early stage the discase allows doctors take appropriate actions in time before it enters into late severe stages. In image classifications, recogmition from super classes to sub classes
coarse-to fine manner has shown improved classification performance (Dutt et al., 2017; Lei et al., 2017). In all these applications, general multi-class classifiers (Liu al., 2019; Fagerström al., 2019; Foret al., 2020) have been developed by treating all the stages multi classes and achieved state-of-the-art prediction performance, but at significantly high spatio-temporal cost. Prior work (Wang et al., 2017) trades off accuracy for low cost but still ignores the key relationship between the classes, so it fails to find the optimal trade-off.
We propose UnfoldML2 cost-aware and uncertainly-based prediction pipeline for dynamic multi- stage classificatton.. It "unfolds" a monolithic multi-class classifier into series of single-stage classifiers, reducing its deployment cost. Each single-stage classifier is then cascaded gradually from cheaper to more expensive binary classifiers. further reducing the cost by dynamically selecting an appropriate classifier for an input query. Figure summarizes the two dimensional (2D) query propagation mechanism designed in UnfoldML. Horizontally it allows query to transition through multiple stages, and vertically it allows the query to progressively upgrade to costlier models constrained by the pre-specified budget limit. It computes classifier's prediction confidence on a query then directs the query through one of the following three gates: 1) "I confidently know NO (IC which rejects the current query and early exits from the pipeline (exil); 2) "I don't 't know" " (LDK). which upgrades the query to higher accuracy but costlier model, producing more confident result within the budget (vertical cascading); and 3) "I confidently know YES" (ICK)), which transitions the query to the next stage of the prediction task (horizontal forwarding). The design of IC Kg gate allows queries to early exit from the Unf ol dML so it reduces the overall spatio-temporal cost of the pipeline. The ICK, gate allows queries to faster transition to next stages so it enables early prediction on late stages, which can be critical in clinical settings (Reyna et al., 2019). Overall, the combined 2D propagation mechanism uniquely enables the navigation of the cost/accuracy tradeoff space for searching an optimal of policies for dynamic model selection at inference time.
We also propose two training algorithms for learning the optimal policies for the designed 2D query propagation in UnfoldML. The key idea of the training algorithms is to learn the optimal thresholds on the gating functions ICKr IDK and ICK,, The first proposed hard-gating algorithm assumes the gating functions to be step functions parameterized by deterministic thresholds on prediction confidence. To find the optimal thresholds, it performs bottom-up grid search over a topologically- sorted model list and stops the cutoff's that minimize the prediction loss subject to the cost constraint. The secondly proposed soft-gating algorithm defines the gating functions to be probabilistic activation functions. It follows Mixture-of-Exxeer (MoE) framework to adaptively determine threshold for one model conditionally on all the other model's prediction confidence for the given query. To obviate the cost of running all models in MoE, we further propose Dirichlet Knowledge Distillation (DKD) to run only a cheap multi-labed classifier that is trained for distilling the Bayesian predictive uncertainties of all models.
We summarize the contributions of this paper as follows:
2Code available sttps://github.... com/ wwwo////tcch-//eeeeeeeeeeeeeeeee sysml/unfoldm
2

---

We design a novel 2D query propagation pipeline that "unfolds" multi-stage prediction workflows by leveraging the "happens-before" relationship between the stages, and achieves a lower-cost prediction pipeline with minimal accuracy degradation.
We propose two learning algorithms for sufficiently navigating the cost/accuracy tradcoff space and searching for the optimal policies realizing the designed 2D query propagation.
We apply the proposed pipeline to two real-world applications and demonstrate it reduces the spatio-tempon cost of inference by orders of magniude.
2
Related Work
The most relevant work to our proposed method is the one-step IDK cascade (Wang et al., 2017), which incorporates prior work of "I don't 't know" (IDK) classes (Trappenberg and Back, 2000; Khani et al., 2016) into cascade construction and introduce a latency-aware objective into the construction comparing with previous cascaded prediction frameworks (Rowley al., 1998; Viola and Jones, 2004; Angelova et al., 2015). Another group of work focus on the problem of feature selection assuming each feature can be acquired for cost. They train cascade of classifiers for optimizing the trade-off between the expected classification error and the feature cost. Early solution (Raykar et al., 2010) limits the cascade to family of linear discriminating functions. Cai et al. (2015) applies boosting method for cascading a set of weak learners. Recent methods (Trapeznikov and Saligrama, 2013; Clertant et al., 2019; Janisch et al., 2019) develop POMDP- frameworks and incorporate deep Q-learning in training the cascades. In contrast to all of the above work that are only 1-D pipelines for one-step prediction task (can be multi-class classifications), our method extends 2D pipeline that can dynamically forward examples to next steps after they are confidently predicted as passed on the current step. Further, we also develop a more efficient pipeline framework based on Mixture-of-Fei of-Experts (MoE) modeling and knowledge distillation, which can apply gradient decent algorithms for learning the parameters efficiently.
The idea of MoE was originally introduced by Jacobs et al. (1991), for partitioning the training data and feeding them into separate neural networks during the learning process. This gate decision design is applied into many domains such as language modeling (Ma et al., 2018), video captioning (Wang et al., 2019), multi-tasking learning (Ma et al., 2018). is also used in network architecture searching (Eigen et al., 2013) by setting gate activation on network layers. Sparse gates are introduced in MoE so that it can efficiently select from thousands of sub-networks (Shazeer et al., 2017) as well as increases the representation power of large convolutional networks by only using a shallow embedding network to produce the mixture weights (Wang al., 2020). We incorporate the idea of sparsely gated MoE (Shazeer et al., 2017; Wang et al., 2020) into our prediction framework, and design soft-gating training algorithm by using ReLU as the sparse gating function and imposing L1-norm regularization on the gating weights for further sparsity.
Confidence criterion has been incorporated into active learning by Li and Sethi (2006) and then extended by Zhu et al. (2010). Lei (2014) proposed confidence based classifiers that identifies the confident region (like JCK class) and uncertain region (like IDK class) in predictions. Confidence are also introduced into word embedding (Vilnis and McCallum. 2015; Athiwaratkun and Wilson, 2018) and graph representations (Orbach and Crammer, 2012: Vashishth et al., 2019). Our method posits thresholds on prediction confidence for activating the gates in pipeline expansion. Bayesian Prior Networks (BPNs) (Malinin and Gales, 2018) have been proposed estimate the uncertainty distribution in model predictions, which is more computationally efficient than traditional Bayesian approaches (MacKay, 1992; Mackay, 1992; Hinton and Van Camp, 1993). We propose Dirichlet Knowledge Distillation (DKD) based on BPNs for distilling prediction uncertainty in large models so that we only need to run a low-cost multi-head model for producing the weights in MoE efficiently.
3 Multi-Stage Dynamic Prediction Pipeline
We introduce a dynamic 2D prediction pipeline UnfoldML, which learns optimal policies for making "I corfidently know" (TCK) predictions on sequential multi-stage classification tasks. An optimal policy will effectively trade off prediction accuracy against spatio-temporal costs in order to maximize the overall system accuracy's AUC while staying under user-imposed cost constraints.

---

3.1 Problem Formulation
Given a E X' at time t, sulti-stage pipeline decides whether the individual should maintain at the current stage s or progress into the next stage s + 1 for time t + 1. If there are total number of stages that need to be detected, we train K, number of models for each specific stage s to form model zoo M {{ms1,... ....,, M,K, K })s=1 We measure each model's is spatio-temporal cost by multiplying the device cost per unit time with the serving time per prediction stage, denoted as cost co)t(m,))) for any model mak M.
To optimize the limited system resources, we design 2D Unf olMML in the following way: (1) start with the simplest model to predict the inttial stage on an incoming data, and (2) upgrade vertically to a costlier model if the current model returns" "I don't 't know" (IDK). or (3) transition horizontally to the next stage in the pipeline if the current model returns "I confidently know YES" (ICK)), or otherwise (4) exit the pipeline if the current model returns "I confidently know NO" (ICKo). Figure I (a) demonstrates the proposed 2D architecture of UnfoldML. The central problem of UnfoldML to learn an optimal policy that specifies the three classes of gating functions with an objective of maximizing the sysde accuracy while minimizing the overall prediction cost.
We formulate UnfoldMo as decision rule mapping function m case X x M + M. which takes the query data Iy coming at time f and the current model choice ma as inputs and determine whether the model for the query should take one of the aforementioned three actions: upgrade vertically, transition horizontally, or exir the pipeline. These decisions rules can be realized by two groups of parameters: a confidence criterion q X x M -> [o, 1] that measures the confidence score of a model's 's prediction on query data, and two gate functions G GICK1 [0, 1] x (True, False) that are applied to the defined confidence score. The two exclusive gates 1DK and ICK, each respectively decides if the current prediction belongs an IDK class such that (s.t.) the system will upgrade the query to a costlirr model but still remaining within the user-defined cost budget, or an ICK, class s.t. the system will transition the query the next stage in the pipeline. The third gate GICKo is then determined as GGIKK A GGCCKi Formally, we can write the decision rule used at each prediction stage as follows
m.(k+1); (IN++I);,IDK GIDK A GGCCK -GICK (daa(x))):
case (Te, m,kig,
m(s+1]1; -GIDK A G (x:)),
mak: -GIDK A ICCK (vox(x)))
where qs (x;)) is a short notation for q( Myg) measuring the confidence of model mex 's prediction on data xg. The goal of configuring optimal pipeline given restricted computation resource be formalized as the following optimization problem:
min (m , ; D) s.t. cost(m ; S
(2)
where G consists of the two gate functions GIDK and GICK; L denotes the end-to-end prediction loss on data D and is a user-specified cost-constraint for the system.
3.2 Gate Parameters Learning
Given training data set (it should be a different set from the data that was used for training the model
zoo M) D { (x.. (ut ft),- .... , (.))) where x; (Ix11". 2xT,) ) is input sequence observed for individual i, yf E {0, indicates whether the individual entered to stage-s, and if yes, we use to E [ Tī] to denote the time index when it entered. We first partition the multi-stage data (Pppee-'-77 V = 1)} J (( -2y vf
0)}. then divide the learning of IDK and ICK gate parameters into two separable nub-problems:
into S one-stage data sets: D
Sub-Objective 1: min L (mmmm, s.t. cost(m G!DK
case, DD) I C, $ = 1, 1, S
(3)
Sub-Objective 2: min GJCK;
where L is the one-stage prediction loss on data Dd, and c, is the cost budget that is pre-allocated for stage-s satisfying E. C. c. Decomposing the end-to-end optimization problem in Eq. (2) into two sub-problems in Eq. (2) allows us to parallelize the training process. We can efficiently learn each stage's 's optimal / DDK gate parameters by solving Sub-Objective 1, and Iearn the optimal ICK, gate parameters by fixing the IDK gate parameters as the optimal values GIDK" learnt from last step and then solving Sub-Objective 2.
4

---

3.2.1 Hard-gating Training Algorithm
In this algorithm, we assume G to be hard-gating function that is parameterized by cutoff ask s.t. the gate is only activated the level of confidence the prediction is below the threshold.
Hard-gating: G (z))(z1)) = (()))()))) < ask);
(4)
where I(.) is an indicator function. Based on the problem definition in Eq. (1), a model makk at stage can only be activated if (Z)) 2 ask) A (zz) < axy)) = 1. So we can write the conditional probability of being at stage-s given an input query It as
Pr(y" = 1/ m) E (x) 2 a.sh) I[ I(q,5(z;) (x,,) < (xa)) mak (xi),
where a (at)) = Pr(y"- = xt; maa). The Sub-Objective 1 loss in hard-Gating algorithm is defined as the negative log-likelihood loss
full - D*) - Ei Σt= Vi log p.c + (1 vit) llog(1 p(t)
(5)
where P. Pr(y" 1 Dit; m ). yir = only y. and t [t! 5,, t!] for some &, time-steps we wish to early detect the next stage + 1.
Algorithm in Appendix 5.1 describes bottom-up grid search algorithm for learning hard-gating parameters. We first sort the model list M {m/,. M,,, in a monotonically increasing order w.r.t both the AUC score and spatial-tempore cost that were provided in model's profiles; this will remove any sub-op sub-optimal models. For any stage s. Algorithm I starts by assigning all the N samples for that stage the first model m,1, and performing a grid search on the gate parameters a,k 's for level & = 1 to Kg. It gradually assigns 7 TDK queries to the next level's model in the list until the cost exceeds the user-defined budget c₂ for that stage. In each iteration of searching the cutoff ask. we set an upper bound maz.A on the maximum searching value to avoid overuupgrading Without setting this bound, the algorithm could overfit and put all the queries into the IDK class, then assign them all to the next level's 's model. This consumes the cost quota quickly, and prevents those highly IDK queries from exploring costlier models in the list.
3.2.2 Soft-gating Training Algorithm
In contrast to grid search on the thresholds, propose soft-gating algorithm formulating an objective function that can be efficiently solved using gradient descent algorithms. In this algorithm, we define the gate function G to be a ReLU function parameterized by a pair of coeflicients (a.k. bat) s.t. the gate is only activated if the linear product ask (0,(zz) - b.t > 0. Formally we define Soft-gating as
Boft-gating: G (ppx( = ReLU(asx (xx())) bzk).
(6)
Therefore the conditional probability of being at the stage-s given inpul o, can be defined as a mixture of the K₂ models available for stage-s prediction:
Pr(y" = 1 1/2;; m= Ek=n GIDK (g. mAs (a:)) Ejnn G (4.(()))))))
Now, the negative log-likelihood loss C. becomes solvable using gradient descent algorithms. However, the normalization term in the mixture of experts requires running all the candidates models in the zoo, which conflicts with our cost-saving goal. Therefore, we propose a Dirichlet Knowledge Distillation (DKD) procedure for training& a small surrogate model per each stage to approximate the prediction confidence of a model without truly rumning it on query. The smaller distilled model only needs to be run once per query, requiring much less cost than running all the models. In our experiment, we utilize the first model m,, from each stage, take the embedding of h₂ (æx) prior to the last activation layers in m.1 and feed into a 4-layer Ky-head Multi-Layer Perceptron (MIP) that is defined as our distilled model in the DKD procedure.
In details, the idea of DKD to posit Dirichlet prior distribution over the parameters л characteriz- ing the predicted output categorical distribution (i.c., binomial in our setup) and surrogate prior network f is fit generate the concentration parameters 0t the prior:
Pr( 24; m.k) Dir(T (iskk) Olsk (askk,o, 1sk,1) f mak).
5

---

If the learnt concentration parameters yield flat prior distribution, it means high uncertainty in the model prediction; if they yield a sharp prior distribution, means low uncertainty. Then an estimation Ωsk (zt) of the confidence score can be computed from the expected predictive probability Pox (Zte) = Di [T]] = / + 1.. For training the DKD model, which defined as a -hehe MLP per each stage in this paper, write the loss function for head k the Kullback-Leiller (KL) divergence between the prior distribution and empirical observed distribution:
((ax)) = Σ
N. T, (L((ir(T)(ass) Paa())))
where Pa=()..) are the true predicted probabilities produced from model m, on input D..- Addition- ally, also add the cross-entropy loss as auxiliary loss when training the DKD.
Once the distilled model is trained, we can replace the 4sy (xy( with the estimated values 9. (3t) s in Eq. (7), and rewrite the loss Lal in Eq. (5) accordingly. Further, we can ensure the selected models do not exceed user-imposed cost constraint by enforeing sparse gating weights over the model choices. Therefore, we define the Sub-Objective loss in soft-gating algorithm
min (m casc, ; + X + u sparse> GIDK
where the second term Lunt takes the cost constraint, controlled by X > 0 and the third term Liparse imposes further sparse regularization on the gating weights w.r.t their L1 norms, controlled by > O. More specifically, we write the last two terms as Lęear (max(0, (ost cx))² 2 and Ciparse Er, || (zx)))) Now wecan use stochastic gradient descent algorithms to learn the optimal a' and b' minimizing the above loss. Algorithm in Appendix 5.1 summarizes the soft- sattngatig training algorithm.
3.2.3 Overall Training Algorithm
To complete the overall training algorithm,, we need to learn the optimal gate parameter GICKi in Sub- Sub-Objective 2. Given the model Ma picked by one-stage IDK cascade (cither using hord-ggting or soft- gating algorithms to solve aab-Objective 1) for data query æ; at stage s, we grid search for the optimal thresholds 0 's the predictive probabilities s.t. the type error class ICK, and type II error on IC are both minimized. Several existing methods (Liu, 2012; Perkins and Schisterman, 2006; Unal, 2017; Miller and Siegmund, 1982) have been proposed for minimizing both type error and type 1l error in various ways, we pick the Closet-to-(O,,)) (Perkins and Schisterman, 2006) method that finds the optimal threshold achieving the most left upper corner in the ROC curve. Finally, Algorithm& 3 in Appendix 5.1 gives the end-to-end training Algorithm for UnfoldML, where to learn the optimal IDK gating policy. we can use either Algorithm
Experiments
We evaluate UnfoldML on two real-world tasks. The first task is to predict and when a patient who was newly admitted into the Intensive Care Unit (ICU) of hospital will develop sepsis (Stage- 1), which may worsen and progress into septic shock (Stage-2). The second task is to detect the subcategory-of-inees which utilizes the label hierarchy first filter out queries that are not in the coarse class of interest in Stage-1 and then refine the predictions into fine classes in Stage-2.
4.1 Task 1: Sepsis-Septic Shock prediction
We use MIMIC-III Critical Care Database (Johnson et al., 2016). The database consists of deidentified health records from over 50, 000 critically ill patients who stayed in the ICUs of the Beth Isracl Deaconess Medical Center between 2001 and 2012. Wc detail our data preparation in Appendix 5.2 The final cohort includes total of 34, 475 ICU patients, from which 270 (6.8%) presented with Sepsis, from which a total of 229 (9.7%) progressed into Septic Shock. We randomly split our cohort of patient data into a training set (70%), validation set (20%) and test set (10%). First,
the training set to train a set of models to formulate a model zoo. Next, we use the validation set to train the Unfoldi dML policy. Finally, we the test set to evaluate performance.
6

---

4.1.1 Experimental Setup
Model Zoo. We construct our model zoo by training two sets of binary classifiers for Stage-1 (Sepsis) and Stage-2 (Septic Shock) from a variety model architectures and input features. We select CPU-based models such as Logistic Regression, Decision Tree and Random Forest, and GPU-based models such as LSTM (Long Short-Term Memory) that thus far the state-of-art approach for early detection of Sepsis and Septic Shock (Fagerström et al., 2019; Liu et al., 2019). Specifically for LSTM, we vary the hidden size ranging from 100 to 400, the number of layers from 1 to 4. We also vary the input window of patient data from 1, 6, to 12 hours. Furthermore, at each stage train different model structures on different combinations of input data modalities: all models start with with basic demographic features and vital signs, and then progressively add more features such laboratory test results, other beside monitoring biomarkers, and medication/IV treatments.
Confidence Measures We consider four choices for measuring the confidence q of model prediction. Given p, the model's predictive probability of oulput Y being 1, we define
Max probability: max (1 p.P),
Entropy: (p log + (1 - p)
log(1 - p)),
Entropy of expected:
on to (bb(a)) vba1)) + paao + ta),
Mutual Information:
Entropy
Entropy of expected.
Given the trained model zoo, we profile spatio-temporal costs for each of the models. The spatio-temporal cost is the total time spent in each hardware due to inference/forward. pass calls of the models (temporal cost) multiplied with the hardware's 's cost per unit time (spatial cost). This cost serves as a proxy for the real dollar cost as it is the basis for pricing models in cloud offerings such as AWS On-Demand, Lambda, or Spot.
Cost.
Baseline. To our knowledge, UnfoldML is the first system to provide 2-dimensional cascading predictions. A reasonable baseline for our method is the set of models that frame the multi-stage sequential task as a multi-class classification task by ignoring the happen- -eforee relationship between the stages. The goal of these baseline models becomes classifying patients into one of the three classes: Non-Septic, Septic, and Septic Shock. We take LSTM as our baseline since it is the state-of- art classifier for the task. We evaluate the performance of LSTM's across different spatio-temporal costs by changing the number of layers and hidden sizes in its architecture.
Multi-class LSTM: One unified multi-class model works end to end to predict the multi classes.
Since most of the existing work implements 1-D prediction on single-stage tasks, we reduce UnfoldML to a 1-D pipeline by unfolding it only vertically with no ICK transitions and compare the proposed soft-gating algorithm with prior works. We evaluate them on the two single-stage classification tasks of predicting sepsis and septic shock respectively.
IDK-cascade (Wang et al., 2017): Prior work of 1-D prediction cascade on IDK classes using a hard-gating like algorithm.
Single-Stage binary classifiers: CPU-based models such Logistic Regression, Decision Tree, Random Forest and TREWScore (Henry et al., 2015). a cox proportional hazards model that is well-known in early detection of septic shock; GPU-based models such LSTMs.
4.1.2 Evaluation and Results
End-to-End Classification Performance. To evaluate the prediction performance on our multi-stage task, treat UnfoldML multi-class classifier which predicts one of the same three classes as defined in Baseline. UnfoldML generates predictions at time-step for every patient, we take the maximum of the predictive probabilities along the prediction horizon for each patient and normalize them with a sum of 1. We compute the multi-class ROC AUC scores by averaging the pairwise ROC AUCs (known as one-vs-one) of each classes.
Better Cost-AUC tradeoff. We evaluate the model performance in trading off between the end-to-end AUC and spatio-temporal cost follows: vary the user-defined cost constraint in Eq. 2, and train Unf dMd repeatedly while searching the trade off in a 2D space. Each run contributes point in the scatter plot of Figure 2b. We compute the convex hull of the searched points in the set and
7

---

0.00
End-to-End AUC
0.86
0.86
End AUC
0.84
Malt-Closs (STM Reci
0.80
0.20
0.70
Usds dMu--1 gaant (oofigsiogg)
UnfoldML S (Soff Gollin Searching
Nulll LSTTM
0.76
Mult-ciess LSTM Searching
7
55
403
Matti Ciass
Spatio-Temporal Cost
SputioTeuupecll
145
(a) Couvex hull comparison for the set of points searched by different methods in the End-to-End AUC vs. Spatio-Tempont Temporal Cost tradeoff space.
(b) Search the trade off between End-to-Ean End AUC and Temporal cost by varying the cost con- straint (x-axis is in logarithmic scale).
Figure 2: Tradcoff space of the End-to-End AUC vs. Spatio-Tempore Cost.
UnfoldML-H
UnfoldML-S Multi-class LSTM
Approx. area the convex hull
17.58
31.17
24.20
Table 1: The approximated area of the convex hull searched by different methods Figure 2a.
demonstrate the resulting area of the hull in Figure 2a. We train our baseline method repeatedly by varying the LSTM architecture and plot the same way as in Figure 2a. In comparison, UnfoldML-S (soft-gating) searches significantly higher AUC regions than the multi-class LSTM baseline and performs more consistently than the hard-gating algorithm. To quantify the trade-off evaluation in Figure 2a, we calculate the area of the convex hull searched by each method (Table 1).
UnfoldML-S presents the highest score comparing to the baseline and Unf HMfo-MM--H. addition, we pick several critical points from the searched convex hull in Figure 2b and present them in Table 2. A recommended configuration for a system should effectively trade off the ppatio-temporal cost and end-to-end AUC, which is to select the model along the frontier line drawn through the points that reaches the top-left corner of the hull in Figure 2b. As shown in Table 2, the recommended configuration by our pipeline, namely Unf ol dML-S (Rec.), can achieve comparable end-to-end AUC score (88.%) with the most accurate baseline Multi-Class LSTM-A (88.8%), while operating at 19.6X spatio-temporal cost reduction. In comparison with UnfoldML-A, which naively uses the most accurate single-stage models each stage, UnfoldML-S outperforms the end-to-end AUC while providing 22. 7X reduction in ppatio-temporal cost. Alternatively, UnfoldML-S (Rec.) outperforms Multi-class LSTM (Rec.) at an AUC gain of 5.3% with only 1.3X more cost. The improvement in end-to-end AUC scores from UnfoldHi is result of the dynamic nature of inference pipeline, which selects the optimal pathway for each patient. The savings in spatio-temporal cost can be attributed to the usage of low cost models when they are confident enough, contrast, baselines (UnfoldML-A. Multi-Class LSTM-A) always use the most accurate model.
Single-Stage Performance. We report single-stage performance Sepsis and Septic Shock pre- dictions in Table For both predictions, UnfoldML achieves a significant reduction in costs with only marginal loss of AUC compared to the highest accuracy baseline LSTM-A 32.3x lower spatio-temporal cost with 1.7% lower AUC for Sepsis prediction and 26.8x Xx lower costs with 0.7% lower AUC for Septic Shock prediction. The baseline IDS-Casedde can achieve comparable AUC as UnfoldML-S, but requires 3.4x higher costs.
Better Early-hour Prediction. Table 2 also reports the average early prediction, showing UnfoldML- predicts septic shock earlier than other methods. predicts 2.1 hrs prior to the strongest baseline Mutli-class LSTM-A. It benefits from the multi-stage cascaded prediction in UnfoldML such that the system can exit early from the sepsis stage once it in the ICK, class and proceed to shock prediction before sepsis is truly diagnosed.
4.2
Task 2: Subcategory Classification
Our second experiment evaluates UnfoldML on a computer vision task for detecting a subcategory- of-interest. this subcategory-oof-mteees task, aim to accurately predict if image falls within

---

tasks based Sepsis-Septic Shock Prediction
end-to-end AUC((%)
Pemtio-e Cost Per
Early Prediction on Septie Shock (hr)
Inference Call (S)
Multi-class LSTM-C
75.1
5.1
12.6
Multi-class LSTM-A
88,9
269.0
24.0
Multi-class LSTM (Rec.)
83.5
6.0
15.4
UnfoldMI-CC
76.9
5,8
22.9
UnfoldMI-A
87.2
311.8
16.4
UnfoldMI. (Rec.)
84.4
34.2
14.2
UnfoldMI-S (Rec.)
13.7
26.1
Table 2: End-to-end performance comparison two-stage prediction ('-C' denotes the model choice of the cheapest and least accurate; '-A' denotes the model choice of the costliest and most accurate; 'Rec.'' denotes the recommended model choice with a good trade-off between cost and accuracy in Figure 2b; '-S' denotes soft-gating; 'HH' denotes hard-gating
Sepsis Prediction
Septic Shock Prediction
Single-Stage AUC (%)
Tempooleemprral Cost Per Inference Call (S)
Single-Stage AUC (%)
Spatio-Temporal Cost
Per Inference Call ($)
TREWScore (Henry et al., 2015)
83.0
2.3
Logistic Regression
74.5
2.3
87.0
2.3
Decision Tree
70.4
2.4
83.1
2.4
Random Forest
75.7
148
85.2
148
LSTM-C LSTM-A
88.6
5.1
90.3
5.1
92.8
268
96.9
268
IDK Cascade (Wang al., 2017)
91.7
28.1
95.2
34.0
UntoldML-S
91.1
8,3
96.2
10.0
Table 3: Single-stage performance
specific subcategory of a dataset with classes that form a hierarchical structure, while still ensuring our spatio-temporal costs remain within the user-defined budget. For our experiment using the CIFAR- 100 dataset (Krizhevsky et al., 2009), we define two separate subccateory--of-iteee on the real-world system applications of computer vision systems. We want to identify images of specific subcategory from one of two chosen categories: 'people' (baby, boy, girl, man, woman) and 'vehicles' (bicycle, bus, motorcycle, pickup truck, train). CIFAR-100 consists of 60,000 32x32 colour images in total 100 'fine' classes. There are 500 training images and 100 testing images per class. In addition, each image also is assigned 'coarse' label indicating which category (such people, vehicles, trees, etc) belongs to. In our experiment, we do a multi-class classification on the 'coarse- granularity' labels in Stage-1 1 detect the category, then do multi-class classification the original fine-granularity' CIFAR-100 labels in the second step to identify the subcategory-of-inttee
4.2.1 Experimental Setup
Our model zoo is made of WideResNets (WRN) using Sharpness- Aware Minimization (Foret et al., 2020) which is cited as having state-of-the-ar performance on the CIFAR-100 image recognition task. For each stage we train models with widths from [2, 4, 6, 8, 10) and depths from [16, 22, 24, 28), For Stage-1 we train K₁ number of multi-class classification models using the full dataset of CIFAR-100 with coarse- granularrtt labels as our target subcategories, resulting in a total of 20 classes. We train models for Stage-I I using the full 32x32 image and as well as random crops of size 24x24 or 16x16. For Stage-2 2 we train K₂ multi-class models to identify the next subcategory that happens-aue -fter the coarse subcategory from Stage-1. We train granularity label for our subcategory in Stage-1. 1. We train Stage-2 models using the full 32x32 image or random crop of 24x24. We profile each model by computing the number of Multiply- Accuuulate Operations (MACs) and use it as proxy measure of our spatio-temporal cost.
Stage-2 models only using data with the coarse-
4.2.2 Evaluation and Results
We compare UnfoldML against a multi-class WRN classifier basclinc which directly predicts labels at the fine-granularity. We train the baseline with width factor of 10 and depth of 28 using the full-sized 32x32 image. We evaluate both accuracy and MACs, and report the results in Table 4. The results show that UnfoldML is able to find the optimal gating policy for the pipeline and achieve a spatio-temporal cost-savings of 6.9X with 1% reduction in accuracy for 'people' subcategory classification; we achieve cost-savings of 4.7X with 0..4% reduction in accuracy for 'vehicle'

---

Subcategory 'Poople'
Subcategory 'Vehicles'
Accuracy (%)
Macs (10M)
Accuracy (%)
Maes (10M)
WRN-A
98.1
525.0
99.3
667.0
InfoldMI-S
97.1
77.0
98.9
142.3
Table 4: Performance subcategory classification
subcategory-oo-itere identification. Thus, UnfoldML achieves a savings in spatio-temporal cost for this task without compromising the overall accuracy. UnfoldML can accomplish this because when it is confident that queries do not belong to our subcategory-of-intere exit the query from the prediction pipeline. When UnfoldM is not confident (IDK), it upgrades queries to costlier and more accurate models. Only when UnfoldML can confidently predict the subcategory (1C in Stage-1 will it transition the query to Stage-2 and make the final prediction. This results in a significant cost-savings without compromising accuracy.
(ICKo)) we are able to early
A recent alternative method for trading-off accuracy and cost in image classification is CwCF (Classification with Costly Features) (Janisch et al., 2019), which traverses the trade-off space by varying cost parameter that limits the number of selected features and uses Deep Q-Learning (DQL) model to train a feature-cost-aware classifier. In addition, also includes a pre-trained cost-unaware nigh-Peeffomance Classifier (HPPC), which is called when decides to include all the features. In order to establish a fair baseline against UnfoldMI, we used the most accurate multi-class WRN for the HPC in CwCF, We vary their cost parameter lambda for searching the optimal trade-off point in the accuracy- -oot space, however, it only returns two extreme data points that gives either low accuracy of 1% with only < 10 features or high accuracy as the WRN-A shows in Table 4 with all the features. We observe similar failure in traversing the trade-off space for CIFAR-10 in Figure 4(c) of their paper, so we do not include this baseline for this task.
Conclusion
ML models, including for healthcare applications, are growing exponentially in size and cost of inference. This is problematic for resource constrained hospital environments. Costlier, monolithic models require expensive hardware, and can not fit on bed-side compute or even on site compute clusters with clinical implications. UnfoldMy dML proposes a set of mechanisms and policies to address the growing cost of monolithic classifiers for healthcare applications. It implements a query propagation mechanism that "unfolds" a monolithic multi-class classifier into sequence of single-class classifiers, each with its own cascade progressively more complex models. Each query is allowed to (1) confidently exit the pipeline with an ICKn. (2) transition horizontally to the next stage in the pipeline with an ICK1, 1, or (3) upgrade vertically to a more complex model within the horizontal stage with an IDK. This mechanism is coupled with set of policies, such soft-gating, that set the thresholds at which the state transitions occur for queries to the system. UnfoldML builds on fundamental insight that classes may have a "happens before" relationship between them, which can be leveraged to "unfold" a classifier, Icading to savings in spatio-temporal cost (how much resource used for how long) and clinically significant earlier onset prediction. Unf dML improves the frontier of optimality in the cost/accuracy tradeoff space and is able to nearly match (within 0.1%) SoTA AUC performance for septic shock prediction at the 1 of the baseline cost. UnfoldML and the application of the "happens before" insight generalizes to computer vision tasks with 5x cost savings gained for a mere 0.4% drop in accuracy. Limitations of this work include demonstrations on more than 2 stages tasks.
Acknowledgements
This material is based upon work supported by the National Science Foundation under Grant Numbers NSF IIS-2106961, CAREER IIS-2144333 and CCF-2029004. Wc would also like to acknowledge Dr. Kevin Maher and Dr. Alaa Aljiffry of Children's 's Healthcare of Atlanta for their medical insights and clinical guidance as well as the Neurips 22 Area Chairs and reviewers for their insightful feedback, which contributed to the improved quality of this paper. Disclaimer: Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
10

---

References
Anelia Angelova, Alex Krizhevsky, Vincent Vanhoucke, Abhijit Ogale, and Dave Ferguson. 2015. Real-time pedestrian detection with deep network cascades. (2015).
Ben Athiwaratkun and Andrew Gordon Wilson. 2018. On Modeling Hierarchical Data via Probabilistic Order Embeddings. In International Conference on Learning Representations. attps://openrevveu: http:/iwww..neeeeeeeeeeeeeeeeeee aett/fforummid-HHcXZZbb.a
Zhaowei Cai, Mohammad Saberian, and Nuno Vasconcelos. 2015. Learning complexity- aware cascades for deep pedestrian detection. In Proceedings of the IEEE International Conference on Computer Vision. 3361-3369.
Matthieu Clertant, Nataliya Sokolovska, Yann Chevaleyre, and Blaise Hanczar. 2019. Interpretable cascade classifiers with abstention. In The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2312-2320.
Anuvabh Dutt, Denis Pellerin, and Georges Quénot. 2017. Improving image classification using coarse and fine labels. In Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval. 438 442.
David Eigen, Marc' Aurelio Ranzato, and Ilya Sutskever. 2013. Learning factored representations in deep mixture of experts. arXiv preprint arXiv: 1312.4314 (2013).
Josef Fagerström, Magnus Bång, Daniel Wilhelms, and Michelle S Chew. 2019. LiSep LSTM: machine leamning algorithm for early detection of septic shock. Scientific reports 9, (2019), 1-8.
Chen Feng, Paul Griffin, Shravan Kethireddy, and Yajun Mci. 2020. A boosting inspired personalized threshold method for sepsis sereening. Journal of Applied Statisties (2020), 22.
Pierre Foret, Ariel Kleiner, Hosscin Mobahi, and Behnam Neyshabur. 2020. Sharpness-awar minimization for efficiently improving generalization. arXiv preprint 20X0000010 (2020).
Katharine E Henry, David N Hager, Peter J Pronovost, and Suchi Saria. 2015. A targeted real-time carly warning score (TREWScore) for septic shock. Science translational medicine 7, 299 (2015), 299ra122-299ra111
Geoffrey Hinton and Drew Van Camp. 1993, Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sixth annual conference Computational learning theory. 5-13.
Robert A Jacobs, Michael Jordan, Steven J Nowlan, and Geolfrey E Hinton. 1991. Adaptive mixtures of local experts. Neural computation (1991), 79-87.
Jaromír Janisch, Tomáš Pevný, and Viliam Lisý. 2019. Classification with costly features using deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.
MIMIC-III, freely accessible critical care database. Scientific data 3 (2016), 160035.
Fereshte Khani, Martin Rinard, and Percy Liang. 2016. Unanimous Prediction for 100% Precision with Application to Learning Semantic Mappings. Proceedings of the 54th Anmual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers). Association for Computational Linguistics, Berlin, Germany, 952-962. https://doi .aaiddi.oo///oo
org/ooo//// 10. 18653/v1//16---/ 1090
Alex Krizhevsky, Geoffrey Hinton, et al. 2009. Learning multiple layers of features from tiny images. (2009).
Jing Lei. 2014. Classification with confidence. Biometrika 101, 4 (2014), 755-769.
Jie Lei, Zhenyu Guo, and Yang Wang. 2017. Weakly supervised image classification with coarse and fine labels. In 2017 14th Conference on Computer and Robot Vision (CRV). IEEE, 240-247.
11

---

Mingkun Li and Ishwar K Sethi. 2006. Confidence-based classifier design. Pattern Recognition 39, 7 (2006), 1230-1240.
Ran Liu, Joseph L Greenstein, Stephen J Granite, James C Fackler, Melania M Bembea, Sridevi V Sarma, and Raimond L Winslow. 2019. Data-driven discovery of novel sepsis pre-shock state predicts impending septic shock in the ICU. Scientific reports 9, (2019), 1 9.
Xinhua Liu. 2012. Classification accuracy and cut point selection. Statistics in medicine 31, 23 (2012), 2676-2686.
Jiaqi Ma. Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018. Modeling task relationships in multi-task learning with multi-gate mixture-of-cxperts, In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 1930-1939.
David JC MacKay. 1992. A practical Bayesian framework for backpropagation networks. Neural computation 4, 3 (1992), 448-472.
David John Cameron Mackay. 1992. Bayesian methods for adaptive models. Ph. D. Dissertation. California Institute of Technology.
Andrey Malinin and Mark Gales. 2018. Predictive Uncertainty Estimation via Prior Networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS' 18). Curran Associates Inc., Red Hook, NY, USA, 7047-7058.
Rupert Miller and David Siegmund. 1982. Maximally selected chi square statistics. Biometrics (1982), 1011 1016.
Matan Orbach and Koby Crammer. 2012. European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 323-338.
Graph-based transduction with confidence. Joint
Neil Perkins and Enrique Schisterman. 2006. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American journal of epidemiology 163,7 7 (2006), 670-675.
Vikas Raykar, Balaji Krishnapuram, and Shipeng Yu. 2010. Designing efficient cascaded classifiers: tradeoff between accuracy and cost. In Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining. 853-860.
Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. 2019. Early prediction of sepsis from clinical data: the Chyspuine/Compuning in Cardiology Challenge 2019. In 2019 Computing in Cardiology (CinC). IEEE, Page-1.
Andrew Rhodes, Laura E Evans, Walced Alhazzani, Mitchell M Levy, Massimo Antonelli, Ricard Ferrer, Anand Kumar, Jonathan E Sevransky, Charles L Sprung, Mark E Nunnally, al. 2017. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock: 2016. Intensive care medicine 43, 3 (2017). 304 377.
Henry A Rowley, Shumeet Baluja, and Takeo Kanade. 1998. Neural network-based face detection. IEEE Transactions on pattern analysis and machine intelligence 20, 1 (1998), 23-38.
Christopher W Seymour, Vincent X Liu, Theodore Iwashyna, Frank M Brunkhorst, Thomas D Rea, André Scherag, Gordon Rubenfeld, Jeremy M Kahn, Manu Shankar- Hari, Mervyn Singer, et al. 2016. Assessment of clinical criteria for sepsis: for the Third International Consensus Definitions for Sepsis and Septic Shock (Sepsis-3). Jama 315, 8 (2016), 762-774.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously Large Neural Networks: The Sparsely-Gated Lixture-offExpertt Layer. https://openreviiet htwpeenrreieee.ee/dddd.eee
?
Mervyn Singer, Clifford Deutschman, Christopher Warren Seymour, Manu Shankar-Hari, Djillali Annane, Michael Bauer, Rinaldo Bellomo, Gordon R Bernard, Jennel Chiche, Craig M Coopersmith, et al. 2016. The third international consensus definitions for sepsis and septic shock (sepsis-3). Jama 315, 8 (2016), 801-810.
12

---

Reisa A Sperling, Paul S Aisen, Laurel A Beckett, David A Bennett, Suzanne Craft, Anne M Fagan, Takeshi Iwatsubo, Clifford R Jack Jr, Jeffrey Kaye, Thomas Montine, al. 2011. Toward defining the preclinical stages of Alzheimer's s disease: Recommendations from the National Institute on Seiig-Alzheimest s Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimer's 's & dementia 7, (2011), 280-292.
Kirill Trapeznikov and Venkatesh Saligrama. 2013. Supervised sequential classification under budget constraints. In Artificial intelligence and statistics. PMLR, 581-589.
Thomas P Trappenberg and Andrew D Back. 2000. A classification scheme for applications with ambiguous data. In Proceedings of the IEEE-INNY BEEE-NNS--A MINS-ENNS International Joint Conference on Neural Networks. IJCNN 2000. Neural Computing: New Challenges and Perspectives for the New Millennium, Vol. 6. IEEE, 296-301.
IIker Unal. 2017. Defining an optimal cut-point value ROC analysis: an alternative approach. Computational and mathematical methods in medicine 2017 (2017).
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579-2605 (2008), 85.
Shikhar Vashishth, Prateek Yadav, Manik Bhandari, and Partha Talukdar. 2019. Confidence-based graph convolutional networks for semi-supervised learning. The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 1792-1801.
Luke Vilnis and Andrew McCallum. 2015. Word Representations via Gaussian Embedding.. In ICLR.
attp://arxiv wrgg/bs///////// 11t.b./111....60 6623
Paul Viola and Michael Jones. 2004. Robust real-time face detection. International journal of computer vision 57, 2 (2004), 137-154.
Xin Wang, Yujia Luo, Daniel Crankshaw, Alexey Tumanov, Fisher Yu, and Joseph E Gonzalez. 2017. Idk cascades: Fast deep learning by learning not to overthink. arXiv preprint arXiv: / 770.000885 (2017).
Xin Wang. Jiawei Wu, Da Zhang, Yu Su, and William Yang Wang. 2019. Learning compose topic- awarc mixture of experts for zero-shot video captioning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 8965 8972.
Xin Wang, Fisher Yu, Lisa Dunlap, Yi-An Ma, Ruth Wang, Azalia Mirhoseini, Trevor Darrell, and Joseph E Gonzalez. 2020. Deep mixture of experts via shallow embedding. In Uncertainty in Artificial Intelligence. PMLR, 552-562.
Saining Xie, Ross B. Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated Residual Transformations for Deep Neural Networks. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR) (2017), 5987-5995.
Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun. 2018. RAIM: Recurrent Attentive and Intensive Model of Multimodal Patient Monitoring Data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2565-2573.
Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and S. Li. 2018. Occlusion-aware R-CNN: Detecting Pedestrians in Crowd. In ECCV.
Jingbo Zhu, Huizhen Wang, Eduard Hovy, and Matthew Ma. 2010. Confidence-basse stopping criteria for active learning for data annotation. ACM Transactions on Speech and Language Processing (TSLP) 6, 3 (2010), 1-24.
13

---

Checklist
[Yes] Section 5 provides limitations of 1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] The primary goal this work to offer a better trade-off AUC and cost trade-off space and significantly better early hour prediction of sepsis and shock
(b) Did you describe the limitations of your work?
proposed approaches.
(c) Did you discuss any potential negative socictal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including thcoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] Refer to Section
4
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? Yes]] The details of experimental setup are provided Appendix
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and scrcenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "eig0FCw9Pt1", "reviewer": "Reviewer_cok9", "summary": "The authors propose a cost-aware 2D prediction pipeline UnfoldML to solve multi-stage classification problem under the limited resources. In order to better find the optimal trade-off between model performance and cost, the author define a search space and three gate parameters (ICK_0, IDK, ICK_1) to navigate the trade-off. In addition, two learning algorithms (hard-gating traning algorithm and soft-gating training algorithm) are proposed to solve this optimization problem with search space. Finally, two experiments on two different real-world tasks show that the proposed method can achieve good performance as well as reducing the spatio-temporal cost compared to other baseline models.\n", "questions": "Does the cheaper algorithm DKD have the similar prediction with the expensive MoE algorithm on gate parameters?\n\nHow long will it take to search an optimal cost-AUC tradeoff in the defined search space?\n\nAccording to Table 2, The proposed method UnfoldML-S (Rec.) performs worse than the baseline Multu-class LSTM (Rec.), because both have similar AUC but the proposed method has more cost and worse early prediction hours.", "limitations": "For the search sapce, the vertical space is not well-designed. According to the experiments, at each stage, the authors only train models with different predefined setting of architectures and different combination of predefined feature sets.\n\nThere is no proof, experiment or data to validate that the proposed (hard-gating or soft-gating) algorithms find the optimal policy in the defined search space. It may be sub-optimal.\n\nThe authors only did experiments on two-stage classification task, which is simple on multi-stage classification.\n\n", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Compared to previous work, the authors defined a cost-performance search space and propose two algorithms to solve this optimization problem. The authors did lots of experiments with different models and evaluate the model performance using AUC curve instead of accuracy (can be biased). \n\n\nThe proposed hard-gating training algorithm is unstable. Its performance is expected to vary with the upper bound maxA.\n\nThe format of this paper is not good. Equations and figures are too small, and there are lots of typos. For example, line 240: Hard-Gating (3); line 306: Eq. ??; line 308: figure 3; line 314: Table 4.1.2. \n\nDidn't find Appendix section", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "R4tJI_BlfCa", "reviewer": "Reviewer_5CL8", "summary": "This paper proposed a cost-aware prediction pipeline that \"unfolds\" a single multi-class classifier into a series of single-stage classifiers, reducing its deployment cost. To be more specific, three classes of gate functions are learned to maximize the system-wise accuracy while minimizing the prediction cost. ", "questions": "The method focuses on the \"subcategory-of-interest\" task, would it be generalized to classification on all classes? \n", "limitations": "The author discussed the limitation but not negative societal impacts of the work. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n  - the spatial-temporal cost of inference is reduced by orders of magnitude by the method. \n  - the proposed method navigates a more complete cost-AUC trade-off space.\n\nWeakness\n  - The experiments focus mainly on two-stage tasks (such as cifar100, first on coarse classes then fine classes), it would be interesting to demonstrate the effectiveness on more than 2 stage tasks. \n  - Some explanations with a simple example (such as classifying CIFAR-100) to illustrate the method would be easier to understand. \n  - The method focuses on the \"subcategory-of-interest\" task, would it be generalized to classification on all classes? \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ja67EEiALK0", "reviewer": "Reviewer_f34b", "summary": "Sequential classification problems, where there is an existing hierarchy among the classes with one class transforming into the next set of classes, are commonly observed in many clinical settings of disease progression. Generally, the state-of-the-art methods in these settings treat all the stages as a multi-class problem without considering the hierarchy. This paper proposes a model unfolding strategy that trains a series of models of increasing complexity for each stage and progressively moves a data sample to next stages only when the first stage is detected. The proposed model leads to a reduction in both memory as well as inference time costs instead of running a single complex classifier. A Sepsis-Septic Shock prediction task and subcategory detection in CIFAR-100 dataset is used to evaluate the proposed method and baselines.", "questions": "Please refer to above section.", "limitations": "Not applicable", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this paper studies an important area of model decomposition. The key concern in the paper is experimental results. Although the model unfolding idea is interesting, the baselines used are not explained adequately. Further analysis based on the algorithm's robustness and ease of use need to be done.\n- An important component of the proposed methodology is the measured confidence criterion of a model. Accurate estimation of the uncertainty of a deep learning model has been an extensive area of research. This paper lacks the comparison with other uncertainty-quantification methods like MC-dropout, Bayesian approximations, etc., which are more state-of-the-art. Additionally, the algorithms depend on the reliability of the individual stage classifiers, and thus under-trained parts classifiers can lead to faulty signals for gating operations.\n- In addition, the baselines are not adequately explained, which makes it very hard to judge the comparisons properly. For example, the terms LSTM-C, LSTM-A, Multi-class LSTM (Rec.) has been used without proper explanations.\n- The writing in the paper lacks clarity which makes it difficult to understand. A few examples are listed below:\n1. Line 156, the process of calculating the model's confidence is absent.\n2. The optimization of Objective II (Sec. 3.2.3) is not explained properly. The paper is not self-contained, with most of the discussion deferred to the Appendix.\n3. Experimental settings are also not explained in the paper (Sec. 4.1). What are the features?\n4. The Baseline section (line 280) is poorly written, with the baselines not matching the terms being used in the tables and figures.\n5. Table 4.1.2 (line 314) is absent in the paper.\n6. Several other typos need to be fixed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FB-2lcVwYw1", "reviewer": "Reviewer_qEBg", "summary": "This paper aims to unfold a monolithic multi-class classifier to a series of single-stage classifiers for reducing deployment cost. The proposed method is a dynamic 2D projection pipeline which makes ICK predictions on sequential multi-stage classification tasks. The topic is interesting and this manuscript shows promising results on two real-world tasks.", "questions": "More detailed comments are as follows:\n- What exactly \"spatio-temporal cost\" means?\n- Would a lower-cost model interference must result in accuracy degradation?\n- Two training algorithms (i.e., hard-gating and soft-gating), what is the relationship beween them? What is the method novelty here?\n- Comparison methods in Tables are relatively out-of-date. Are there any newer methods for comparison on this reserch topic?\n- In Sec. 4.2, it is claimed \"our second experiment is a computer vision task demonstrating the generalizability of UnfoldML\". The claimed generalizability is in terms of what? How is it validated?\n\nMinor comments:\n- Typo in Line 306, missing Eq. number.\n- Tables should add standard deviations of the results.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The topic of unfolding a multi-class classifier to a series of single-stage calssifiers is interesting\n- Experiment on real-world dataset with a relatively large number of samples\n\nWeakness:\n- Methodological novelty is unclear\n- Lack of experimental comparison with recent methods\n- There is no theoretical analysis of the proposed method\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept