Title: SkipPredict: When to Invest in Predictions for Scheduling
Venue: NeurIPS 2024
Accepted: yes
Keywords: Algorithms with predictions; scheduling
Abstract: Expanding on recent work on scheduling with predicted job sizes, we consider the effect of the cost of predictions in queueing systems, removing the assumption in prior research that predictions are external to the system’s resources and/or cost-free. Additionally, we introduce a novel approach to utilizing predictions, SkipPredict, designed to address their inherent cost. Rather than uniformly applying predictions to all jobs, we propose a tailored approach that categorizes jobs to improve the effectiveness of prediction on performance. To achieve this, we employ one-bit “cheap predictions” to classify jobs as either short or long. SkipPredict prioritizes predicted short jobs over long jobs, and for the long jobs, SkipPredict applies a second round of more detailed “expensive predictions” to approximate Shortest Remaining Processing Time for these jobs. Importantly, our analyses take into account the cost of prediction. We derive closed-form formulas that calculate the mean response time of jobs with size predictions accounting for the prediction cost. We examine the effect of this cost for two distinct models in real-world and synthetic datasets. In the external cost model, predictions are generated by external method without impacting job service times but incur a cost. In the server time cost model, predictions themselves require server processing time and are scheduled on the same server as the jobs.

SkipPredict : When to Invest in Predictions for Scheduling

Abstract

Expanding on recent work on scheduling with predicted job sizes, we consider the effect of the cost of predictions in queueing systems, removing the assumption in prior research that predictions are external to the system's resources and/or cost-free. Additionally, we introduce a novel approach to utilizing predictions, SkipPredict , designed to address their inherent cost. Rather than uniformly applying predictions to all jobs, we propose a tailored approach that categorizes jobs to improve the effectiveness of prediction on performance. To achieve this, we employ one-bit 'cheap predictions' to classify jobs as either short or long. SkipPredict prioritizes predicted short jobs over long jobs, and for the long jobs, SkipPredict applies a second round of more detailed 'expensive predictions' to approximate Shortest Remaining Processing Time for these jobs. Importantly, our analyses take into account the cost of prediction. We derive closed-form formulas that calculate the mean response time of jobs with size predictions accounting for the prediction cost. We examine the effect of this cost for two distinct models in real-world and synthetic datasets. In the external cost model, predictions are generated by external method without impacting job service times but incur a cost. In the server time cost model, predictions themselves require server processing time and are scheduled on the same server as the jobs.

1 Introduction

Machine learning research is advancing rapidly, reshaping even traditional algorithms and data structures. This intersection has led to the rise of 'algorithms with predictions', also called learningaugmented algorithms, where classical algorithms are optimized by incorporating advice or predictions from machine learning models (or other sources). These learning-augmented algorithms have demonstrated their effectiveness across a range of areas, as shown in the collection of papers (git, [n. d.]) on the subject and as discussed in the surveys (Mitzenmacher and Vassilvitskii, 2020, 2022).

Queueing systems are an example where the learning-augmented algorithm paradigm has been applied for scheduling. Several studies have examined queues with predicted service times rather than exact service times, generally with the goal of minimizing the average time a job spends in the system (Dell'Amico et al., 2015; Dell'Amico, 2019; Mitzenmacher, 2019, 2021; Wierman and Nuyens, 2008), and additionally some recent works also consider scheduling jobs with deadlines (Salman et al., 2023b,a). Minimizing response time is crucial in data centers for reduced latency, in cloud computing for an enhanced user experience and optimized resource utilization, and in realtime telecommunication systems for maintaining reliability and performance through quick task processing.

However, existing works do not adequately model the resources required to obtain such predictions. They often assume that predictions are provided 'for free' when a job arrives, which may not be a

realistic assumption in the practical evaluation of a system. Incorporating the cost of predictions is essential, as it could be argued that the resources devoted to calculating predictions might be more effectively used to directly process the jobs themselves. This perspective challenges the potential efficiency of integrating predictions into real-world queueing systems. As a result, the following questions arise:

When does the use of predictions, including their computation, justify their costs? Should all jobs be treated uniformly by computing predictions for each one?

As a simple example, let us consider a model where predictions of the service time arrive with the job and do not affect the arrival or service times, but do introduce a fixed cost c per job, so the total cost per job is the sum of the mean response time and fixed prediction cost. When we look at Shortest Predicted Remaining Processing Time (SPRPT) policy (Mitzenmacher, 2019), which simply always runs the job with the smallest predicted remaining time, the improvement in the cost over FCFS naturally varies with the prediction cost.

In this paper, we focus on the cost of predictions in settings where two stages of predictions are available. We consider the setting of an M/G/1 queueing system: jobs arrive to a single-server queue, according to a Poisson arrival process with general i.i.d service times. We examine two distinct cost models. In the first model, referred to as the external cost model, predictions are provided by an external server, and they do not affect job service time, but we do factor in a fixed 1 cost for these predictions. The expected overall cost per job in this model is the sum of the job's expected response time within the system and the cost associated with the time for prediction. In the second model, referred to as the server time cost model, predictions themselves require a fixed time from the same server as the jobs to produce, and hence a scheduling policy involves also scheduling the predictions. In this model, the expected overall cost per job is determined by the expected response time. As this model integrates the prediction process within the primary job processing system, it offers a different perspective on the cost implications of predictions as compared to the external model. (In particular, for heavily loaded systems, adding time for jobs to obtain predictions could lead to an overloaded, unstable system.)

As adding a single prediction to an M/G/1 model is relatively straightforward, we consider systems where we have two stages of prediction. In the first stage, we may simply predict whether a job is short or long. This type of one-bit prediction (studied in (Mitzenmacher, 2021)) is very natural for machine learning, and in practice may be much simpler and faster to implement. We therefore call these 'cheap predictions.' In a possible second stage, we predict the service time for a job, which we refer to as 'expensive predictions,' as in practice we expect them to be substantially more costly. (While we focus on these types of two stages, one could alternatively consider variations where the two stages could yield the same type of prediction; e.g. service time, with the cheap prediction being less accurate but consuming fewer resources.) We introduce a scheduling policy, called SkipPredict ( Skip or Predict ), which first categorizes jobs into short and long jobs with the first prediction, prioritizes short jobs over long ones, and restricts additional service time predictions exclusively to long jobs. We analyze the effect of the cost of prediction by considering SkipPredict with the previously described external cost model and server cost model. Our approach could be applied to applications where the cost is critical, such as data centers, cloud computing systems, and real-time telecommunication systems; in all these applications, we can reduce the expected time in system, optimizing resources and the user experience while accounting for the overall costs.

We compare SkipPredict with three distinct previously studied policies, re-analyzing them with prediction costs in the two proposed models. First, we consider First Come First Served (FCFS), which does not require predictions (and hence incurs no cost from predictions). Second, 1bit (Mitzenmacher, 2021), a policy using only cheap predictions, separates jobs into predicted short and long jobs, and applies FCFS for each category, thereby eliminating the need for a second stage of prediction. Third, Shortest Predicted Remaining Processing Time (SPRPT) performs expensive predictions for all jobs, and no cheap predictions. We find that service time predictions are particularly effective in high-load systems. Our analysis shows that SkipPredict potentially outperforms the other policies (FCFS, 1bit, SPRPT) in both cost models using real-world and synthetic datasets, especially when there is a cost gap between cheap and expensive predictions. Additionaly, we present another alternative algorithm called DelayPredict which avoids cheap predictions by running jobs for a fixed period before executing an expensive prediction. DelayPredict initially schedules all jobs FCFS but

limits them to a time L . Jobs exceeding the limit L are preempted, and given lower priority, and then they are scheduled by SPRPT. We find SkipPredict can also perform better than DelayPredict when there is a cost gap between these predictions.

While we focus on scheduling, we believe the approach of using multiple layers of predictions may be useful for other algorithms with predictions problems.

2 Related Work

We make extensive use of the SOAP framework (Schedule Ordered by Age-based Priority) (Scully and Harchol-Balter, 2018), which was recently developed to analyze age-based scheduling policies. Weuse this framework to derive mean response time formulas. We provide a brief background to the framework in Section B.1. For scheduling with predictions, Mitzenmacher (Mitzenmacher, 2019) demonstrated that the analyses of various single-queue job scheduling approaches can be generalized to the context of predicted service times instead of true values. Later, Mitzenmacher (Mitzenmacher, 2021) considered scheduling algorithms with a single bit of predictive advice for whether a job is 'long' or 'short', based on if its size is above or below a certain threshold. This work shows that even small amounts of possibly inaccurate information can yield significant performance improvements. Scully, Grosof, and Mitzenmacher (Scully et al., 2022) design a scheduling approach for M/G/1 queues that has mean response time within a constant factor of shortest remaining processing time (SRPT) when estimates have multiplicatively bounded error, improving qualitatively over simply using predicted remaining service times. Azar, Leonardi, and Touitou study similar problems in the online setting, without stochastic assumptions, and consider the approximation ratio versus SRPT (Azar et al., 2021, 2022).

Our work has a similar flavor to various '2-stage' problems, such as 2-stage stochastic programming and 2-stage stochastic optimization (e.g., (Grass and Fischer, 2016; Kolbin, 1977; Shmoys and Swamy, 2004; Swamy and Shmoys, 2006)). Here, somewhat differently, our two stages are both predictions of service time at different levels of specificity.

Two recent works have explored incorporating costly predictions into online algorithms. (Benomar and Perchet, 2024) examines how online algorithms can optimally use a limited budget to query for advice. (Drygala et al., 2023) studies the trade-off between the expense of acquiring predictions and the operational benefits they offer. We note that these papers address traditional online problems, such as the ski rental problem. However, the queueing setting introduces unique complexities. Notably, much of the literature in this area focuses on budgeted settings, where the number of predictions is limited. In contrast, our work seeks to optimize overall costs by considering prediction and operational expenses.

3 Model

We consider M/G/1 queueing systems with arrival rate λ . The processing times for each arriving job are independent and drawn based on the cumulative distribution F ( x ) , with an associated density function f ( x ) .

Scheduling Algorithm: SkipPredict SkipPredict initially categorizes jobs based on a binary prediction of either short or long, which we refer to as a cheap prediction. Only jobs predicted as long are further scheduled for a detailed expensive prediction to get the predicted processing time. With SkipPredict , jobs that are predicted short have priority over all other jobs. Specifically, predicted short jobs are not preemptible and are scheduled based on First-Come, First-Served (FCFS). Jobs predicted to be long are preemptible and scheduled according to the Shortest Predicted Remaining Processing Time (SPRPT) with predicted size given by the expensive prediction.

We focus on a model where, given a modulated threshold parameter T , the cheap predictions are assumed to be independent over jobs, a job of true size x being predicted as short (less than T ) with probability p T ( x ) . Similarly, the expensive predictions are assumed to be independent over jobs, and they are given by a density function g ( x, y ) , where g ( x, y ) is the density corresponding to a job with actual size x and predicted size y . Hence ∫ ∞ y =0 g ( x, y ) dy = f ( x ) . We consider two different models, external cost, in which the predictions are provided by an external server, and server cost, in which the predictions are scheduled on the same server as the job.

Single-Queue, External Cost In this model, a job can be described by a triple ( x, b, r ) ; we refer to this as a job's type. Here x is the service time for the predictor, b is the output from a binary predictor that determines whether the job is short or long, and for any long job, r is the result of a service-time predictor that provides a real-number prediction of the service time. If a job is predicted to be short, we do not consider r , and so we may take r to be null. Again, we refer to b as the cheap prediction and r as the expensive prediction.

In this model, the predictions do not affect the service time of the job, and we treat the overall arrival process, still, as Poisson. Accordingly, SkipPredict can be viewed as a two-class priority system: Class 1 is for short jobs, managed by FCFS within the class. Class 2 is for long jobs, according to SPRPT using service time prediction. However, we do associate a cost with predictions which is the mean response time to get the prediction. All jobs obtain a cheap prediction at some constant fixed cost c 1 , and all long jobs obtain an expensive prediction at some fixed cost c 2 . Accordingly, we can model the total expected cost for predictions per job in equilibrium as C = c 1 + c 2 z , where z = ∫ f ( x )(1 -p T ( x )) dx is the expected fraction of jobs requiring the second prediction. In general, both z and c 1 will depend on our choice of first layer prediction function, and similarly c 2 will depend on the choice of second layer prediction function. Therefore, for some parameterized families of prediction functions, we may wish to optimize our choice of predictors. Specifically, letting T be the expected response time for a job in the system in equilibrium, we might typically score a choice of predictors by the expected overall cost per job, which we model as a function H ( C, T ) , such as the sum of the C and T .

Single-Queue, Server Time Cost The server time cost model refers to the setting where predictions are scheduled on the same server as the jobs, so there is a server time cost based on a defined policy. Jobs predicted as short are categorized as non-preemptible while in execution, thereby prioritizing their completion before predicting new jobs. However, jobs predicted as long are further scheduled for a detailed expensive prediction. Thus, cheap predictions outrank expensive predictions and long jobs. Similarly, expensive predictions are prioritized over predicted long jobs.

SkipPredict now can be viewed as a four-class priority preemptive system. Class 1 (highest priority) is designated for predicted short jobs, managed by FCFS within the class. Consequently, short jobs are non-preemptible, and are prioritized over predictions for new jobs. This priority is appropriate because even if new jobs are predicted to be short, they will run after the already running short jobs (from FCFS). Class 2 is for (entering) jobs awaiting cheap predictions, which are also handled using FCFS. Class 3 is for jobs awaiting expensive predictions, and also uses FCFS. Finally, Class 4 is reserved for predicted long jobs, served using SPRPT using the service time predictions the jobs obtained when they were Class 3. (Note all jobs enter in Class 2, and the either move to Class 1, or to Class 3 and then Class 4.)

Definition 1. Suppose E [ T ] PS ext , E [ T ] PL ext , E [ T ] PS srv and E [ T ] PL srv are the expected response time for predicted short job and predicted long job in the external cost model and the server cost model in equilibrium respectively. Then, the total cost in the external cost model is

while the total cost in the server cost model is

where z is the expected fraction of jobs requiring the second prediction and C is the expected cost for prediction per job.

4 SkipPredict Theorems

For SkipPredict with a given T in both models, the expected mean response times for a predicted short job ( E [ T ] PS <model> ) and predicted long job of true size x J and predicted size r ( E [ T ( x J , r )] PL <model> ) are given in Table 1. We provide proofs in Appendix C in Lemmas 1, 2, 3and 4. We get E [ T ] PL <model> using Lemma 5 and by plugging E [ T ] PS <model> and E [ T ] PL <model> into Definition 1, we get the total cost of SkipPredict in both models. c 1 is the cost of the cheap prediction, c 2 is the cost of the expensive prediction and ( r -a ) + = max ( r -a, 0) . The symbols used in the equations are later expressed and described in Table 2.

4.1 Proof intuition of SkipPredict

External Cost Model. Response time is defined as the sum of the waiting time in the queue and the residence time, which is the time from when the job is first processed until completion. For SkipPredict in the external cost model, a predicted short job has to wait behind only previous short jobs. This case is straightforward and follows the mean response time in the system for FCFS, which is given by Equation 23.15 in (Harchol-Balter, 2013). Since there is no preemption in this case, the residence time is simply the service time for the predicted short job, E [ S ′ <T ] . A predicted long job of true size x J and predicted size r has to wait behind previous short jobs, expressed by the term E [ S ′ <T 2 ] , and long jobs with a shorter remaining time, expressed by the terms E [ S ′ ≥ T,r 2 ] and a ( r ) as well as the final term involving the load. The waiting time is represented by the first term. Since we perform preemption for predicted long jobs, there is a preemption delay that causes a slowdown in the residence time due to the load from jobs predicted as short and jobs predicted as long but with a service time prediction of less than r .

Server Time Cost Model. For the server cost model, a predicted short job has to wait behind previous short jobs and previous cheap predictions. This addition of the cheap prediction, compared to the external cost model, is expressed by terms involving c 1 . In this case, there is no preemption, so the residence time is simply the service time for the predicted short job, E [ S ′ <T ] .

Apredicted long job of true size x J and predicted size r must wait behind previous short jobs along with their cheap predictions, expressed by the term E [ S ′′ <T ( c 1 )] . It must also wait behind cheap and expensive predictions of long jobs that arrive earlier but with a larger remaining time, expressed by ( c 1 + c 2 ) 2 · Q ( T, r ) , and long jobs with a shorter remaining time frame along with their cheap and expensive predictions, expressed by E [ S ′′ ≥ T,r 2 ( c 1 + c 2 )] + a ( r ) . The slowdown due to preemption is affected by the load from jobs predicted as short, jobs predicted as long but with a service time prediction less than r , and the load from all predictions.

4.2 SkipPredict Models Comparison

In the server cost model, we observe that the mean response times for both predicted short and long jobs are consistently higher than those in the external cost model, because predictions are scheduled on the same server as the jobs. When we set the costs to zero, so there is no cost to predictions, both models yield identical mean response times. This follows from the definitions of S ′′ ≥ T,r ( c 2 ) and S ′′ <T ( c 1 ) (Table 3) because with zero costs, these definitions match with those of S ′′ ≥ T,r and S ′′ <T . Additionally, setting the threshold T to zero in SkipPredict results in SPRPT (Shortest Predicted Remaining Processing Time) scheduling. With this threshold, there are no short jobs (i.e., E [ S ′ <T ] = 0 ), necessitating expensive predictions for all jobs. Thus, the mean response time for the predicted long jobs aligns with the mean response time of SPRPT (Mitzenmacher, 2019) (analyzed for both of the two models in Appendix D.1).

We emphasize that while we have derived equations for total costs for both models, comparing the practical implications of these total costs for the two models is challenging. First, resource allocation differs between the models: in the external cost model, predictions are scheduled on a server separate from the one handling the jobs, whereas in the server cost model, both predictions and jobs are scheduled on the same server. Incorporating a fixed cost into the mean response time for the external cost model does not translate directly to service time. This leads to potential differences in the interpretation of the costs of predictions between the two models.

Baselines. As baselines, we compare SkipPredict with three distinct policies in the two proposed models. These policies are 1) FCFS 2 , a non-size-based policy that does not require predictions; 2) SPRPT, which involves performing expensive predictions for all jobs; and 3) 1bit advice (Mitzenmacher, 2021), which uses only cheap predictions, separating jobs into predicted shorts and predicted longs, and using FCFS as a scheduling policy for each category. These policies, along with SkipPredict , can be placed on a spectrum based on their prediction costs. FCFS requires no predictions, while SPRPT requires expensive predictions for each job. The 1bit policy and SkipPredict are positioned in the middle of this spectrum, with the 1bit policy incurring lower prediction costs than SkipPredict . The question becomes, given prediction costs, what is the most cost-effective policy?

To compare all these policies, we analyze SPRPT and 1bit policies in the external cost model and the server cost model. These policies, initially introduced by Mitzenmacher (Mitzenmacher, 2019, 2021), were analyzed without considering the cost of predictions, so in Appendix D.1 we re-analyze them with prediction costs.

Without looking at the formulas, it is intuitive that under even moderate loads, when the cost of the expensive prediction is low (close to c 1 ), the total cost of SkipPredict would be greater than that for SPRPT. Also in this case SPRPT would outperform the 1bit approach, as size-based policies are generally more effective than non-size-based ones when predictions are reasonably accurate. However, when the expensive prediction cost c 2 is high, SkipPredict or 1bit would be better options than SPRPT. Note SkipPredict and 1bit have the same response times for jobs predicted to be short, as both schedule these jobs in the same way. Thus, the efficiency of SkipPredict over the 1bit approach depends on the value of using an SPRPT-strategy for the remaining jobs.

SkipPredict limitations. SkipPredict 's performance depends on the accuracy of its predictions, as poor prediction quality would diminish its effectiveness in practical applications. Our experiments (Section 5) demonstrate a direct correlation between prediction accuracy and overall performance. Regarding robustness, inaccurate cheap predictions could potentially lead to more long jobs delaying short jobs, highlighting the importance of achieving high prediction accuracy for cheap predictions.

4.3 What if the cheap predictions are not really cheap?

Another limitation of SkipPredict arises in situations where sufficiently low-cost predictions are not available or are not substantially less costly than expensive predictions. In such scenarios, SkipPredict may be less effective. However, on the positive side, our work could provide insights into the necessary cost thresholds for achieving performance gains. Here, we also suggest an alternative algorithm, DelayPredict , as a potential solution in these cases. Rather than apply predictions to all jobs, DelayPredict does not use cheap predictions, while still avoiding expensive predictions for short jobs. DelayPredict schedules jobs initially with FCFS, but limits each job to a given limit L

(a)

large

cost

gap,

c

1

=

0

.

5

, c

2

= 4

(b)

small

cost

gap,

c

1

=

3

.

5

, c

2

= 4

(c)

cost vs. threshold

14

14

10.0

9.5

12

12

9.0

Cost

10

Cost

10

8.5

8

8

8.0

7.5

6

7.0

6.5

0.5

0.6

0.7

入

8'0

0.9

0.5

0.6

0.7

入

0.8

0.9

口

1

3

7 10 1

4

5

6

7

SPRPT

SkipPredict

DelayPredict

of service time, at which point it is preempted, and treated as a long job . At that point, the job can go through expensive prediction and be scheduled based on SPRPT scheduling. A job that finishes before L units of service would, in this setting, be a short job that finishes without any prediction. We define z ′ = ∫ ∞ x = L f ( x ) dx as the expected fraction of jobs requiring the expensive prediction and use it in Definitions 1 to analyze DelayPredict cost in the external cost model and in the server cost model. We provide mean response times for DelayPredict with proofs in Appendix E.

DelayPredict vs. SkipPredict . The main difference between DelayPredict and SkipPredict is in the waiting time. In SkipPredict , a predicted short job sees only short jobs while a short job in DelayPredict sees all jobs in the queue ahead of it, limited to size L . Similarly, in DelayPredict a long job has to wait behind incoming jobs (including all other long jobs) for at lest time L . Thus, while DelayPredict saves the cheap predictions, its waiting time can be higher than SkipPredict . Figure 1 shows a setting where there is a cost gap between prediction costs, and SkipPredict outperforms DelayPredict . However, DelayPredict still performs better than SPRPT (with c 1 = 0 . 5 , c 2 = 4 ). When the costs are close, DelayPredict is better than both SkipPredict and SPRPT, as in this case SkipPredict is less effective. Since the waiting time of DelayPredict depends on L , we see in Figure 1(c) that as L increases, the cost of DelayPredict gets higher. (Note for comparison purposes the L for DelayPredict and the T for SkipPredict are chosen to be the same value.)

4.4 Generalization to Non-Fixed Costs

While we assume that the prediction costs as fixed, our approach naturally generalizes theoretically to random costs from a distribution, where that distribution may also depend on the service time of the job. Here we outline the necessary modifications in the analysis for this generalization.

We may consider prediction costs that are assumed to be independent over jobs. The cheap prediction cost is given by a density function k 1 ( x, c 1 ) , where k 1 ( x, c 1 ) is the density corresponding to a job with actual size x and cost of cheap prediction c 1 . Hence, ∫ ∞ c =0 k 1 ( x, c ) dc = f ( x ) . Similarly, the expensive prediction cost is given by a density function k 2 ( x, c 2 ) , where k 1 ( x, c 2 ) is the density corresponding to a job with actual size x and the cost of expensive prediction is c 2 .

In our analysis of the server cost model with fixed costs, for jobs with rank 4 in the first dimension, we used r -a to encode the second dimension of the rank rather than the predicted remaining service time, which we noted is technically r -( a -c 1 -c 2 ) . When c 1 and c 2 are fixed, doing so does not change the rankings of jobs, but for non-fixed costs, we would want to use the actual predicted remaining service time r -( a -c 1 -c 2 ) for the rank function.

5 Experimental Results

To gain more insight into when to invest in prediction and how SkipPredict compares to other policies, in this section we compare SkipPredict , SPRPT, 1bit and FCFS using simulation with realworld and synthetic traces in the setting of the single queue with Poisson arrivals. For real-world traces, we used three traces from Amvrosiadis et al. (Amvrosiadis et al., 2018): Twosigma, Google, and Trinity. Their system leverages machine learning to predict job runtimes in large clusters, utilizing features like user IDs, job names, and input sizes, Appendix F contains more details about the datasets and the predictor. For the synthetic traces, we considered two job service time distributions: exponentially distributed with mean 1 ( f ( x ) = e -x ) and the Weibull distribution with cumulative distribution F = 1 -e - √ 2 x (which also has mean 1). Also, for the synthetic traces, we use two prediction models where each of the two-stage predictors could be from a different model: 1) Exponential predictions (Mitzenmacher, 2019), where a prediction for a job with service time x is itself exponentially distributed with mean x . 2) Uniform predictions (Mitzenmacher, 2019), where a prediction for a job with service time x is uniformly distributed between (1 -α ) x and (1 + α ) x for a parameter α . For the synthetic datasets, in this section we present results for the exponentially distributed service time with uniform predictor. Appendix F contains results for the rest of the combinations.

To handle two-stage predictions, both for the real-world traces and for the synthetic ones, the cheap predictor here returns a single bit by comparing the predicted value with the threshold; this is not how an actual prediction would work, but it is just a test model for simulation. We note that, as part of verifying our equations and our system, we have checked simulation results for single queues against the equations using the 'perfect predictor,' as the integrals are simpler in this case.

Cost vs. arrival rate λ . Figures 2, 3(b), 3(d), 4(b), 4(d) show the cost using real-world and synthetic datasets. These Figures show that investing in service time predictions is more beneficial at higher arrival rates. SkipPredict yields the lowest cost above λ = 0 . 7 in all datasets. We should note, however, that under extremely high load in the server cost model, prediction-based scheduling (SPRPT, SkipPredict or 1bit) risks overflowing the system (since the average time per job with predictions is larger than 1), making FCFS a better option.

Cost vs. T . In Figures 2(g), 2(h), 3(a), 3(c) 4(a), 4(c), we compare the cost vs. T . As T increases, both SkipPredict and 1bit demonstrate reduced costs. However, past a certain T threshold, which depends on the dataset, arrival rate, and actual costs, we observe a rise in costs due to the decreasing number of jobs requiring expensive prediction. This leads to a reduced load for expensive predictions, making job size prediction for scheduling between predicted long jobs less effective. In the synthetic datasets, as T gets large, SkipPredict and 1bit become the same, as both serve predicted short jobs similarly, and for large T nearly all jobs are predicted short.

Accuracy/Cost tradeoff. To evaluate a setting where one has to choose accuracy levels by accuracy/cost tradeoff, we conducted experiments where we controlled the accuracy, and consequently, the cost, so higher accuracy corresponded to higher prediction costs. Again, the point here is to show a possible use case, where a user might decide from a set of possible available accuracies and costs to optimize the system. The measured total cost is presented as a heat map matrix in Figures 5(a), 5(b) using exponential service time and T = 1 . In this matrix, rows represent the probability of accurate prediction and associated costs for the cheaper model, columns represent the 1 -α values of the uniform predictor, and each cell displays the corresponding total cost for that setting. These figures indicate that reducing the accuracy of the cheap prediction is more crucial since the cheap model's prediction is performed for every job. For expensive predictions, the relationship between α values and the total cost is not strictly linear and the cost can be optimized based on expensive prediction costs and number of predicted long jobs.

SkipPredict benefit increases with larger cost gaps. We have found that SkipPredict is more costeffective than other policies when there is a difference between the costs of the two predictions, with a greater gap leading to higher improvement. In Figures 5(c) 5(d), we have exponential service times, T = 1 , c 1 is fixed to the default value, and we change c 2 by varying k where c 2 = kc 1 . For similar or very close costs ( k = 1 ), SkipPredict is less useful, and SPRPT may be a better option. However, as c 2 values increase, SkipPredict becomes more cost-effective. FCFS and 1bit are not affected as they do not require expensive predictions.

(a)

λ

, Twosigma, server

(b)

λ

, Twosigma, external

(c)

λ

, Google, server

(d)

λ

, Google, external

(e)

λ

, Trinity, server

(f)

λ

, Trinity, external

(g)

T

, Trinity, server

(h)

T

, Trinity, external

103

103

102

102

Cost

Cost

Cost

Cost

102

102

101

101

0.5

0.6

0.7

0.8

0.9

0.5

0.6

0.7

0.8

6°0

0.5

0.6

0.7

0.8

0.9

0.5

0.6

0.7

0.8

6'0

入

入

入

入

80

35.0

37.5

103

102

OL

32.5

60

30.0

Cost

Cost

Cost

OS

Cost

27.5

102

25.0

40

22.5

101

30

20.0

101

20

17.5

0.5

0.6

0.7

入

0.8

0.9

0.5

0.6

0.7

入

0.8

0.9

T

SPRPT

X

FCFS

1bit

SkipPredict

(a)

cost vs T - External

(b)

cost vs.

λ

- External

(c)

cost vs T - Server

(d)

cost vs.

λ

- Server

10

10

101

6

Cost

8

Cost

Cost

8

Cost

101

0.5

0.6

0.7

0.8

60

0.5

0.6

0.7

8'0

0.9

入

入

SPRPT

X

FCFS

1bit

SkipPredict

6 Conclusion

We have presented SkipPredict , the first prediction-based scheduling policy we are aware of that takes into account the cost of prediction. SkipPredict is designed for systems where two levels of prediction are available, good (and cheap) and better (but expensive) prediction. While here we have focused on having a binary prediction (short/long) and a prediction of the service time, our framework would also work for other settings. For example, both the cheap and expensive predictions could be for the service time, with the expensive prediction being a more refined, time-consuming variation of the cheaper process (that even takes the cheap prediction as an input). We considered the cost of prediction in scheduling in two models; the external cost model with externally generated predictions, and the server time cost model where predictions require server time and are scheduled alongside jobs. We derived the response time of SkipPredict and analyzed total cost formulae in the two cost models for SkipPredict . We similarly derived formulae in these models for previously proposed prediction policies where previous analyses ignored prediction costs, namely 1bit and SPRPT, as well as a new policy, DelayPredict . We have demonstrated, using both real-world and synthetic datasets, that SkipPredict potentially outperforms FCFS, 1bit, SPRPT, and DelayPredict in both

(a)

cost vs T - External

(b)

cost vs.

λ

- External

(c)

cost vs T - Server

(d)

cost vs.

λ

- Server

10.0

9.5

OT

9.0

101

101

Cost

8.5

Cost

Cost

8

Cost

8.0

7.5

7.0

6.5

0.5

0.6

0.7

0.8

6°0

0.5

0.6

0.7

0.8

6'0

入

入

SPRPT

X

FCFS

1bit

SkipPredict

(a)

External

(b)

Server

(c)

External

(d)

Server

(E0)50°0

15

0.05 (0.008)

15

12

14

14

Cheap:probability(C1)

593.77

278.46

200.82

500

10

10

13

9

9

0.5 (1.3)

9.5

8.5

8.6

12

(St0)S°0

127.89

77.30

106.12

400

300

Cost

8

Cost

11

(0)60

10

(E'0) 6°0

200

6

10

9.4

9.3

72.33

97.25

65.63

100

0.9 (5.0)

Expensive:1-α(C2)

0.4 (2.5)

0.1 (2.0)

0.9 (0.5)

Expensive:1 -α (c2)

0.4 (0.25)

0.1 (0.22)

2

E

k,C)=k·C1

4

5

6

7

Z

E

k,C)=k·C1

4

5

6

SPRPT

X

FCFS

1bit

SkipPredict

cost models, especially when there is a significant cost gap between cheap and expensive predictions. While we focused on analyzing SkipPredict for cases with prediction costs, there are many potential variations to explore. These include using separate servers for prediction, offering more than two priority classes, selectively predicting only some jobs with probability to reduce costs while prioritizing between predicted short and long jobs, as well as employing load-based predictions to make predictions only when the queue length exceeds a certain threshold.

While scheduling itself is a foundational problem that provides sufficient motivation for our work, we believe the issue of accounting for the costs of predictions in the 'algorithms with predictions' framework (and arguably other similar problems) is understudied. Our idea of choosing one or more predictions, and optimizing the overall cost including that choice, is an approach that we believe will be useful for other similar problems.

Impact Statement. This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.

Acknowledgments

Rana Shahout was supported in part by Schmidt Futures Initiative and Zuckerman Institute. Michael Mitzenmacher was supported in part by NSF grants CCF-2101140, CNS-2107078, and DMS2023528.

References

[n. d.]. Algorithms with Predictions Paper List. https://algorithms-with-predictions. github.io .

George Amvrosiadis, Jun Woo Park, Gregory R Ganger, Garth A Gibson, Elisabeth Baseman, and Nathan DeBardeleben. 2018. On the diversity of cluster workloads and its impact on research results. In 2018 USENIX Annual Technical Conference (USENIX ATC 18) . 533-546.

Yossi Azar, Stefano Leonardi, and Noam Touitou. 2021. Flow time scheduling with uncertain processing time. In STOC . 1070-1080. https://doi.org/10.1145/3406325.3451023

Yossi Azar, Stefano Leonardi, and Noam Touitou. 2022. Distortion-Oblivious Algorithms for Minimizing Flow Time. In ACM-SIAM . 252-274. https://doi.org/10.1137/1. 9781611977073.13

Ziyad Benomar and Vianney Perchet. 2024. Advice querying under budget constraint for online algorithms. Advances in Neural Information Processing Systems 36 (2024).

Matteo Dell'Amico. 2019. Scheduling with inexact job sizes: The merits of shortest processing time first. arXiv preprint arXiv:1907.04824 (2019).

Matteo Dell'Amico, Damiano Carra, and Pietro Michiardi. 2015. PSBS: Practical size-based scheduling. IEEE Trans. Comput. 65, 7 (2015), 2199-2212.

Marina Drygala, Sai Ganesh Nagarajan, and Ola Svensson. 2023. Online algorithms with costly predictions. In International Conference on Artificial Intelligence and Statistics . PMLR, 80788101.

Emilia Grass and Kathrin Fischer. 2016. Two-stage stochastic programming in disaster management: A literature survey. Surveys in Operations Research and Management Science 21, 2 (2016), 85100.

Mor Harchol-Balter. 2013. Performance modeling and design of computer systems: queueing theory in action . Cambridge University Press.

Benjamin Hindman, Andy Konwinski, Matei Zaharia, Ali Ghodsi, Anthony D Joseph, Randy Katz, Scott Shenker, and Ion Stoica. 2011. Mesos: A platform for { Fine-Grained } resource sharing in the data center. In 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI 11) .

Viacheslav Viktorovich Kolbin. 1977. Stochastic programming . Number 14. Springer Science & Business Media.

Michael Mitzenmacher. 2019. Scheduling with predictions and the price of misprediction. arXiv preprint arXiv:1902.00732 (2019).

Michael Mitzenmacher. 2021. Queues with small advice. In SIAM Conference on Applied and Computational Discrete Algorithms (ACDA21) . SIAM, 1-12.

Michael Mitzenmacher and Sergei Vassilvitskii. 2020. Algorithms with Predictions. In Beyond the Worst-Case Analysis of Algorithms , Tim Roughgarden (Ed.). Cambridge University Press, 646662. https://doi.org/10.1017/9781108637435.037

Michael Mitzenmacher and Sergei Vassilvitskii. 2022. Algorithms with predictions. Commun. ACM 65, 7 (2022), 33-35. https://doi.org/10.1145/3528087

Salman Salloum, Ruslan Dautov, Xiaojun Chen, Patrick Xiaogang Peng, and Joshua Zhexue Huang. 2016. Big data analytics on Apache Spark. International Journal of Data Science and Analytics 1 (2016), 145-164.

Shaik Mohammed Salman, Van-Lan Dao, Alessandro Vittorio Papadopoulos, Saad Mubeen, and Thomas Nolte. 2023a. Scheduling firm real-time applications on the edge with single-bit execution time prediction. In ISORC . 207-213.

Shaik Mohammed Salman, Alessandro Vittorio Papadopoulos, Saad Mubeen, and Thomas Nolte. 2023b. Evaluating Dispatching and Scheduling Strategies for Firm Real-Time Jobs in Edge Computing. In IECON . 1-6.

Ziv Scully, Isaac Grosof, and Michael Mitzenmacher. 2022. Uniform Bounds for Scheduling with Job Size Estimates. In ITCS . 114:1-114:30. https://doi.org/10.4230/LIPIcs.ITCS. 2022.114

Ziv Scully and Mor Harchol-Balter. 2018. SOAP bubbles: Robust scheduling under adversarial noise. In 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton) . 144-154.

David B Shmoys and Chaitanya Swamy. 2004. Stochastic optimization is (almost) as easy as deterministic optimization. In 45th Annual IEEE Symposium on Foundations of Computer Science . IEEE, 228-237.

Chaitanya Swamy and David B Shmoys. 2006. Approximation algorithms for 2-stage stochastic optimization problems. ACM SIGACT News 37, 1 (2006), 33-46.

Alexey Tumanov, Angela Jiang, Jun Woo Park, Michael A Kozuch, and Gregory R Ganger. 2016. CMU-PDL-16-104 September 2016. (2016).

Abhishek Verma, Luis Pedrosa, Madhukar Korupolu, David Oppenheimer, Eric Tune, and John Wilkes. 2015. Large-scale cluster management at Google with Borg. In Proceedings of the tenth european conference on computer systems . 1-17.

Adam Wierman and Misja Nuyens. 2008. Scheduling despite inexact job-size information. In Proceedings of the 2008 ACM SIGMETRICS international conference on Measurement and modeling of computer systems . 25-36.

Appendix

Table of Contents

A Supplementary Figures

102

FCFS

SPRPT(C=0)

C

SPRPT (C=0.5)

Time

+

SPRPT (C=1)

SPRPT (C=2)

SPRPT(C=8)

：Response

101

Cost=

0.5

0.6

0.7

0.8

0.9

1.0

入

1-bit prediction:short

long

long

short

SkiporPredict:

skip

predict

predict

skip

short

short

long

short

X2,X，X3，X0

Xo

X1

X2

X3

B Formulas via SOAP Analysis

We employ the SOAP framework (Scully and Harchol-Balter, 2018), a (relatively) recently developed analysis method, to obtain precise formulas for mean response time 3 of SkipPredict in both the external cost model and in the server cost mode. While we could analyze the external cost model without SOAP as a two-class system, we choose to use SOAP for a consistent analysis.

B.1 SOAP Background

The SOAP framework can be used to analyze scheduling policies for M/G/1 queues that can be expressed in terms of rank functions. Recent research by Scully and Harchol-Balter (Scully and Harchol-Balter, 2018) has classified many scheduling policies as SOAP policies. These policies determine job scheduling through a rank, always serving the job with the lowest rank. (In cases where multiple jobs share the lowest rank, the tie is resolved using First Come First Served.) The rank function determines the rank of each job, and it can depend on certain static characteristics of the job, often referred to as the job's type or descriptor. For example, the descriptor could represent the job's class (if the model has different classes), and other static attributes, such as its size (service time). The rank can also depend on the amount of time the job has been served, often referred to as the job's age. A key assumption underlying SOAP policies is that a job's priority depends only on its own characteristics and its age, an aspect that aligns with our model and scheduling algorithm SkipPredict . We refer the interested reader to (Scully and Harchol-Balter, 2018) for more details.

SOAP analysis uses the tagged-job technique. That is, we consider a tagged job, denoted by J , of size x J and with descriptor d J . We use a J to denote the amount of time J has received service. The mean response time of J is given by the sum of its waiting time (the time from when it enters to when it is first served) and the residence time (time from first service to completion). To calculate the waiting time, SOAP considers the delays caused by other jobs, including 'old' jobs that arrived before J and 'new' jobs that arrived after J . A key concept in SOAP analysis is the worst future rank of a job, as ranks may change non-monotonically. The worst future rank of a job with descriptor d J and age a J is denoted by rank worst d J ( a J ) . When a J = 0 , the rank function is denoted by r worst = rank worst d J (0) . Ranks are written in ⟨ angle brackets ⟩ .

In the SOAP framework, waiting time is shown to be equivalent to the transformed busy period in an M/G/1 system with arrival rate λ and job size X new [ rank worst d J ( a )] 4 . The initial work of this period represents the delay caused by old jobs. To deal with the delay due to old jobs, SOAP introduced a transformed system where jobs are categorized based on their rank. Discarded old jobs, exceeding the rank threshold r worst , are excluded from the transformed system. Original old jobs, with a rank at or below r worst , are considered as arrivals with rate λ and a specific size distribution X old 0 [ r worst ] 5 . Recycled old jobs , currently at or below r worst but previously above this threshold, are treated as server vacations of length X old i [ r worst ] 6 for i ≥ 1 in the transformed system. As explained later, in SkipPredict jobs could only be recycled once, so we only have X old 1 [ r worst ] .

SOAP shows that, because Poisson arrivals see time averages, the stationary delay due to old jobs has the same distribution as queueing time in the transformed M/G/1/FCFS system. This system is characterized by 'sparse' server vacations, where ( original ) jobs arrive at rate λ and follow the size distribution X old 0 [ r worst ] .

Theorem 1 (Theorem 5.5 of (Scully and Harchol-Balter, 2018)) . Under any SOAP policy, the mean response time of jobs with descriptor d and size x J is:

B.2 Rank functions of SkipPredict

The relevant attributes to SkipPredict are the size, the 1-bit prediction, and the predicted service time. We can model the system using descriptor D = [size, predicted short/long, predicted time] = [ x, b, r ] . SkipPredict in the external cost model results in the following rank function:

which uses the first dimension to encode the class priority (short or long), and the second dimension to enforce the priority for each class (FCFS for short jobs, SPRPT for long jobs). In such nested rank function, the first dimension serves as the primary rank, with the priority ordering following a lexicographic ordering.

In the server cost model, SkipPredict results in the following rank function:

Note entering jobs have ranked 2 in the first dimension, placing them behind short jobs awaiting or receiving service. Since after predictions short jobs would simply be placed behind other short jobs, it makes sense to deprioritize the cheap predictions. On the other hand, we prioritize long predictions over long jobs to implement SPRPT.

Finally, for jobs with rank 4 in the first dimension, we use r -a as the secondary rank. Technically the predicted remaining service time is r -( a -c 1 -c 2 ) , since the job's age includes service for predictions of time c 1 + c 2 . As c 1 and c 2 are fixed, using r -a is equivalent to using r -( a -c 1 -c 2 ) for ranking, and we use r -a for convenience.

C SkipPredict Analysis

C.1 External cost model

Lemma 1. For SkipPredict in the external cost model, the expected mean response time for a predicted short job, E [ T ] PS ext is

Proof. While SOAP can be used to analyze the mean response time for predicted short jobs, this case is straightforward and follows the mean response time in the system for FCFS, which is given by Equation 23.15 in (Harchol-Balter, 2013).

Lemma 2. For SkipPredict in the external cost model, if we let a ( r ) = ∫ ∞ t = r ∫ ∞ x = t -r (1 -p T ( x )) g ( x, t )( x -( t -r )) 2 dxdt , the expected mean response time for a predicted long job of true size x J and predicted size r is

Where ρ ext r = λ ( E [ S ′ <T ] + E [ S ′ ≥ T,r ] ) is the load due to jobs of predicted short and jobs predicted long but their service time prediction less than r and ( r -a ) + = max ( r -a, 0) .

Proof. To analyze SkipPredict for a predicted long job in the external cost model using SOAP, we first find the worst future rank and then calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for predicted long job. As described in (1), the rank function for predicted long jobs is monotonic (here the job descriptor is d J = [ x J , 1 , r ] ), and every job's rank is strictly decreasing with age, thus

J 's worst future rank is its initial rank, here: rank worst d J ( a ) = ⟨ 2 , r -a ⟩ and r worst = rank worst d J (0) = ⟨ 2 , r ⟩ .

X new [ rank worst d J ( a )] : Suppose that a new job K of predicted size r K arrives when J has age a J . J 's delay due to K depends on whether K is predicted to be short or long. If K is predicted short then it will preempt J and be scheduled till completion because it has a higher class. Otherwise, if K has a predicted job size less than J 's predicted remaining process time ( r -a J ), K will always outrank J . Thus

X old 0 [ r worst ] : Whether another job I is original or recycled depends on its prediction as short or long, and in the case it is long, it also depends on its predicted size relative to J's prediction. If I is predicted short, then it remains original until its completion. Otherwise, if I is predicted long, I is original only if r I ≤ r , because then until its completion its rank never exceeds r .

X old i [ r worst ] : If another job I is predicted long and if r I > r , then I starts discarded but becomes recycled when r I -a = r . This starts at age a = r I -r and continues until completion, which will be x I -a I = x I -( r I -r ) . Thus, for i ≥ 2 , X old i,x I [ ⟨ 2 , r ⟩ ] = 0 . Let t = r I :

Applying Theorem 1 leads to the result.

C.2 Server time cost model

To analyze the server cost model, as explained, considering the rank function defined in (2), we first find the worst future rank of J , denoted as rank worst d J , as follows:

Lemma 3. For SkipPredict policy in the server time cost model, the expected mean response time for a predicted short job, E [ T ] PS srv is

where ρ srv PS = λ ( c 1 + E [ S ′ <T ] ) is the load due to jobs of predicted short and their cheap prediction cost.

Proof. To analyze SkipPredict for predicted short jobs, we calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for these predicted short jobs in the server cost model, where the job descriptor in this case is ( b, r ) = (1 , ∗ ) .

X new [ rank worst d J ( a )] : Let's consider a new job K arriving when J is at age a J (where a J ≤ min( r, x J ) ). The worst rank of J depends on whether J is predicted to be short or long. If J is predicted short, then J 's worst future rank is its current rank ⟨ 2 , -a J ⟩ . Given that K 's initial rank is ⟨ 2 , 0 ⟩ , at least equivalent to J 's worst future rank, the delay J experiences due to K is: X new x K [ ⟨ 2 , -a J ⟩ ] = 0

X old 0 [ r worst ] : Suppose that J witnesses an old job I of initial size x I . The duration for which I remains original depends on whether its prediction is short or long. If I is predicted short, it remains original until completion. Alternatively, if I is predicted long, it would remain original until the cheap prediction phase (lasting c 1 ), after which its rank shifts to ⟨ 3 , 0 ⟩ .

X old i [ r worst ] : There are no instances of recycled jobs because either I completes its service (if predicted short) or it is discarded completely (if predicted long), and thus never gets a rank lower than ⟨ 2 , 0 ⟩ . Thus, for i ≥ 1 ,

Applying Theorem 1 yields the result.

Lemma 4. For SkipPredict policy in the server time cost model, the expected mean response time for a predicted long job of true size x J and predicted size r is

Where Q ( T, r ) = ∫ ∞ y = r ∫ ∞ x =0 (1 -p T ( x )) · g ( x, y ) dxdy ( r -a ) + = max ( r -a, 0)

is the load due to jobs of predicted short and jobs predicted long but their service time prediction less than r along with the load of the jobs predictions.

Proof. Now we calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for predicted long jobs in the server cost most, here again the J's descriptor is ( ( b, r ) = (0 , r ) ).

X new [ rank worst d J ( a )] : J 's delay due to K also depends on K is predicted to be short or long.

Employing the joint distribution g ( x, y ) , and setting 1 ( r K < r -a J ) = ∫ r -a J y =0 g ( x K , y ) dy , we can derive J 's expected delay due to any random new job:

X old 0 [ r worst ] : Regardless of I 's prediction, an old job has higher priority than J , therefore J will be delayed for the duration of I 's service.

X old i [ r worst ] : If J is predicted long, job I may be recycled. This occurs when I is predicted as long and with an expensive prediction r I > r . I is initially considered as original and is served during the cheap and expensive prediction phases, then discarded. At age a I = r I -r , I is recycled and served till completion, which will be x I -a I = x I -( r I -r ) . For i ≥ 2 , X old i,x I [ ⟨ 2 , r ⟩ ] = 0 , let t = r I :

Applying Theorem 1 yields the result.

Lemma 5. The mean response time for predicted long jobs in the external cost model is

and in the server cost model is:

D Baselines Analysis

D.1 SPRPT Analysis

In SPRPT the job descriptors only include the job size and service time predictions, e.g. D = [size, predicted time] = [ x, r ] . Thus, the rank function in the external cost model is:

In the server cost model:

Lemma 6. For SPRPT in the external cost model, the expected mean response time for a job of true size x J and predicted size r is

where ρ ′ r = λ ∫ r y =0 ∫ ∞ x I =0 x I · g ( x I , y ) dx I dy .

Proof. SPRPT has a rank function rank ( r, a ) = r -a for job of size x and predicted size r . As this rank function is monotonic, J 's worst future rank is its initial prediction:

When a J = 0 , the rank function is denoted by r worst = rank worst d J ,x J (0) = r .

X new [ rank worst d J ,x J ( a )] computation: Suppose that a new job K of predicted size r K arrives when J has age a . If K has a predicted job size less than J 's predicted remaining process time ( r -a ), K will always outrank J . Thus

X old 0 [ r worst ] computation: Whether job I is an original or recycled job depends on its predicted size relative to J's predicted size. If r I ≤ r , then I is original until its completion because its rank never exceeds r .

X old i [ r worst ] computation: If r I > r , then I starts discarded but becomes recycled when r I -a = r . This means at age a = r I -r and served till completion, which will be x I -a I = x I -( r I -r ) , let t = r I :

Thus, we have

Applying Theorem 5.5 of SOAP (Scully and Harchol-Balter, 2018) yields that the mean response time of jobs with descriptor ( r ) and size x J is as follows. Let

Then

Let f p ( y ) = ∫ ∞ x =0 g ( x, y ) dx . Then the mean response time for a job with size x J , and the mean response time over all jobs are given by

Lemma 7. For SPRPT in the server time cost model, the expected mean response time for a job of true size x J and predicted size r is

Proof. X new [ rank worst d J ( a )] : J 's worst future rank is ⟨ 2 , r -a J ⟩ . In this case, J 's delay due to a new job K is c 2 plus x K is its predicted service time less than J 's remaining process time.

X old 0 [ r worst ] computation: In this model, old job I delays J at least c 2 . In addition, If r I ≤ r , then I is original until its completion because its rank never exceeds r .

X old i [ r worst ] computation: If r I > r , then I starts discarded but becomes recycled when r I -a = r . This means at age a = r I -r and served till completion, which will be x I -a I = x I -( r I -r ) , let t = r I :

Thus, we have

For i ≥ 2 ,

Applying Theorem 5.5 of SOAP (Scully and Harchol-Balter, 2018) yields that the mean response time of jobs with descriptor ( r ) and size x J is as follows. Let

Then

Let f p ( y ) = ∫ ∞ x =0 g ( x, y ) dx . The mean response time for a job with size x J and the mean time for a general job are give by

D.2 1bit Analysis

For the 1bit policy, we define the rank function of this approach and then analyze it using SOAP framework. Here the job descriptors only include the job sizes and binary predictions, e.g. D = [size, predicted short/long = [ x, b ] .

In the server cost model, this approach results in the following rank function:

The mean response time for predicted short jobs of this approach is similar to predicted short jobs of SkipPredict as nothing has changed.

Lemma 8. For the 1bit policy in the external cost model, the expected mean response time for a predicted long job of true size x J is

where ρ ext new = λ ∫ ∞ 0 p T ( x ) xf ( x ) dx , the load due to predicted short jobs.

Proof. To analyze 1-bit advice for predicted long job in the external cost model using SOAP, we first find the worst future rank and then calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for predicted long job. For predicted long jobs, the rank function is monotonic as described in (5), therefore J's worst future rank is its initial rank.

X new [ rank worst d J ( a )] : Suppose that a new job K arrives when J has age a J . J 's delay due to K depends on K is predicted to be short or long.

Only if K is predicted short then it will preempt J and be scheduled till completion because it has a higher class as long jobs are scheduled also according to FCFS so in case of K predicted long it will not preempt J

X old 0 [ r worst ] : Nowif old job I either predicted short or long is an original job then it remains original until its completion (regardless of if it is predicted long or short). Thus, for i ≥ 1 , X old i,x I [ ⟨ 2 , 0 ⟩ ] = 0 .

Applying Theorem 5.5 of SOAP (Scully and Harchol-Balter, 2018) yields the result:

where ρ ext new = λ (∫ ∞ 0 p T ( x ) xf ( x ) dx ) .

As a second way of thinking about this proof, it can be said that this is the original FCFS system with slowdowns caused by subsystem 1 (predicted short jobs) which is 1 1 -ρ ext new .

Lemma 9. For the 1bit policy in the server time cost model, the expected mean response time for a predicted long job of true size x J is

where ρ srv new = λ ( c 1 + ∫ ∞ 0 p T ( x ) xf ( x ) dx ) and ρ c 1 = λ (∫ ∞ 0 ( x + c 1 ) f ( x ) dx ) .

Proof. In this case, according to (6), J's worst future rank is

X new [ rank worst d J ( a )] : Let's say J has age a J when K arrives. The delay caused by K for J depends on whether it is predicted to be short or long for K . If K is predicted short then it will preempt J and be scheduled along with its cheap prediction till completion. Otherwise, if K is predicted long, it will delay J only for the cheap prediction.

X old 0 [ r worst ] : Each old job is scheduled for cheap prediction (which costs c 1 ), and an old job I , regardless of whether it is predicted long or short remains original until its completion. Thus, for i ≥ 1 , X old i,x I [ ⟨ 3 , 0 ⟩ ] = 0 .

Using Theorem 5.5 of SOAP (Scully and Harchol-Balter, 2018) yields the result:

E DelayPredict Analysis

Rank function of DelayPredict : We model the system using D = [size, predicted time] = [ x, r ] . Here we assume that the service time prediction r is greater than L , because jobs that require expensive prediction are longer than L . Since a job's age is L when it is preempted and obtains a prediction, in the external model, the predicted remaining time for long jobs is r -L -( a -L ) = r -a . For the service time model their age after being predicted starts at L + c 2 , so the predicted remaining time is r -L -( a -L -c 2 ) = r -a -c 2 . As c 2 is fixed among all jobs, instead of the predicted remaining time we can use r -a as the rank for convenience. Accordingly, DelayPredict in the external cost model has the following rank function:

In the server cost model, DelayPredict results in the following rank function:

Lemma 10. For DelayPredict in both the external cost model and the server time cost model, the expected mean response time for a short job is

where ρ L = λ ( ∫ L 0 xf ( x ) dx + ∫ ∞ L Lf ( x ) dx ) , the load due to jobs while limiting their sizes to L .

Proof. We first find the worst future rank and then calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for short jobs. As both the external cost model and the server cost model treat short jobs the same, and their worst future rank is the same, the analysis holds for both.

For short jobs, the rank function is monotonic, therefore J's worst future rank is its initial rank:

X new [ rank worst d J ( a )] : Since short jobs have higher priorities than long jobs, and they are scheduled FCFS, a new job does not preempt J . Hence X new x K [ ⟨ 1 , -a ⟩ ] = 0 .

X old 0 [ r worst ] : If an old job I is short, it remains original until it is completed. Otherwise, it remains original only for L times. Thus, for i ≥ 1 , X old i,x I [ ⟨ 1 , 0 ⟩ ] = 0 and

Applying Theorem 5.5 of SOAP (Scully and Harchol-Balter, 2018) yields the result

where ρ ext L,r = λ ( ∫ L x =0 xf ( x ) dx + ∫ r y =0 ∫ ∞ x = L x · g ( x, y ) dxdy + ∫ ∞ y = r ∫ ∞ x = L L · g ( x, y ) dxdy ) is the load due to short jobs, long jobs predicted to be less than r , and other long jobs with their size limited at L . Here ( r -a ) + = max ( r -a, 0) .

Proof. To analyze DelayPredict for a long job in the external cost model using SOAP, we first find the worst future rank and then calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for long job. As described in (7), the rank function for long jobs is monotonic, and every job's rank is strictly decreasing with age, thus J 's worst future rank is its initial rank, here: rank worst d J ( a ) = ⟨ 2 , r -a ⟩ and r worst = rank worst d J (0) = ⟨ 2 , r ⟩ .

X new [ rank worst d J ( a )] : Suppose that a new job K of predicted size r K arrives when J has age a J . J 's delay due to K depends on whether K is short or long. If K is short then it will preempt J , since it has a higher priority, and be scheduled till completion. If K is long and has a predicted job size less than J 's predicted remaining process time ( r -a J ), K will preempt J and proceed until completion. Otherwise, If K is long and has a predicted job size more than J 's predicted remaining process time, it will preempt J but will be scheduled only for L time.

Thus

X old 0 [ r worst ] : Whether another job I is original or recycled depends on whether it is short or long, and in the case it is long, it also depends on its predicted size relative to J's prediction. If I is short, then it remains original until its completion. Alternatively, if I is long, it is original until completion if r I ≤ r , otherwise, it is original until L .

X old i [ r worst ] : If another job I is long and if r I > r , then I starts discarded but becomes recycled when r I -a = r . This starts at age a = r I -r and continues until completion, which will be x I -L -a I = x I -L -( r I -r ) . Thus, for i ≥ 2 , X old i,x I [ ⟨ 2 , r ⟩ ] = 0 . Let t = r I :

Applying Theorem 1 leads to the result.

Lemma 12. For DelayPredict policy in the server cost model, the expected mean response time for a long job of true size x J and predicted size r is

where ρ srv L,r = λ ( ∫ L x =0 xf ( x ) dx + ∫ r y =0 ∫ ∞ x = L ( x + c 2 ) · g ( x, y ) dxdy + ∫ ∞ y = r ∫ ∞ x = L ( L + c 2 ) · g ( x, y ) dxdy ) is the load due to short jobs, predictions for long jobs, long jobs predicted to be less than r , and other long jobs with their size limited at L . Here ( r -a ) + = max ( r -a, 0) .

Proof. In this case, J 's worst future rank is rank worst d J ( a ) = ⟨ 3 , r -a ⟩ and r worst = rank worst d J (0) = ⟨ 3 , r ⟩ . Now we calculate X new [ rank worst d J ( a )] , X old 0 [ r worst ] and X old i [ r worst ] for long jobs in the server cost most.

X new [ rank worst d J ( a )] : J 's delay due to K also depends on K is short or long. If I is short, then it remains it is scheduled until its completion. Alternatively, if I is long its prediction is scheduled for c 2 . In addition, if r I ≤ r then it is scheduled until completion, otherwise it is scheduled until L .

X old 0 [ r worst ] : The analysis is similar to the new arrival job. Whether another job I is original or recycled depends on whether it is short or long, and in the case it is long, it also depends on its predicted size relative to J's prediction.

X old i [ r worst ] : As described before, if another job I is long and if r I > r , then I starts discarded but becomes recycled when r I -a = r . This starts at age a = r I -r and continues until completion, which will be x I -L -a I = x I -L -( r I -r ) . Thus, for i ≥ 2 , X old i,x I [ ⟨ 3 , r ⟩ ] = 0 . Let t = r I :

Applying Theorem 1 leads to the result.

F Simulation

F.1 Datasets descriptions

F.1.1 Real-world datatsets

For real-world traces, we use datasets from Amvrosiadis et al. (Amvrosiadis et al., 2018).

TwoSigma Dataset: The workload traces originate from two data centers of TwoSigma, a hedge fund company. The workload comprises data analytics jobs processing financial information. A portion of these jobs utilize Spark (Salloum et al., 2016), while the remaining jobs are handled by proprietary data analytics frameworks developed in-house. The dataset spans a period of 9 months, starting in January 2016, across the two data centers' operations, encompassing a total of 1313 identical compute nodes equipped with 31512 CPU cores and 328TB of RAM. The logs contains 265 , 029 jobs and were collected by an internally-developed job scheduler running on top of Mesos (Hindman et al., 2011).

Google Dataset: In 2012, Google released a trace of jobs that ran on one of their compute clusters. The workload encompasses both long-running services and batch jobs (Verma et al., 2015). Some of these were issued through the MapReduce framework, and executed on 12583 heterogeneous nodes in May 2011. The dataset we used contains 385 , 072 jobs. Google has not disclosed the exact hardware specifications of each cluster node in this trace.

Trinity Dataset: Trinity is the largest supercomputer at Los Alamos National Laboratory (LANL), dedicated to capability computing. Capability clusters are large-scale, high-demand resources introducing cutting-edge hardware technologies that aid in achieving significant computing milestones, such as higher-resolution climate and astrophysics models. Trinity's hardware was deployed in two pre-production phases before being fully operational, and the trace was collected before the completion of the second phase. Trinity consisted of 9408 identical compute nodes, totaling 301056 Intel Xeon E5-2698v3 2.3GHz cores and 1.2PB of RAM, making it the largest cluster with a publicly available trace by number of CPU cores. The dataset we used contains 18 , 872 jobs collected by the MOAB scheduler.

Their system leverages machine learning to predict job runtimes in large clusters, utilizing features like user IDs, job names, and input sizes. For each job, we have the actual runtime and predicted runtime in seconds, with runtimes summed across tasks for multi-task jobs. We only consider successfully completed jobs and normalize runtimes to a mean of 1, consistent with the synthetic traces.

Service time predictor for the real-world datasets

For these datasets, (Amvrosiadis et al., 2018) used JVuPredict, the job service time predictor of the JamaisVu scheduling system (Tumanov et al., 2016).

JVuPredict is a runtime (service time) prediction module that is part of the JamaisVu system. Its primary objective is to predict the runtime of a job when it is submitted, utilizing historical data regarding past job characteristics and runtimes. JVuPredict deviates from conventional approaches by attempting to identify jobs that repeat, even when successive runs are not explicitly declared as repeats. This approach proves to be more effective because only the relevant historical data pertaining to the newly submitted job is utilized to generate the runtime estimate. To achieve this, JVuPredict leverages various features of submitted jobs, such as user identifiers and job names, to construct multiple independent predictors. These predictors are then evaluated based on the accuracy they achieve on historical data, and the most accurate predictor is selected for generating future predictions. Once a prediction is made for a new job, that job's data is added to the historical dataset, and the accuracy scores of each predictor model are recalculated. Based on the updated accuracy scores, a new predictor is chosen, and the process is repeated for subsequent job submissions. In general, JVuPredict employs a dynamic approach, continuously refining its prediction models by incorporating new job data and selecting the most accurate predictor for each new job submission.

F.1.2 Synthetic datasets

For the synthetic traces, we considered two job service distributions; exponentially distributed with mean 1 ( f ( x ) = e -x ) and the Weibull distribution with cumulative distribution F = 1 -e - √ 2 x . The Weibull distribution is heavy-tailed, so that while the average service time remains 1, there are many more very long jobs than with the exponential distribution. For predictions, we used two prediction models; exponential and uniform predictions. Table 4 summarizes the quantities p T ( x ) and g ( x, y ) for our prediction models. Each of the two-stage predictors could be from a different model.

F.1.3 Simulation

In our simulation, each data point is obtained by simulating initially empty queues over 1 , 000 , 000 time units. The average response time is calculated for all jobs that terminate after time 100 , 000 . This process is repeated over 100 simulations, and the reported data point represents the average response time across these 100 simulations. We implemented the simulation in Python 3 . 7 . 6 . The evaluation was performed on an AMD EPYC 7313 16 -Core Processor running Ubuntu 20 . 04 . 6 LTS with Linux kernel 5 . 4 . 0 -172 -generic.

F.2 Simulation with real-world datasets

Here we present the cost vs. T simulations of the the rest of the real-world datasets, Twosigma and Google.

(a)

Twosigma, server

(b)

Twosigma, external

(c)

Google, server

(d)

Google, external

700

140

160

600

1500

120

140

500

1250

100

120

Cost

400

00E

Cost

1000

750

Cost

80

60

Cost

100

80

60

200

500

40

40

100

250

20

20

0

0

L

SPRPT

X

FCFS

1bit

SkipPredict

F.3 Simulation with synthetic datasets

Here we present some additional simulations of the the Weibull distribution using both predictors.

(a)

cost vs

c

2

- External

(b)

cost vs T - External

(c)

cost vs.

λ

- External

(d)

cost vs

c

2

- Server

(e)

cost vs T - Server

(f)

cost vs.

λ

- Server

25

25

Cost

20

Cost

20

Cost

15

15

101

10

10

3

k, C =k·C1

4

5

3

4

T

M

0.5

0.6

0.7

入

0.8

0.9

25

25

20

20

Cost

Cost

Cost

101

15

15

10

10

3

k, C2 =k·C1

4

5

6

T

4

5

6

0.5

0.6

0.7

入

8'0

0.9

SPRPT

X

FCFS

lbit

SkipPredict

(a)

cost vs

c

2

- External

(b)

cost vs T - External

(c)

cost vs.

λ

- External

(d)

cost vs

c

2

- Server

(e)

cost vs T - Server

(f)

cost vs.

λ

- Server

25

20

Cost

Cost

Cost

101

15

101

10

0.5

0.6

0.7

入

0.8

0.9

T

4

0.5

0.6

0.7

入

0.8

0.9

25

25

20

20

Cost

15

Cost

15

Cost

101

10

10

k, C2=k·C1

4

5

1

3

T

4

5

6

0.5

0.6

0.7

入

0.8

0.9

SPRPT

X

FCFS

lbit

SkipPredict

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims are justified by the Theorems in Section 4 and experiments in Section 5.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss our approach limitations in Lines 241-259.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We state our model in Section 3 (assumption) and provide proofs for all the theorems listed in Table 1 in Appendix C. Furthermore, we re-analyze the baseline policies we are comparing against, taking into account the prediction costs, in Appendix D.1.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the experimental datasets in Section 5 and Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide our code and scripts to reproduce all the results in the paper and the Appendix with readme file contains instructions as supplemental material. We didn't include the real-world datasets, following to these usage restrictions on the data we received from Two Sigma:

Please do not share this data outside of your immediate research team. Instead, please forward any data requests back to Two Sigma and we will process.'

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: In Section 5 and in Figure captions.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Each data point is obtained by simulating initially empty queues over 1,000,000 time units. The average response time is calculated for all jobs that terminate after time 100,000. This process is repeated over 100 simulations, and the reported data point represents the average response time across these 100 simulations. The paper does not report error bars because we are calculating the response time in a system in equilibrium (steady-state), and the reported data points represent the average behavior of the system in this steady-state condition. (We note that, because of the long simulations, results are concentrated and errors would be very small. We can add error bars if desired for simulated data in the next version, given more time.)

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Appenix F and Section 5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We include 'Impact Statement' in Section 6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In Section 5.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the effectiveness of machine-learned predictions for queuing systems, and falls into the area of learning-augmented algorithms / algorithms with predictions.\nIn particular it studies the M/G/1 queue with poisson arrival and i.i.d. service times with the objective to minimize the average response time.\nThis problem has previously studies with service time predictions or with single-bit predictions, indicating whether a job is 'short' or 'long' depending on a certain threshold. Moreover, previously it has been assumed that predictions are for free.\nIn contrast to these previous works, in this paper an algorithm needs to 'pay' for receiving predictions. This payment can either be some additional cost (external cost), or an additional load to the machine (server time cost) which delays the execution of other jobs. The latter is motivated by the scenario that the prediction itself first needs to be computed. Moreover, in this paper an algorithm has access to both, (cheaper) 1bit predictions and (more expensive) service time predictions. \n\nThe authors propose the algorithm 'SkipPredict' which first queries a 1bit prediction for every job that arrives, and in case that this returns 'long', also queries the service time prediction. 'Short' jobs are generally preferred and scheduled First-Come-First-Serve (FCFS). Furthermore, they study a second algorithm 'DelayPredict' which only uses service time predictions. In contrast to SkipPredict, it starts scheduling every job until it can be categorized as being 'long', and only then queries a service time prediction.\n\nThe authors compare both algorithms to the baselines FCFS, 1bit (only cheap predictions) and SRPPT (only expensive predictions). They find that in theory and in empirical experiments, the benefit of SkipPredict and DelayPredict depends on the how the threshold, the system load and the costs for both prediction types relate.\n\nSTRENGTHS:\n- The idea of using a two-stage prediction setup composed of a cheap but less expressive and an expensive but more expressive prediction is to the best of my knowledge new in the area of learning-augmented algorithms, and might have an impact on this field.\n- I really like the idea that the generation of a prediction requires processing volume itself. I think such a model could also be interesting for other scheduling problems, also in adversarial models.\n- The results show that the use of this two-stage prediction setup help to lower the overall cost compared to the baselines for certain instances.\n\nWEAKNESSES:\n- I think the main weakness of this work is that the results seem not to be very surprising. In particular, I think it is clear that the benefit of such a two-stage algorithm like SkipPredict heavily depends on how the costs relate. I am somehow missing some hard statements which summarize the findings more precisely.\n- Moreover, the two main components of SkipPredict, 1bit and SRPPT, have been previously studied and analyzed. Thus, to me it seems that the paper does not provide many interesting techniques or analyses.\n\nGiven the conceptually interesting models, I tend to rate this paper as borderline right now. However, I am not too familiar with queueing theory and, thus, cannot for sure assess the weight of the second weakness I mentioned.\n\nQUESTIONS:\nTypos:\n- Line 114 \"somewhat\"\n- Line 224 missing whitespace\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a scheduling problem that aims to minimize the expected response time, where jobs arrive online and the algorithm needs to decide the priority of jobs. This paper proposes a new algorithm called skip-prediction. Namely, the algorithm first uses a cheap prediction to partition the whole job set into two parts: long job set and short job set. For jobs in short job set, they have the highest priority and no prediction will be made. And, the algorithm ranks these short jobs by the first-come-first-serve rule. For jobs in long job sets, an expensive prediction will be made to predict the size of these long jobs. Then, the algorithm ranks these long jobs using the shortest remaining processing time first rule.\nThe paper considers two different models; in the first model, predictions consume some extra costs and in the second model, predictions consume server times. For each model, the total cost of the algorithm is defined as the total expected response time plus the prediction cost. The authors provide theoretical proof to compute the formula for the total cost of the proposed skip-prediction algorithm, and then run an experiment on some datasets.\n\nSTRENGTHS:\nThe general motivation of the paper is good. Namely, it considers the case where the prediction does not get for free. The algorithm needs to optimize the scheduling objective with the prediction cost together. On the positive side, I expect that it will have a positive impact on practice application. I also expect that one may be able to abstract some interesting theoretical models from this paper; so it’s likely to influence future work.\n\nWEAKNESSES:\nI have to say that I am usually working on the theoretical analysis of the algorithms (e.g., approximation or online algorithms). I am not in the right position to judge the quality and novelty of the experimental paper. To me, the presentation of the paper is not clear; the authors describe an algorithm in the model section instead of defining a problem. Besides this, the authors give the formula of the total cost of skip-prediction in Table 1, but it is not clear how good the quality of these formulas is from the theoretical perspective.\n\nQUESTIONS:\nN.A.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMotivated by recent prediction based scheduling of ML jobs in data centers, the paper considers the problem of prediction cost aware scheduling to optimize mean response time. It considers two cost models - external and server time. The paper proposes a novel algorithm, SkipPredict, that uses a two level hierarchical prediction with a cheap short/long prediction at the first level and an expensive total size prediction at the second level. It then uses this information to schedule short jobs as FCFS and long jobs by SRPT. The paper presents an analysis of the expected response time for both models using the SOAP framework and present a comparison against FCFS, one-level FCFS+SRPT and SRPT. Finally, it provides experimental evidence of SkipPredict’s superior performance on synthetic and real world datasets.\n\nSTRENGTHS:\n- The paper is the first to consider the important aspect of cost of prediction in scheduling\n- The proposed algorithm, SkipPredict, elegantly utilizes an efficient hierarchical scheme and is practical to implement\n- Thorough and well presented analysis of SkipPredict as well as the baselines including generalisation to non-fixed costs\n- Baselines compared against are comprehensive\n\nWEAKNESSES:\n- While the motivation for the problem comes from scheduling ML jobs, the algorithm and analysis are solely queueing theoretic arguments which makes me question the relevance/interest of this work to the NeurIPS community. MLSys/SIGMETRICS perhaps might be a better fit. \n- Further, I believe none of the experiments presented reflect the characteristics of a machine learning job. The Exponential and Weibull distributions of service considered in the synthetic data are unlike the very light tailed (almost deterministic) nature of ML jobs. The real world datasets consist of CPU computations which make them unlikely to be from ML jobs.\n\nQUESTIONS:\n- While the paper analytically shows that SkipPredict does better than the baselines for any general service distribution, it would also be good to see this in the experiments with distributions/datasets representing ML jobs\n- (Minor) It would be nice to have a short explanation of how the SOAP framework is used in the main paper as well instead of just in the appendix.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers the job scheduling in the M/G/1 queueing model when the system has access to predictions regarding job lengths. The paper explicitly considers the cost incurred for obtaining the predictions in such a system. Two models are considered - (i) external cost: obtaining predictions incurs a fixed cost, but do not affect the service time of a job, and (ii) server time cost: predictions are obtained via operations that themselves need to be scheduled incurring time that delays jobs.\n\nThe authors consider a setting in which there are two kinds of predictions - (i) a cheap 1-bit prediction that only classifies whether the job length is above or below a fixed threshold T, and (ii) an expensive prediction that predicts the actual job length.\nThe authors propose a natural algorithm, called SkipPredict, that first obtains cheap predictions for all jobs to classify each job as either \"short\" or \"long\". All jobs classified as \"short\" are scheduled via FCFS. The algorithm then obtains expensive predictions for all long jobs which are then scheduled using the shortest predicted remaining time first rule.\n\nThe authors analyze this algorithm under both cost models using the SOAP analysis framework and obtains explicit expressions for the mean response times of jobs.\n\nSTRENGTHS:\n- The paper raises an interesting question. Most work on learning augmented algorithms assumes that predictions are available to the algorithm for \"free\" which is certainly not true in practice. Explicitly modeling prediction costs is a good area for further research. The server time cost model that models the delays due to predictions is especially interesting for scheduling problems.\n\nWEAKNESSES:\n- The paper is missing some references to prior work. For example, \n(i) \"Online algorithms with Costly Predictions\" (Drygala et al) considers costly predictions in other learning augmented settings. \n(ii) \"Parsimonious Learning-Augmented Caching\" (Im et al) considers caching with goal of using few predictions.\n(iii) many papers on learning augmented scheduling.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}