Budgeted-Bandits with Controlled Restarts with Applications in Learning and Computing
Authors: Semih Cayci
Yilin Zheng
Atilla Eryilmaz
Venue: Accepted by TMLR ()

Abstract: Maximizing the cumulative reward of a sequence of tasks under a time budget has been ubiquitous in many applications in computing and machine learning. Often, tasks can have random completion time and the controller needs to learn the unknown statistics while making optimal decisions. In addition to the classic exploration-exploitation trade-off, it has been shown that restarting strategy can boost the performance of the control algorithm by interrupting ongoing tasks at the expense of losing its reward. In this work, we consider a bandit setting where each decision takes a random completion time and yields a random (possibly correlated) reward at the end, both with unknown values at the time of decision. The goal of the decision-maker is to maximize the expected total reward subject to a stringent time budget $\tau$. As an additional control, we allow the decision-maker to interrupt an ongoing task and forgo its reward for a potentially more rewarding restart. Unlike previous works, we do not: assume any prior knowledge on the system statistics, or limit the action space of restarting strategies to be finite. Under this general framework, we developed efficient bandit algorithms to find optimal arms and restart strategies with $O(\log(\tau))$ and $O(\sqrt{\tau\log(\tau)})$ regret for both finite and continuous set of restart times, respectively. Furthermore, through numerical studies, we verified the applicability of our algorithm in the diverse contexts of: (i) algorithm portfolios for SAT solvers; (ii) task scheduling in wireless networks; and (iii) hyperparameter tuning in neural network training.

Published in Transactions on Machine Learning Research (07/2025)
Badgetted-Baditte with Controlled Restarts with Applications in Learning and Computing
Semih Cayci*
aoyclennath..rubbcccceee...
Deportment of Mathematics RWTH Aachen University
Yilin Zheng"
yllinzhenggkggonll.ccom com
Google
Atilla Eryilmaz
eryilmaz. edu
Department of Electricol and Computer Engineering The Ohio State University
Reviewed on OpenReview: https:
// openreview. net/ forum? id= Lub5qDAa4B
Abstract
Maximizing the cumulative reward a sequence of tasks under a time budget has been ubiquitous many applications computing and machine Icarning. Often, tasks can have random completion time and the controller needs learn the unknown statistics while making optimal decisions. In addition the classic exploration-exxllotttion trade-off, it has been shown that restarting strategy interrupting ongoing tasks at the expense of losing its reward. In this work, we consider bandit setting where each decision takes random completion time and yields random (possibly correlated) reward at the end, both with unknown values at the time of decision. is maximize the expected total reward subject to a stringent
boost the performance of the control algorithm by
The goal the time budget т. As an additional control, we allow the decision-maat to interrupt an ongoing task and forgo its reward for potentially more rewarding restart. Unlike previous works, we do not: assume any prior knowledge on the system statistics, limit the action space of restarting strategies to be finite. Under this general framework, developed elficient bandit algorithms to find optimal arms and restart strategies with ((log(7) and O(VTlog(T) regret for both finite and continuous set of restart times, respectively. Furthermore, through numerical studies, we verified the applicability of our algorithm the diverse contexts of: (i) algorithm portfolios for SAT solvers; (ii) task scheduling in wireless networks; and (iii) hyperparameter tuning in neural network training.
1 Introduction
Processes under time budget, which continue until the total lime spent exceeds a given time horizon, have long been a focal point of scientific interest as a consequence of their universal applicability in broad class of disciplines in computing and machine learning (Rednor 2001 [ondamin et al. 2007). It has been shown that any process with time budget can employ controlled restarts with the goal expediting the completion times, thus increasing the time-efficiency of stochastic systems (Pal & Reuveni 2017). Consequently, restart. strategies have attracted significant attention boost the time-efficiency of stochastic systems in various contexts. They have been extensively used to study diffusion mechanics (Evans & MMjumdda 2011 Pal
el al. 2016). largel search applications (Kusmierz et al. 2014 Eliazar el al. 2007), throughput maximiza Lion
[Asmsssen et al. 2008), and run-ting uun-times of randomized algorithms (Hoos & Stützle 2004 Luby et al.
1993
"Equal contribution

---

Published in Transactions on Machine Learning Research (07/2025)
Selman et al. 1994p. In particular, they have been widely used a tool for the optimization of randomized algorithms that employ stochastic local search methods whose running times exhibit heavy-tailed behavior (Lbyy et al. 1993 Gomes et al. 1998).
In this paper, we investigate the cumulative reward maximization problem in the context of budgeted decision processes under controlled restarts continuous time. Building on the knapsack bandit framework, we address the challenge of selecting the optimal restart time in addition the exploration-explitttiio trade-off. This learning problem has unique dynamics: the cumulative reward function is a controlled and stopped random walk with potentially heavy-tailed increments, and the restart mechanism leads to right- censored feedback, which imposes specific information structure. In order to design efficient learning algorithms that fully capture these dynamics, we incorporate new design and analysis techniques from bandit theory, renewal theory, and statistical estimation. For both finite and continuous sets of restart times, we develop provably efficient online learning algorithms to learn the optimal action and restart strategy at the same time.
In addition to the theoretical performance guarantee, we investigale the budgeted-bandiis learning problem under controlled restart strategies through numerical studies. In particular, our empirical studies show that our algorithm can be useful in boosting the performance of SAT solvers, scheduling tasks in distributed networks, and deciding the early stopping time when training neural networks. These results verify the applicability of this general framework and the performance of the proposed algorithms.
1.1 Related Work
Bandit problem with knapsack constraints and its variants have been studied extensively in the last decade. In György et al. (2007); (Badanidiyuru et al. (2013); Combes et al. (2015 Xia et al. (2015); Tran-Thanh ct al. (2012): : Cayci et al. (020). the problem was considered in a stochastic sctting. This basic sctting
was ex extedded to linear contextual setting (Agrawal & Devanur 2016), combinatorial semi-bandit, setting
Sankararaman & Slivkins 2017), adversarial bandit setting (Immorlica et al., 2019). For further details
in this branch of bandit theory, we refer to Slivkins et al. (2019). Bandits with delayed feedback (Joulani
et al., 2013 Vernade al.) 2020 (Cesa-Bianet et al.) 20166) allow the reward to be delayed. However, as a substantial difference, these works do not incorporate a restart or cancellation mechanism into the learning problem. While in our work, restarting improves reward-per pew-ndd-e--uni-tinne due to increasing residual complction times.
The restart mechanisms have been popular, particularly in the context of boosting the Las Vegas algorithms. The pioncering work in this branch is Luby et al. (1993). where a minimax restart strategy is proposed to achieve optimal expected run-time up to a logarithmic factor. In Gagliolo Schmidhuber/ (2007). a hybrid learning strategy between Luby scheme and a fixed restart strategy is developed in the adversarial setting, In Streeter & Golovin (2009), an algorithm portfolio design methodology that employs a restart strategy is developed as an extension of the Exp3 Algorithm in the non-stochastic bandit setting, and polynomial regret is shown with respect to constant factor approximation of the optimal strategy. These works are designed in a non-stochastic setting for worst-case performance metrics. In contrast, our work incorporates the statistical structure of the problem and learns the underlying statistics of the system based on empirical observations, Note that this restarting mechanism is completely differrntt from the stochastic bandit (Cheung et al. 2019 Zhao et al. 2020 Kang et al., 2023) sctup where restarting is for the sampling process, for example, using a sliding window or discounted factor to forget obsolete information.
Combining the budgeted- bangettd-bbandis bandits setting and controlled restart strategy is a particularly challenging problem. In Cayci et al. (2019). they tried to address the problem with prior knowledge of the moments. In addition, they only considered uncorrelated rewards and a finite set of restart times. Later works like Liu & Fang (2023 2024) extended the framework by adding additional stochastic constraints. However, they still focused on a limited set of restart times. Our work improves on previous works by climinating the necessity of prior statistical knowledge and allowing a general (possibly continuous) set of restart times. In addition, we can also allow correlations between the completion time and reward.
2

---

Published in Transactions on Machine Learning Research (07 (225))
1.2 Contributions
In light of the related work above, main contributions in this paper are follows:
This work provides a principled approach continuous-time exploration--expoitatoon problems that require restart strategies for optimal time efficiency. We study and explain the impact of restart strategies in a general knapsack bandit setting that includes potentially hcavy-tailed and corvelated lime-reward pairs for each arm. The design and regret analysis of learning algorithms require tools from renewal theory and stochastic control, as well as concentraaion inequalities for rate estimation. These results can be useful in other bandit problems as well (ef. Section 2 to Section
For a finite set of restart times, we study algorithms that use variance estimates and the information structure that stems from the right-censored feedback so as to achieve O(log(7)) regret bounds with no prior knowledge. In particular, we propose:
the UCB-RB Algorithm based on rate estimation based on sample-mean estimator to achieve particularly good performance for finite and small restart times (ef. Theorem 2).
the UCB-RM Algorithm based on median-hoosted rate estimation without assuming any knowl- edge of higher-order moments to achieve good performance a very general setting of large (potentially infinite) restart limes at the expense of degraded performance for small restart times (ef. Theorem
For a continuous decision set for restart strategies, we propose an algorithm called UCB-RC that achieves (og(T) regret (cf. Theorem 3
We evaluate the performance of the learning algorithm developed in this paper for boosting the performance of SAT solver, task scheduling in distributed networks, and finding early stopping time for training neural networks (cf. Section
1.3 Notation
In this subsection, we define some notation that will be used throughout the paper. I denotes the indicator function. For any E Z4, [k] denotes the set of integers from to k i.e., [k] = (1.2,...,,k) For x E R (x)+ = max(0, x). For two real numbers x, y, we define I A y = min(x, y} and x V y = max(z,y).
1.4 Motivating Examples
The problem we proposed has many potential real-world applications where a controller has to make decisions (possibly with restart) under a time budget with no prior knowledge of the statistics. provide some motivating examples that fall within the scope of this framework.
In this section, we
(Algorithm Portfolio Selection for SAT Solvers) Boolcan satisfiability (SAT) is canonical NP- NP-complete problem (Hoos & Stützle 2004 Arora & Barak 2009). As consequence, the design of efficient SAT solvers has been an important long standing challenge, and an annual SAT Competition is held for SAT solvers (Heule et al. 2019 In these competitions, the objective of a SAT solver is to solve as many problem instances as possible within a given time interval [0, 7]. which defines a budgeted decision process. As a result of the completion time distributions, the restart strategies have been an essential part of the SAT solvers (Luby ct al. 1993 Selman al.] 1994). Therefore, the proposed framework can be used as a meta-learning algorithm to improve the performance of SAT solvers.
(Task Scheduling in Distributed Server System) In many distributed server systems such web servers, data base servers, and computing clusters, tasks for service arrive randomly and must be assigned to one of the host machines for processing (Liu & Fang 2023). This is especially important if the communication is through wireless channels. Under a limited time interval [0, r], a controller will try to maximize the mumber of completed tasks. has been shown that the task completion time has a heavy-tail distribution in many applications (Harchol-Baie Balter 1999). Therefore, incorporating restarting strategy as designed in the proposod framework can be useful.
3

---

Published in Transactions on Machine Learning Research (07/2025)
(Tuning the Stopping Time for Neural Network Training) It is well-known that over-fitting can happen when training a machine learning model such as neural network on a training set for too many steps (Caruana et al. 2000 Prechelt 2002). Early stopping is common strategy to terminate the training in order to achieve higher testing performance on unseen data. It has been shown that carly-stopped neural networks are consistent {Ji el al. 2021 and more robust. to noise (Li et al. 2020). However, the decision of early stopping is often made based on experience of specific networks and applications. As an alternative, our framework can be used as a general control method to boost the testing performance while finding an optimal early stopping time.
We will revisit these examples with simulations in Section 6 In the following section, we describe the problem setting in detail.
2
System Setup
We consider a budgeted decision process with a given time horizon 7 >> 0. The increments of the process are controlled as follows: if arm & E [A] is selected at the n-th trial, the random completion time is X that is independent across and independent over n for cach k with the following wcak conditions:
B[XE, 00, 3p 4,
(1)
N2, 0, a.s.
(2)
Note that Xk,n potentially be heavy-tailed random variable. We do not assume any prior knowledge of the higher-order statisties, and the bounded fourth-order moment assumption enables us to use empirical Bernstein inequality, which operates without any prior statistical knowledge. Pulling arm k at trial n yields a reward RR,n at the end. The joint variable (NR,u, Rx,n) is sampled identically for every n. Note that we allow RE to be possibly correlated with Xa,0; and for simplicity, we assume that Ra,n E [0, 1] almost surely.
In our framework, the controller has the option to interrupt an ongoing task and restart it with potentially different arm. Namely, for set of restart (or cutoff) times T C [0, oo], the controller may prefer to restart the lask from its initial after lime L E T at. the loss of the ultimate rewards and possible additional Lime-varying cost of restarting the process. We model the resetting time (cost of restarting) as (t( E [0, which is a deterministic function of / E T known by the controller for simplicity. Note that the design and analysis presented in this paper can be extended to random resetting time processes (Evans & Majumdar 2011).
If the n-th trial of arm & is restarted at time E T, then the resulting total time and reward are as follows:
Uk (t) min m t) + Cx(t)L(Xx.x > t}.
33)
(t, = RI tt,
(4)
Note that the feedback (Un,n()) (t) in this case right-censored and the time-cost here evolves continuous time.
Incorporating the restart mechanism, the control of the decision-maker consists of two decisions at n-th trial:
Ta = (In un) E [K] x T.
Here, In E [K] denotes the arm decision, and Un E denotes the restart time decision. Under policy
{m n 2 1}, let
=
be the history until n-th trial. We call a policy a admissible if Tx+1 is -eameauurable for all n 2 1. For given time horizon 7 > 0, the objective of the decision-maker is to maximize the expected cumulative reward collected in the time interval [0, r]. Specifically, letting
(T) = inf n Σ Un)((s) > 7
(5)
4

---

Published in Transactions on Machine Learning Research (07 (07/2025)
be the number of pulls under policy The number of pulls depends since each decision incurs decision-dependenl random time, which is subject to stringent budget. The cumulative reward under æ is defined as follows:
N=(r)
REW (7) = Σ (un)
(6)
The decision-maker attempts to achieve the optimal reward:
DPT(T) (T))
(7)
well
where II is the set of all admissible policies. Equivalently, it aims to minimize the expected regret:
REG, (r) = DPT(T) - E[REW,
(8)
2.1 Right- Censored Feedback and Information Structure
Since the leedback obtain for decision of (k, is the pair of right-censored random variables (U (t), (t)) as in equation for any t' S we have the following:
Uk,( (t') (Uk,((t)) -eeauurall, (t' a (Vr.n ( measurable.
In other words, the feedback from restart time decision 0 can be faithfully used feedback for another restart time decision 1' S t. This implies the information structure is asymmetric, i.e. the information gain by large E T is larger compared to l' < t. As we will see later, learning effectively from the right-censored data using the information structure it incurred is important for designing an effective online learning algorithm.
3 Summary of Main Results
In this section, we present the results of the paper informally. Proposition 1 (Informal Regret Upper Bound for UCB-RM), For a finite set of restarting times [T] = L with co E T and K arms, the regret of UCB-RM is
REG,M S ar log(T) Σ +KKK))
where o% 00 k [K] are constants (with respect to T) that depend the higher-order moments of the random completion time and reward for arm k. The full statement with exact conslants is given in Theorem 7
Proposition 2 (Informal Regret Upper Bound for UCB-RB). For finite set of restarting times [T] L with oa & T and K arms, the regret of UCB-RB is
REG, (T) S log((T E
where & > 0, & E [K] are constants (with respect to T) that depend on the higher-order moments of the random completion time and reward for arm k. Typically, these coefficients are smaller than cr. The full slatement with exact conslants is given in Theorem
Note that in the general definition of T, we included the action oo in T, which corresponds the case where each arm pull is completed without restarting. In Proposition 2 impose the condition that each arm has finite restart time to obtain the regret guarantees for UCB-RB.
Proposition 3 (Informal Regret Upper Bound for UCB-RC). Using fixed discretization step-size for the continuous set of reslarting lime, the vegrel of UCB-RC salisfies
(T)
lim sup
(9)
7-300
/lg((T

---

Published in Transactions on Machine Learning Research (07 (07/2025)
where is problem dependent constant that is independent of time The full statement with exact constants is given in Theorem 3 In the next section, we first give an asymplooicalll-oopinaa offline policy baseline.
4 Asymptoticall--ptinnal Offline Policy with Known Statistics
The control problem described in equation 7 is a variant of the well-known unknown stochastic knapsack problem [Keiinbrgg & Tardos 2006). In the literature, there are similar problems that are known to be PSPACE-hard (Papadimitriou & Tsitsiklis 1999; [Badanidiyuru et al., 2013). Therefore, we need to first. find a tractable algorithm approximate the optimal policy. In this section, we propose a simple policy for the problem introduced in Section 2 and prove its efficiency by using the theory of renewal processes and stopping times.
First, we define the (renewal) reward rate, which will be the main quantity of interest when designing algorithms.
Definition 1 (Reward Rate). For decision (k, t), the (renewal) reward rate is defined follows:
I[ S (t) = E[(XA,I Cx(OI((e)) > (H (()
[V,,
(10)
(t) is the growth rate of the expected total reward over time if the controller persistently chooses the action (k,t), known the reward rate in renewal theory. In other words, the reward rate represents the time average reward per unit time. As consequence of the elementary renewal theorem (Gut) 2009), the reward of the
offline policy that persistently makes a decision (k, t) is E[REW=(T)) = (t) T o(T). the following, we will provide an upper bound for (PT(r) based on (t) and the time horizon т.
Proposition 4 (Upper bound for OPT). Let the optimal reward rale be defined as follows:
r" arg (t(t). (k.t)
(11)
If there erists a > 2 and u < oo such that [ Poo S upper bound for OPT:
holds for all k E [K], then we have the following
OPT(T) S
(12)
for any 5 > 0 where (u) is constant that is independent of T.
Proof. The proof generalizes the optimality gap results in (Xia et al. 2015 Cayci et al. 2020). Under any admissible policy ı E II, an extension of Wald's identity yields the following upper bound:
E|REW, (7)) < r'B(UN+())
where (+) is the first-passage time of the controlled random walk under æ. The excess over the boundary, [[)] -, is known be o(T) for simple random walks by the elementary renewal theorem (Gul 2009 For controlled random walks, [allley & Lorden 1986) shows that [[)N,(+)] - O(u), which is independent x of t, if F0 S < oo holds for all k [K] for some Po 2. Thus, under the moment assumption
equation I we have ((),,(t)
((1) as 7
From the discussion above, it is natural to consider an algorithm that optimizes (t) among all decisions (k, !) as an approximation of the optimal policy. Accordingly, the optimal offline policy, denoted as π offf is given Algorithm |
The performance analysis of off is fairly straightforward since the random process it induces is a simple random walk.

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Algorithm 1: Asymphotiialll-pppmall Offline Policy nof
0: Sn = 0: while S, T do
(h' (1)) arg sup (t); (k.1)
// Maximize equation
10
if Xx-.a 1s then
Su+1 = Sn + Xxt,ni
Oblain reward Ra
else
Sut1 = Sn +t Cx-(t");
// /Restart time
n 11
/ //New epoch starts
Proposition 5. The reward under off is bounded as follows:
Y
E REW oorr r (T+
(13)
[E(Y.))2)
where Y, (t") is the completion time of an epoch under noIE, and (t)) is defined Algorithm 1
The proof of Proposition follows from Lorden's inequality for renewal processes (Asmussen 2008
As consequence of Proposition 4 and Proposition 5 the optimality gap of the offline policy is bounded for all + > 0.
Corollary 1 (Optimality Gap of o.f)). For any > 0, the oplimality gap of not is bounded: OPT(T) - E[REW,.. ou (r) S ((u),,
(14)
where (uu) is the constant in Proposition 4
As we can see, the expected reward of the simple offline policy only has a bounded gap with the best possible policy, which can only be achieved by an NP-hard algorithm. addition, we observed that the reward rate (t( the dominant component of the cumulative reward. Next, we will analyze the behavior of rx(() with respect the restart time f.
4.1 When is it optimal to restart?
For any given arm [K] and any set of restart times T,, let the optimal restart time be defined as follows:
t E arg sup rx(t). IET
(15)
If the completion time Xam and reward R are independent, then it was shown in CCacii et al. 2019 that
it is optimal to restart a cycle at a finite time if the following condition holds:
B[Xx,1 ( > 1 B[Xx,1]
(16)
for t T\ {oo}, and it is noted that all heavy-tailed and some light-tailed completion time distributions satisfy this condition. Exponential distribution corresponds to the boundary case where for all t 2 O, which follows from the memoryless property. If X and R& are correlated, this is no longer true and the situation is more complicated. In the following, we extend this result. to correlated (Xa,1, Ra.1) pairs.
> 1] xx,)
Proposition (Optimal Restart Time). For a given arm (XA,1, RA,1, (t()),, we have to 30 if and only if the following holds:
E R
X
E R
(17)
XR.1 (t+ Ca(t)) X >
E

---

Published in Transactions on Machine Learning Research (07/202) 2025)
10.5
Renterd rate 1.
10f
Rostort time
Figure 1: Impact of correlation between Xg,n and RR,n the optimal restart time for XR,n Pareto(1, 1.2) and y E [-,, Positive correlation between the completion time and reward leads to higher restart times, and waiting until the completion of every task is optimal for y 2 1 since waiting becomes more rewarding.
for some 0.
The proof follows from showing equation 17 is equivalent ru(t) EERR,,)/B/Xx.i for any
Interpretation of Proposition is as follows: for any restart time & T {xc), if the reward rate of waiting until the completion (i.e., the residual reward rate) is lower than the reward rate of new trial, then it is optimal to restart.
Note that equation 16 is a special case of Proposition with immediate returns, i..., Cx(t) = 0, and Xx.n and RE, are independent.
4.2 What is the impact of correlation on the restart time?
The correlation between Nk, and Rk has a substantial impact on whether the optimal restart times are finite not. As an example, let Xg. Pareto(1, for some a E (1,2), and C2 (t) 0 for all t.
1. If Xk, and Ran independent, then equation 16 holds and t co, i.e. it is optimal to restart after a finite time.
2. If Rg, = wXy kn for some > 0 and 1, then it is optimal to wait until the end of the task, i.e.,
4 = =00
3. If Rk. = w kn for w>> 0 and 1, then we have t < 0o.
The impact of the correlation between X and Re on the behavior of optimal restart time is illustrated in Figure I
In the next section, we develop online learning algorithms for the problem, and present the regret bounds.
5 Online Policy for Controlled Restarts with Unknown Statistics
In this section, develop online learning algorithms with provably good performance guarantces. The right-censored nature of the feedback due to the restart mechanism imposes interesting information structure to this problem. We first describe the nature of this information structure.
We consider a finite T= (t1, 12. .t, such that
4 < 12.
< t1 0o.
(18)
Throughout the paper, we assume that any action set. T satisfies the following assumption:

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Assumption 1. Given decision set T, there erists pax that satisfies the following:
Bfmin{Xxs 1,,,,,111111 2 Me,
and
P(XR11 S Ru.1 = =p) 2 E [0,1],
for all E [K], where 1 = minn T.
Note that Assumption is a simple technical condition that ensures efficient cstimation (x) from the samples of (N ) for all E
In order capture the benefits of the information structure, for arm is and restart time decision tt, let
[n)(n) {i n = (k, 4)).
Then, the available feedback for decision (k, is follows:
(n) = Jn,(n))
V21
The size of [x,((n), ie., the number of samples available for (ki 1) defined as follows:
Te,(n) [n)] Σ Tn),(n),
(19)
P21
From above, it observed that the information structure increases the number of samples substantially for each decision, i.., Th,(n) 2 (n)(n) for all k,
Compared with previous works (Cayci et al. restarting times. Under this general framework, the radius of the set of restart times T has crucial impact on algorithm design and performance, depending on the tail distributions of the completion times. In the following subsection, we propose two algorithms for both small and large (potentially infinite) fL, and compare their characteristics.
2019 Liu & Fang 2023
2024).
allow general set of
5.1 Finite Set of Restart Times: UCB-RM and UCB-RB
In this section, we considered a finite set of restart times and propose two algorithms UCB-RM and UCB-RB based on median-of-mean estimators and empirical mean estimators respectively (Algorithm 2 Note that the information structure that stems from the right-censored feedback is utilized. As we will see, this information structure will lead to substantial improvements in performance.
5.1.1 UCB-RM Algorithm
For an index set C Z4, let the empirical mean Eg and empirical variance V a random sequence (Y, : 2 1} be defined follows:
Es =
Σ Yo.
(20)
S iES
Y Es(r))
Vs(y] =
(21)
S
iE
In general, if the completion time distributions are such that the optimal restart time is very large or potentially infinite for some arms, the radius of the set of restarting time T will be large. In this case, the empirical rate estimation will not provide fast enough convergence rates, which can lead to substantially deteriorated regret. bounds.

---

Published in Transactions on Machine Learning Research (07/2025)
In order to overcome this, we design UCB-type policy that incorporates median-based rate estimator which is defined in Definition 2 This estimator can achieve good performance in very general setting that allows not restarting as possible action, i.e., we consider a decision set
T [t1 < <
< <t o0),
where 4r. oo implies the controller can wait until the lask is completed.
Definition 2 (Median-based rale estimator). Consider (k;,4) [K] T, and let G1, G2...., Gm be partition of the set (n) such that |G,] [Tk,(n)/m]] Vj [m]] for [3.5a log(n]] 1, 2. The median-of-mean estimator for Un (fi) is defined as follows:
M (Un(t)) median ( [Ux(tt)) Ea, .... EG.. (ttt).].
(22)
where [X is the empirical mean of a sequence over the inder defined in equation 20 Then, the median-base rate estimator for the action (k,ti) is defined as follows:
()),,()) (Vi(
(23)
Fk MF, (Uk(tt))
Intuitively, the median-of-meat -fann estimators are more robust to potentially large samples caused by heavy-tail distribution. Similarly, for variance estimation, using the empirical variance Vs defined equation 21 we can define
VM
(Ux(t)) median (aeiaannvss. VV. (Uk()) Vo.. VVc.
With these median- beedian-based estimators, we can construct the confidence radii as
(1 B)2
The + P thim M (Uk((t))
(0,1);
(24)
k.
(1 B)
where the parameters are defined as
28
2. (,,) UUx (4i) log(na)
M k.1.a = 11
(25)
(k,
2
I, (m) (Vi
log(na) (o)(()))
Nh.n = 11
(26)
(n)
In this way, the inequality kln + kki. > rx(tt) holds with high probability for sufficiently large (n). Therefore, we can construct UCB-RM algorithm denoted as T² (Altorighm D At cach decision step, UCB-RM Algorithun makes a selection for the (n + 1)-th arm pull using the following criterion:
(J441)Pa41) E arg max (xx))(
(FR. + cha.
(27)
In the following theorem, we analyze the performance of the UCB-RM Algorithm. Theorem 1 (Regret Upper Bound for UCB-RM). For any arm k and restart time t,, let AE, = rs - (x((tt
and
Ct,1 (2
(28)
K K(UR, (41) (V),
(29)
(1 13)
2(3) max {2v2
(1 - (133)3 3 e (0,1),
(30)
10

---

Published in Transactions on Machine Learning Research (07 (07/2025)
where (XX) and K(X) is the coefficient of variation and kurtosis of random variable X, respectively
Gallager 2013). Then the regret under the UCB-RM Algoritimm is bounded as follows:
REG,M (T) S Σ
3a (4 5 ( 1 O(KL),
k
where for each &s E [K] and some constant Ç 0, 22
c (1)=2(3))k.
+ 1024 + S)
(31)
Au.
r + 1024((KE. + }
c max Dk. max (1)C
(32)
A
Note that the regret upper bound in Theorem [ dependent only on the first- and second-order moments of X and Rg,.- There is no dependency fr. Therefore, the UCB-RM Algorithm is efficient for the cases with very large (potentially infinite) tz. Also, compared with the modian-basee algorithm UCB-BwI in (Cayci et al. 2019), UCB-RM Algorithm does not require any prior knowledge of the statistics like moments. Instead, it uses empirical estimates for reward rate, mean completion time, and variances.
5.1.2 UCB-RB Algorithm
In many cases, the optimal restart time finite for all arms, i.c., 1 < 00, Vk. Therefore, the action set T can be localized around the potential optimal restart times and tr can have finite value. Under this scenario, the support set of the completion times (t) is small for all t E T, which enables the use of fast estimation techniques. Based on this, propose algorithm with better performance based empirical Bernstein incquality inspired by the UCB-B2 Algorithm in the classical stochastic bandits with knapsacks setting (Cayci el al. 2020). We denote the new algorithm as UCB-RB (T²). Since it uses empirical estimation, it inherently assumes that tz < sc.
For any (ks, ft) pair, we define the empirical reward rate as:
E Ei [ta(
fk.
(33)
E E (t))
For a a>> and / E (0, 1), we can construct the confidence radii as
(3 + 1)a Me..n + FRiLnkkin 1 3
Ck
(34)
E (U((t)))(((tt)
where the parameters defined as
3t, log(n")
2V F. (m)) (Ui) log(na)
6k,i,
(n.
(n)
3 )
2W Te, (Va(ti)) log(n*)
Tk.((n)
Tk.(n)
Then, the controller under UCB-RB makes decision at (n 1)-th using the following criterion:
(In+1, (x41)
arg max (fx.m + c.d.
(KA))(K])T
The following theorem provides problem-depenndent regret upper bounds for the UCB-RB Algorithm.
11

---

Published in Transactions on Machine Learning Research (07/2025)
Algorithm 2: Online Learning Algorithms for Finite Set of Restart Times
UCB-RM (x*) and UCB-RB (x²)
n == 1,
// Begin inilialization for init trials
while & init & S,-1 $ B do
for KE E [K], E [L] do
(/, = ti): S. Uk))(();;
Obtain reward V..nn(.))
12 1;
i= 1;
while Sn B do
//If using UCB-RM Algorithm
(4n, 1/,) = arg max fdin- + M (R.t))
// If using UCB-RB Algorithm
(In, kn) = arg max (f + c (k.42)
if X S U/ then
Sn Sn X Kani
Obtain reward Ri
else
L S, = 1n-1 + Ck. (vn);
Update the estimates for 1 t S Uni
Add + (n) for all 1 t S Vn: n=
Theorem 2 (Regret Upper Bound for UCB-RB). For any le and restart time t,, let A r. -rk(ti), Nel = B(Ux,1(4)) and
24
BA.1 B(VA,1 (1t)
(35)
(4))
Then, the regret under the UCB-RB Algorithm is bounded as follows:
REG, (T) S Σ 3a min(Ex, C log(T) O(KL),
where for each & E [K] and some a > 2,
r
2z(B)r.Ba.
&
Σ DR. pk. {3(3)
+8(KE, BB2,)}
(36)
A
DK.1
1=1
& max (Aar m..)) max (3)
r +22(8)BB.
To
+ +KK,.. + BB,))
(37)
A
DR.1
with Ck.,, KR,,, and 2(3) defined in Theorem 1
(Comparison between UCB-RM and UCB-RB) Comparing the cocfficients of the log(T) term in Theorem 2 and Theorem ! we observe that the UCB-RM Algorithm suffers from a considerably large scaling coefficient. This suggests that the optimal restart times are known to be small, then the UCB-RB Algorithm is more efficient than the UCB-RM Algorithm.
On the other hand, the regret upper bound in Theorem grows at rate O log(r)), where the constant additive term independent. of and 4 if [[XE,) < for some p 2. Therefore, if the optimal restart.
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
time can take on large value, then the regret performance deteriorates significantly. This dependence Ir. stems from the nature of the empirical mean estimator used for estimating the reward rate, and it is inevitable (Audibert et al. 2009). Therefore, the UCB-RB Algorithm is suitable only for the cases where the restart times are small.
(Importance of Information Structure) For any arm k; E.: corresponds to the coefficient without using the information structure, grows linearly with L, and shows that the dependence of the regret on 4x,1 is O(1/BR,)) On the other hand, & reflects the effect of exploiting the structure, and it is usually much lower than &x- Hence, ignoring the constants, let 4 miny Ak,, Vl, k ke, we have the following order result for the regret:
REG (B) B) 0f2
Σ Aegh
1
+
1
Σ max Ax.) log(T))
max
Ak
miny A
since KK(Z)= & 6² for bounded random variable Z e [0, In other words, as a result of exploiting the information structure, the effect of large L on the regret is climinated.
Proof Skelch; The proof of Theorem and Theorem 2 follows a similar strategy as (Cayci al. 2019), and can be found in detail in the Appendix. We will provide a proof sketch here. The main challenge in the proof is two-fold: (1) analyzing the effect of using empirical estimates, and (2) finding tight upper and lower bounds for the expectation of the total reward REW, (T), which is controlled and stopped random walk with non-i.i.d increments.
For any (k, 4)) E [K] x T, let (n)) be the number of times the controller makes the decision (k, 4)) in the first n stages, and As, be the expected 'regret per unit time' if (k, t() chosen. Then, by using tools from renewal theory and martingale concentration inequalities, we express the regret as follows:
REG, (T) =
Σ
((1) (TT),
+ O(KL),
(R.1))A,,1.00
where no(T) is high-probability upper bound for (T), the total number of pulls in [0, 7]. By using a clean-event bandit analysis akin to (Audibert et al. 2009) to bound (no)))( prove the theorem.
Note that the UCB-RM and UCB-RB Algorithm makes use of the median-of-mean method and empirical estimates to achieve regret performance guarantee without any prior knowledge, and we devise novel tools to analyze the impact of using these estimation methods on the regret. In the next section, we will develop a learning algorithm for the case T continuous.
5.2 Continuous Set of Restart Times: UCB-RC
When the set of restarting times T is continuous, one common technique to discretize the set. The key question is how to design a good discretization method with provably good regret performance. In this section, we propose algorithm called UCB-RC for smooth and unimodal rl) that has /7 log7 regret. guarantee.
For the notation simplicity, we will consider learning the optimal restart time for a single arm. The extension to K >> is straightforward. Specifically, we make the following assumptions on the action set T and reward rate function (t).
Assumption 2. The set of restart times T. and reward rate ri ri()) satisfy the following properties:
(i) Compactness: T compact subset of Ry::
T [Imin: Lmaz)]
(38)
where O < tmin S tmaa < oc. Furthermore, T satisfies Assumption for some M4 > 0 for efficient estimation of r1()).
(i) Unimodality: There is an optimal restart time tf E [bminy mmaz] such that
Y. = r((()) 2 r1),),
for all 1 ((())
13

---

Published in Transactions on Machine Learning Research (07/2025)
tti
Smoothness: There erists du > 0, 0,02 0,and q such that:
For all 1,1' E [ii, óo] (or [0] 60,17)) 11), the following holds:
a1lt +/r - (r((t) rr
For some S do; of 171 S 8, we have:
r. - ri)) 5 a020
Note that Assumption 2 is satisfied for a broad class of distributions. For example, if X and Ra are independent and Xx.n has a uniform, exponential, or Pareto distribution, then the conditions are trivially satisfied.
Under Assumption we design the UCB-RC algorithm (denoted as x²) based on the UCB-RB Algorithm and the bandit optimization methodology in (Combes & Proutiere 2014 Combes et al. 2013). As we will see, making use of the information structure is necessary to achieve a good regret performance.
Algorithm 3: Online Learning Algorithms for Continuous Set of Restart Times UCB-RC (x²)
7/(1) Discretize the set of restarting times
Set the step size for discrotization
6= 3 (Vog((T)/T 1/4
For rad(T) Emax - tmin: let =(8)= rT)(T)/8
Obtain the discretized set TQ = {t1, t2. (L( where
t = Emi + (1 - 1) [1/0], 1= 1,2.,... L(6).
//(2) Run the algorithm for discrete set Run the UCB-RB Algorithm over the action set TQ.
The following thcorem provides a regret bound for the UCB-RC Algorithm. Theorem 3 (Regret Upper Bound for UCB-RC), Under Assumption the regret for the UCB-RC Algorithm salisfies the sollowing asymplotic upper bound:
(T)
E
3aq
lim sup 7
< 63
C* (8))
(39)
(og((T)
μ.
a1 ai
a1
where
EIUR,,1
E[VE,1 (E(Vi.)))
C¹ = 3²
and z(3) is defined in Theorem
Proof Sketch: The detailed proof of Theorem can be found in the Appendix. Here we give sketch of the proof. First, note that the UCB-RC Algorithm is based discretizing the decision set T, and running the UCB-RB Algorithm over the discretized decision set Tq. Therefore, chosen in UCB-RC can make sure the optimal reward rate over TQ is close enough to r.. Second, we also need to show that the number of quantization levels is kept sufficiently small. Finally, under the compactness and smoothness assumptions summarized in Assumption 2 the regret upper bound can be obtained.
need to show that the step size 8
(Importance of Information Structure) The UCB-RC Algorithm based on an extension of the UCB-type algorithm for unimodal discrete-time stochastic bandits proposed in (Combes & Proutiere] 2014 Combes et al. 2015). A straightforward extension of the algorithm in (Combes et al.) 2015) would yield (log(()) regret. However, the UCB-RC RC Algorithm achieves P(VTloo((7) regret in this case. The order reduction by a factor of ((Vlog(7)) is because UCB-RC incorporates the information structure that stems from the right-censored feedback, as discussed in Section 2.1
14

---

Published in Transactions on Machine Learning Research (07 (025))
6 Numerical Experiments
In this section, we evaluate the performance of the proposed learning algorithms for three different applications: algorithm portfolios for SAT solvers, task scheduling in distributed networks, and tuning stopping time for neural network training.
For these experiments, the restart times are finite. Therefore, we used the UCB-RB Algorithm with 2.01, (1 + () 3) 1.01. For initialization, the controller performed 40 trials for each (k, decision. The average rumning time per decision step is around 120 ms a personal computer. For comparison, used Luby restart strategy with various hand-tuned base cutoff values as a benchmark (Luby et al. 1993). Note that without any prior knowledge, the performance of Luby restart stralegy is hil-or-miss, depending on how close the chosen (guessed) base cutoff value is to the optimal restart time.
From the simulations, we will see that our sctup general enough that can be applied to various different fields. It is worth noting that there may exist fine-tuned algorithms for specific applications that outperform both Luby and our algorithms. However, as general framework, algorithms do not require prior knowledge of the system and can optimize the cumulative reward while finding the optimal restart time.
6.1 Algorithm Portfolios for SAT Solvers
this experiment, we consider a similar setup as the SAT Competition: for a given time interval [0, r], the performance metric is the number of solved problem instances, thus there is a unit reward for cach successful assignment, i.e,, Ri, = 1. We evaluated the performance of the meta-algorithms over the widely used Uniform -andom-3-SAAT benchmark set of satisfiable problem instances in the SATLIB library (Hoos & Stützle 2000). In the dataset uf uf-100-430, there are 1000 uniformly generated problem instances with 100 variables and 430 clauses, therefore it is reasonable to assume i.i.d. completion times. Each successful assignment, yields a reward R1,a = 1. We select = (10-0.5+4x0...
= 0, 0,1....88)
The empirical reward rate as a function of the restart time for the data set is given Figure 2a is observed that the controlled restarts are essential for optimal performance, in accordance with the power-lawe completion time distributions (Gomes et al. 1998) and Proposition D This implies that our design is suitable for this scenario. The nunber of solved problem instances for different values are given in Figure 2b We observe that the UCB-RB Algorithm learns the optimal restart strategy fast without any prior information, and its performance outperforms alternatives, especially at large time horizons. On the other hand, Luby restart strategy, which requires the base cutoff value as an input, is prone to perform badly with inaccurate prior information. Even if a genie provides a well-chosen base cutoff value to Luby restart strategy, is outperformed linearly over 7 by the UCB-RB Algorithm, which requires no prior information.
12505
8000
1003001
0.0
2000
JUSU bas
12
16'
12'
10"
rosson time
(a)
(b)
Figure 2: Performance of the restart stralegies on the Random-3-SAl 3-SATT data set k3.-k3.
15

---

Published in Transactions on Machine Learning Research (07 (07/2025)
6.2 Task Scheduling in Distributed Networks
In this experiment, We considered a task scheduling problem for single machine during its busy period. The machine selected will have random completion time for the task during which the controller can decide whether to interrupt the machine. If the machine completes the task before interruption we receive & reward of 1 otherwise the reward is 0. The objective of the controller is to maximize the cumulative reward under a limited time T.
It has been shown that the task completion time in many distributed networks has heavy-tailed distribution (Harchol-Baan Balter 1999: Xu et al., 2024). therefore our restarting strategy fits into this setup. Using the parameters defined in [Harchol-Balter 1999). we plot the reward rate in Figure 3a As we can see, the heavy-tailed distribution fits well with our setup. We select T = {100.5+ix00. : = O.I,...,.88 The rewards for different budgets are given in Figure 3b. Again, the UCB-RB algorithm putperforms the best Luby algorithm which requires a manual tuning on the base parameter.
0.98
1000
0.58
0.12
1000
000
000
0.34
202
narart
X10
(a)
(h)
Figure 3: Performance UCB-RB algorithms for task scheduling in distributed networks.
6.3 Tuning the Stopping time of Neural Networks
In this experiment, we consider a setup where we need to train set of neural networks to solve a sequence of tasks (with similar statistics). Fach time, we start. training neural network with random initialization. The controller will decide when to stop the training to evaluate the test results. If the test result reaches predetermined level we receive reward 1 otherwise the reward is 0. Under a limited time т (which is equivalent to training resources like GPU time), we would maximize the cumulative reward.
Specifically, in our setup, we trained a RESNET-16 over the CIFAR-10 dataset where we used 80 : 20 split for training set and testing set. To emulate the sequence of tasks, cach time we randomly sample 10 percent of the training data the new lasks. The learning rate is set. to 0.001 using an SGD optimizer with batch size of 64. We choose 0.9 as the reward threshold. As shown in Figure 4a. the empirical reward rate suggests restarting strategy in this work would be a good fit. We select T [102+ix0.25 ; 0,1,..., 88. Finally, we plot the reward against the number of allowed steps in Figure
As expected, the UCB-RB algorithm can learn to optimize the cumulative reward without prior knowledge of the system statistics.
4b
7 Conclusions
In this paper, we considered the contimous-time bandit lcarning problem with controlled restarts, and presented a principled approach with rigorous performance guarantees. For correlated and potentially heavy-tailed completion time and reward distributions, we proposed a simple, intuitive, and near-optimal offline policy with O(1) optimality gap, and characterizod the nature of optimal restart strategics by using this approximation. For online learning, we considered discrete and continuous action sets, and proposed bandit algorithms that exploit the statistical structure of the problem to achieve tight performance guarantees.
16

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
0.45
0.4
4000
5000
0.25
2500
0.01
1000
020
0.32
1/
10"
naraat time
X10
(a)
(b)
Figure 4: Performance of UCB-RB on task solving when training neural network.
In addition to the theoretical analysis, we evaluated the numerical performance of various applications and observed that the learning solution proposed in this paper outperforms Luby restart strategy with no prior information. Another interesting direction is to derive regret lower bounds for this problem. This is complicated by the stochastic nature of the stopping time of the decision-makinn process, and also due to the use of empirical estimators for higher-order moments in the UCB-type algorithms. This makes the lower bound derivation significantly more complicated compared to the traditional stochastic bandit. settings, and therefore we leave it as a future research problem. Finally, relaxing the assumptions, such as using random resetting times and dynamic discretization parameters, among interesting future research directions.
Acknowledgments
This work was funded by the Federal Ministry of Education and Research (BMBF) and the Ministry of Culture and Science of the German State of North Rhino-Westphant Westphalia (MKW) under the Excellence Stratogy of the Federal Government and the Länder. Atilla Eryilmaz's ' research was funded by NSF grants: NSF AI Institute (AL-EDGE) 2112471, CNS-NeTS 2106679, CNSS-NeTS-200223) the ONR Grant N00014-19-1-2611; and the ARO Grant -911NF-2441-0108.
References
Shipra Agrawal and Nikhil Devanur. Linear contextual bandits with knapsacks. In Advances in Neural Information Processing Systems, pp. 3450-3458, 2016.
Sanjeev Arora and Boaz Barak. Computational complerity: modern approach. Cambridge University Press, 2009.
Søren Asmussen. Applied probability and queues, volume 51. Springer Science & Business Media, 2008.
Søren Asmussen, Pierre Fiorini, Lester Lipsky, Tomasz Rolski, and Robert Sheahan. Asymptotic behavior of total times for jobs that must start over a failure occurs. Mathematics of Operations Research, 34): 932-944, 2008.
Jean- Yves Audibert, Rémi Munos, and Csaba Szepesvári. Exploration-cxploiiie exploitation tradeoff using variance estimates multi-armed bandits. Theoretical Computer Science, 187699998769992, 2009.
Ashwinkumar Badanidiyuru, Robert Kleinberg, and Aleksandrs Slivkins. Bandits with knapsacks. In 2013
IEEE 54th Annual Symposium on Foundations of Computer Science, pp. 207 216. IEEE, 2013.
Rich Caruana, Steve Lawrence, and C Giles. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. Advances in neural information processing systems, 13, 2000.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Semih Cayci and Atilla Eryilmaz. Learning for serving deadline-constranee traffic in multi-channel wireless networks. In 2017 15th International Symposium on Modeling and Oplimization in Mobile, Ad Hoc, and Wireless Networks (WiOpt), pp. 1 1, 2017.
Semih Cayci, Atilla. Fryilmaz, and Rayadurgam Srikant. Learning to control renewal processes with bandit feedback. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 3(2):43, 2019.
Semih Cayci, Atilla Eryilmaz, and Srikant. Budget-constrained bandits over general cost and reward distributions. arXiv preprint arXiw:2003.m 00365 (to appear in Proc. of 23rd International Conference on Arlificial Intelligence and Stalistics (AISTATS)): 2020.
Nicol'o Cesa-Bianchi, Claudio Gentile, Yishay Mansour, and Alberto Minora. Delay and cooperation in nonstochastic bandits. In Conference on Learning Theory, pp. 605- 622. PMLR, 2016.
Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Learning to optimize under non-stationarity In The 22nd International Conference on Artificial Intelligence and Statisfics, pp. 1079-1087. PMLR, 2019.
Richard Combes and Alexandre Proutiere. Unimodal bandits: Regret lower bounds and optimal algorithms. In International Conference Machine Learning, pp. 521-529, 2014.
Richard Combes, Chong Jiang, and Rayadurgam Srikant. Bandits with budgets: Regret lower bounds and optimal algorithms. ACM SIGMETRICS Performance Evaluation Review, 43(1):245 257, 2015.
S Condamin, O Bénichou, V Tejodor, R Voituriez, and Joseph Klafter. First-passage times in complex scale-invariant media. Nature, 450(7166):77-80, 2007.
Iddo Eliazar, Tal Koren, and Joseph Klafter. Searching circular dna strands. Journal of Physics: Condensed Matter, 1966)406,14,, 2007.
Martin R Evans and Satya N Majumdar. Diffusion with stochastic resetting. Physical review letters, 106(16): 160601, 2011.
Matteo Gagliolo and Jürgen Schmidhuber. Learning restart strategies. In IJCAI, pp. 792-797, 2007.
Robert G Gallager. Stochastic processes: theory for applications. Cambridge University Press, 2013.
Carla P Gomes, Bart Selman, Henry Kautz, et al. Boosting combinatorial search through randomization. AAAI/IAAI, 98:431-437, 1998.
Allan Gut. Stopped random walks. Springer, 2009.
András György, Levente Kocsis, Ivett Szabó, and Csaba Szepesvári. Continuous time associative bandit. problems. In IJCAI, pp. 830-835, 2007.
Mor Harchol-Balter. The e ect of heavy-tailed job size distributions on computer system design. In Proc. of ASA-IMS Conf. on Applications of Heavy Tailed Distributions in Economics, Engineering and Statistics, 1999.
Marijn JH Heule, Matti Järvisalo, and Martin Suda. Sat competition 2018. Journal on Satisfiability, Boolean Modeling and Computation, 11(1):183-154, 2019.
Holger H Hoos and Thomas Stützle. Satlib: An online resource for research on sat. Sat, 2000:283 292, 2000.
Holger H Hoos and Thomas Stützle. Stochastic local search: Foundations and applications. Elsevier, 2004.
Nicole Immorlica, Karthik Abinav Sankararaman, Robert Schapire, and Aleksandrs Slivkins. Adversarial bandits with knapsacks. In 2019 IEEE 60th Annual Symposium Foundations of Computer Science (FOCS), pp. 202-219, IEEE, 2019.
Ziwei Ji, Justin Li, and Matus Telgarsky. Early-stopped neural networks are consistent. Advances in Neural Information Processing Systems, 34:1805-1817, 2021.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Pooria Joulani, Andras Gyorgy, and Csaba Szepesvári. Online learning under delayed feedback. In International conference machine learning, pp. 155--166.. PMLR, 2013.
Yue Kang, Cho-Jui Hsieh, and Thomas Lee. Online continuous hyperparameter optimization for generalized lincar contextual bandits. arXiv preprint 3rXiv2232.00444.. 2023.
Jon Kleinberg and Eva. Tardos. Algorithm design. Pcarson Education India, 2006.
Lukasz Kusmierz, Satya N Majumdar, Sanjib Sabhapandit, and Grégory Schehr. First order transition for the optimal search time of lévy flights with resetting. Physical review letters, 113(22):220002, 2014.
SP Lalley and G Lorden. A control problem arising in the sequential design of experiments. Armals of probability, 14(1):136-1)) 172, 1986.
Tor Lattimore and Rémi Munos. Bounded regret for finite-armed structured bandits. In Advances in Neural Information Processing Systems, pp. 550 558, 2014.
Mingchen Li, Mahdi Soltanolkotabi, and Samet Oymak. Gradient descent with early stopping is provably robust to label noise for overparameterized neural networks. Tn International conference on artificial intelligence and statistics, pp. 4313 4324. PMLR, 2020.
Qingsong Liu and Zhixuan Fang. Learning to schedule tasks with deadline and throughput constraints. In IEEE INPOCOM 2023-IEEE Conference on Compuler Communicaaiins,, pp. 1-10. IEEE, 2023.
Qingsong Liu and Zhixuan Fang. Online task scheduling and termination with throughput constraint. IREE/ACM Transactions on Networkingg pp. 1-15, 2024. doi: 10.1109/TNET 200.:TTTET...22.442555000
Michael Luby, Alistair Sinclair, and David Zuckerman. Optimal speedup of las vegas algorithms. Information Processing Letters, 47(4):173-180, 1993.
Stanislav Minsker et al. Geometric median and robust estimation in banach spaces. Bernoulli, 21(4): 2308-200 2335, 2015.
Arnab Pal and Shlomi Reuveni. First passage under restart. Physical review letters, 118(())030603, 2017.
Arnab Pal, Anupam Kundu, and Martin R. Evans. Diffusion under time-dependent resetting. Journal of Physics A: Mathematical and Theoretioal, 222200,, 2016.
Christos H Papadimitriou and John N Tsitsiklis. The complexity of optimal queuing network control. Mathematics of Operations Research, 24(2):293-300 1999.
Lutz Prechelt. Early stopping- but when? In Newral Networks: Tricks of the trade, pp. 55-69. Springer, 2002.
Sidney Redner. A guide to first-passage processes. Cambridge University Press, 2001.
Karthik Abinav Sankararaman and Aleksandrs Slivkins. Combinatorial semi-bandits with knapsacks. arXie preprint ar Xrxi::17050000 081100, 2017.
Bart Selman, Henry A Kautz, and Bram Cohen. Noise strategies for improving local search. In AAAI, volume 94, pp. 337-343, 1994.
Aleksandrs Slivkins et al. Introduction to multi-armed bandits. Foundations and Trendso in Machine Learning, 12(1-2):1-22) 286, 2019.
Matthew Streeter and Daniel Golovin. An online algorithm for maximizing submodular functions. In Advances in Neural Information Processing Systems, pp. 1577 1584, 2009.
Long Tran-Thane Archie Chapman, Alex Rogers, and Nicholas R Jennings. Knapsack based optimal policies for madget-limiteed multi-armon bandits. In Twetty-Sirth AT Conference on Arlificial Intelligence, 2012.
19

---

Published in Transactions on Machine Learning Research (07 (025))
Claire Vernade, Alexandra Carpentier, Tor Lattimore, Giovanni Zappella, Beyza Ermis, and Michael Brueckner. Linear bandits with stochastic delayed feedback. In International Conference on Machine Learning, pp- 9712 9721. PMLR, 2020.
Yingce Xia, Haifang Li, Tao Qin, Nenghai Yu, and Tie Liu. Thompson sampling for budgeted multi-armed bandits. In Sehenty-Fourtte International Joint Conference on Artificial Intelligence, 2015.
Yongxin Xu, Shangshang Wang, Hengquan Guo, Xin Liu, and Ziyu Shao. Learning to schedule online tasks with bandit feodback. arXiv preprint oxxi:24022.1444 2024.
Peng Zhao, Lijun Zhang, Yuan Jiang, and Zhi-Hua Zhou. A simple approach for non-stationary lincar bandits. In International Conference on Artificial Intelligence and Stalistics, pp. 746 755. PMLR, 2020.
A Appendix
B Proof of Proposition
Proof. It is optimal to stop if and only if the reward rate of interruption is greater than the reward rate without interruption, i.c. rx (t) - E Rx,,,//B[(x..
XX 1 (t).
(t(t)
R
EIUR. [x 1.11/BBBR..
Using the identities
E(U), B[Xx,1 C++) ((1--((Xer 5)) 1[[x,11]
[() B[R B8( Xx,1 > f](1 P(Xx, 1t
we have
(t) - E[ [
E[XE.1 (t + Cx(1)) > 1 E[R > 8 E[Xx.1
R
= - P(X,1 5()
E(UR, E Rk.1] RR,1)
We get the results in Proposition G by letting the right side be greater than 0.
C Concentration Inequalities for Reward Rate Estimation
In this section, we will provide tight concentration inequalities for the empirical mean estimators and median- of-mean estimators used in the main paper. First, give lemma that provides basis to design these concentration inequalities.
Lemma (Proposition CCayci et al. 2020 ) Consider a pair of parameters pxx PR > 0, and their eslimators fx and μu, respeclively. Let r pn/ fix and r = pR/pxx px. Then, for any (0, 1), have the following inequality:
(1 33 1 n+ ) < P(uux pxl > 6) +(PRR Axl 7):
p(F-r|
(40)
1- 3
for any 1 S and & 3px-
20

---

Published in Transactions on Machine Learning Research (07 (025)
Proof. Let
A(e, = (lux Pxl S of (lun pall S <n),
be the high-probability event. Then, by Proposition 1 in CCyci et al. relation:
2020 we have the following set
A(c,n) { { 1) re
px ef
For any B E (0, e & Bux and n S 8un is satisfied, then we have:
+ rc (1+ 8) n1+ +
S
px 1 1 Ax
within the set A(e, By taking the compliment and using union bound, we obtain the result.
By using Lemmal we can prove tight concentration bounds for the renewal rate as follows.
Proposition 7 (Concentration inequalities for reward rate). Let {(Xn, Rn)) 2 1} be renewal reward process.
Xn E a] and R2 E [0, let
Fn Eim] [X]
Bim] R
and
24 m) (X) log (1/6)
3a log(1 (1/6)
(n)
n
24 ny (R) log(1/8)
3b log ( (1/6)
(6)
Then, for any 3 (0, and 8 (0, 1), we have:
IP P(F r| (1 83
(n) Pncu(())
128,
B
B(m [X]
for any n 2 ( + Ver(X.]
+ 3
c2(Xi)
+
(2( /2
(a)
9
39 R2
Consider renewal reward process such that E, B[XH] 0o and TE| [[
< co. For m [3.5 log(1/8) +1, let G1, G2...., Gm be partition of [ni] such that |Gyl nn/m]. Then, define the median- based mean and veriance estimators as follows:
Mpn (X) median (Ec, [X], Ec, [X]... E. [X]},
(X) median Vo, [X], Vo, ()...., Vo [X]]
Let
E R
PM
(m)) [
and
28 [m] (X) log(1/8)
(6) 11V
n
2w [r) (R) log (18)))
Th (6) 111
21

---

Published in Transactions on Machine Learning Research (07 (025))
Then, for any 3 E (0,1) and d (0, 1), we have:
(1 + 3)² (0) + (0)
|
r|
3 16.86,
1
(n) [X]
for (0)))((X)) K(Ri)) C c) for some Ç > 0.
8
Proof.
If 2 ((K())) + Var(X.) then the following holds with probability at least 46:
(X)) V Var(X;)) - Var(X,)/2,
where K(X)= - Vxr=(X) is the kurtosis of random variable X. Therefore, with probability at least 1 46, we have the following:
3V 3Var(X, log((1)))))(11)) 3a log(1/0) S (1]
n
Hence, if n 2 3 where C(X) /Vrr(X) (or(XX/BB)) [X] is the coefficient of variation. Therefore,
(X(x)
also holds, then the above inequality is automatically satisfied,
a b²
(X1)
(3
b
(K()) + K(R1) + Var(Xi) Var(Ri)
+3
B2
BE
32
31E R
then we have
(6) < A { Tn (6) S (1))
with probability at least 1- 88. Then we use Lemma | in conjunction with empirical Bernstein incquality (see 4Audibert et al. 2009h) to conclude the proof.
The proof follows from identical steps as Part 1, and uses Proposition 4.1 and Corollary 4.2 in (Minsker al. 2015) for the concentration results.
D Proof of Theorem 1 and Theorem 2
The number of trials (T) under admissible policy random stopping time, which makes the regret computations difficult. The following proposition provides useful tool for regret computations.
Lemma 2 (Regret Upper Bounds for Admissible Policies). Let Te,()) be the number of steps where the decision is (ks, l1) in brials, and μ.. ming. (4))) The following upper bound holds for any admissible policy π E II and T > ps/2:
> Tk,1 ( Hx
A.. (U.)( + exp(-THu//t) Σ (t)) + k.l
REG <
1 exp (u2/ (2t?))
k.1
where db is constant.
The proof of Lemma 2 relies on Aauma-Hoeffdiig inequality for controlled random walks, which can be found
in (Cayci et al.) 2019 2020). Note that 27/M. is a high-probability upper bound for the total number of
pulls N₂( (T), and Ax L. ( [t:)] is the average regret per pull for decision tt) Lemma implies that the expected regret after 27 2T/p" pulls (1)).
In the following lemma, quantify the scaling effect of using empirical estimates.
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Lemma 3. Under the UCB-RB RB Algorithm, have the following upper bounds:
(1) 15k kt, 4-- or k + k*, Vl, we have:
Σ [T) S 3 (B) 1 A
To 8(KEs BB,))
+ 23)BB Ar.j
(it) If k=== k and t t/g we have [T)] O(1) for all n.
Proof. Consider a suboptimal decision (ks, ti) where either = = k. ,t > { or tor * k*} is true, and let
E = fir t S Sra
(41)
E,, = U
Pkidiin 2 Ckijm + :r(tt)}.
(42)
321
E,,n
U [2em.in 2 Akit.
(43)
j21
Then, is trivial to show by using contradiction that {(I++1, Pnt1) = (k), 1j20) c U.f Esn In the following, we provide a sample complexity analysis for the events above.
For notational simplicity, for j 2 L, let Un (lj) and V Vl:n((z) for all For sample size and any E (0, 1), let F. = Bu[V] and [u
24 (s) (U) log(1/8)
31, log(1/5)
6x
+
(44)
8
2V (V) log(1 log(1/5)
1og((1/5)
7.
(45)
8
Then, by Proposition
the following holds with probability at least. 125:
(1+ n + faez
F. ra(tt))
(46)
1 3 E [U]
given s is sufficiently large such that
6 S [U])
(47)
T7 S [V)].
By using empirical Bernstein inequality and union bound for the empirical variance, have the following incqualities with probability at least 1 85:
(i)(UU) Var(Ui)) s (ar(U:)/2,
[V)(V) - Var(Vi)) < Var(Vi)/2,
413
if 2 (0, = [[K(UU)+{K(V)) + Var(Un) + Var(Vi]
log(1/6). Therefore, the condition equation 47 implies
the following with probability at least 1 - 85 for (1);
3V ar (U,) log(1/0)
34, log(1/8)
E, =
s (U(])
s 3 3lg((1/0)
(48)
i. 3Var(V;) (og(1/5)
< [V
23

---

Published in Transactions on Machine Learning Research (07 (07/2025)
The inequalities in equation
48
simultancously hold if
8 2 82,( := 3 (C² + E ) log(1/8).
Also, note that
(1 + 3)2 no F.CK
[1+ 8) n. + (x()))
1 3 [
- (1-8)s EE
holds if equation 48 is true. Thus, by using this result, we show that
2 + 8) 1/s + Psta
< (1 (1+3)² 3) 7h + (1(())
< Akj.
1-3 [
(1 (1-3) 3)3 B(U
with probability at least 1 - 88 if 8 2 max 82,, (0), $3,5 (6)} for
$3. (5 33) (1+ 3)4 r 3)6
(C((U))+ + (1 3)2
21 R(())) log(1/0).
(1 3)3 E[1
In summary, the following events simultaneously hold with probability at least - 126:
(1+ n/ + FaCe
F.. (n-))(t))
< 4
(49)
1 B 1
if 2 max(s1,(0), (1))))),,,,,00))) (8){
Note that the UCB-RB Algorithm is designed such that 8 8= (n) and o n= Now we will use the above analysis to provide an upper bound for Σ5 B[Tk., (n)]. First, let
イ;= (T (n) S uj), j2l
for u = max;(sig(NNq)) Then, equation
49 we have the following inequality:
P((U121 A₂) n FF U F2, U Fa..)) -1((T-1)/m 1)/n=-1
(50)
where we used union bound to deal with the random sample size Tk.5 * (n) S n in computing probabilities. Now, recall that Th (n) Eyzj T by definition, and we have the following relation:
U4, c { Σ (r) < max mzs. j21
(51)
121
321
which follows from the fact that U/214/ = Ar (AL nA)) U ( and
A)n(U)S)4)) c A c Cikks S max(0, uj - us+1l}.
Therefore, we have the following inequality:
ΣTe((N) V max Uj
Σ
I(E1, U E2, U Es,))
12
a= uy+1
Taking the expectation in the above equality, and using equation 50 and equation 51 we have the following result:
[k,(N) < max uj + 16(L - l)a
EE
j2l
- 2
521
which yields the result in part (i).
24

---

Published in Transactions on Machine Learning Research (07 (075))
For part (ii), let (ki, ti) be such that & k* and t tg. Following similar analysis as part (i) yields B[TR,(NN) & Ologg(N )), which implies that [g. (N)) = S(N - log( N)). Therefore, since the number
of samples satisfies (N) Te- (N) - Q( - lo)((N)) the decision (k*, 44) is chosen at most O(1) times
Latttmore & Munos 2014 Cayci & Eryilmaz, 2017
Combining Lemma 3 and Lemma 2 the results of Theorem follows by summarizing over all arms. The
Proof of Theorem
are similar except for using the second part of results in Proposition
E Proof of Theorem 3
The proof incorporates a variant of the regret analysis for quantized continuous decision sets given in (Combes & Proutiere 2014 into the regret analysis for budget-const rained bandits presented in Appendix D
Step 1. First, we bound the regret that stems from using a quantized decision set, Recall from Proposition
that
OPT(T) S TT. O(1),
and
(PTQ(T) 2 max ri(t),
TET
where (T) is the optimal reward in the quantized decision set Tq, and r, = maxter r1(t). Then, the regret under the UCB-RC Algorithm is bounded as follows:
(E) (PT(r) B| (7)),
(PT( QPTQ + (PT (T)),,
S r(r, max 1ı rt)) + REG, (T() O(1), xET
where
REG+,.Q(T) = (PTQ E [[]]E,,(T))]
is the regret under the UCB-RC Algorithm with respect to the optimal policy in the quantized decision set.
By Assumption
we have:
T+ max TI (t) agd",
TETy
since 1/1 - arg maxieTg ri ri(1) S 8. Thus, we have the following inequality:
REG,c(T) S 8 REG, + 0(1.
(52)
After we quantify the regret from using a quantized decision set, now we bound REG the regret of the UCB-RC Algorithm with respect to the optimal algorithm in the quantized decision set. We first present a variation of the decomposition in equation 51
Step 2.
Claim Let l" = arg max, ri(l;), and ró = max, ri (1,)) be the optimal reward rale in the quantized decision set TQ. For any 1 let A = re - rr((tt,, and
z = (3 (r²)² A
0(1)) log(N").
(53)
Then, we have the following for anyy 5 L(ő):
E Σ A,,Ti,(N))
s +
Σ
2j
A A1,j11
+
+
16L(8)a
(54)
A
a-2
jー
1=1+
25

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Proof. We have the following relation for any iS k S L(6):
k
US of Σ A15Tig(M) I 2 +
Σ
3j
1
}
(55)
1.
j=H+1
which can be proved by induction. Note that the O(1) term in the RHS of equation 53 is bounded as follows:
2=(3)B1,,rg + 8(K1, + Bi,)) A1, S 22(3)bor. - 8r, + bg)
where bo 2 maxi B1 and 2 maxt 11 are constants independent of 6, but depend on rad(T), and pa
under Assumption
By following the same steps as Lemma the proof follows.
By equation 52 and Proposition the regret under UCB-RC is bounded as follows:
REG. (T) S a2d"T + μ
Σ
E| ++A,
(56)
(E(M(8)
where μ = maxiet (1) and μl, min, (t))
Let the sets A, B, D be defined as follows:
A [1" 1,1 1" + 1}:
B {l: tlmn (I - 1)8 B(tt, n
D (
where B(xx,o) denotes the ball in IR centered at a with radius en > 0.
For any I E } we have the following by Assumption
Ar. Sr. (4) S (20)),
since r. 2 r and 1 -(mmi + (1 1)0) s 25. Thus, we have:
Σ (EA\(?"
E B(AI.TT,
2T
S azp (25)9. p.
27
(57)
11,
For 1 E B, note that
(Imin + (1" (1'-1)6)
(fmin + (1 1)6) 20(11 -1),
which implies the following by Assumption
D aj - 1090
By using this result and Claim
we have the following bound:
L(8) 32 3 (1.7 + ((lo(T))BB)
2T
E
A
Ti.
<a
alll- -0
LE
3x. 2 log(T)
+ +(loo(())BB))
89
79
1=1
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
where C1 C* for all 1 E L(8) with
BlU
C
(B(V 2)
Therefore, we have the following bound:
E
Σ A Ti,
27
log(r) C'q +(log(7)1BB). S 3az =3)²2()rr
(58)
11
0j09 q --1
For 1 E D, have (Imin (1" 1)6) (Imin + -1-13 (1-1)6) 2 60/2, hence the following holds by
Assumption 2
A 2 (0606//220
Consequently, we have the following upper bound:
2T (
(og(
E E
T1
s 3oz (B)r
O(lo(rr))D).
(59)
a (66//2)
Substituting the results in equation
57
equation
58
and equation
59 into equation 56
we obtain the following
upper bound:
log(r) a1 With the choice Vlog(T/TT, we have log(t)((6) ol / (og(())) Therefore,
2p
3aq
REG, (T) S a2 μ+
T(369
C+ (B)rr
+ ((lg(()
(60)
9-1
REG, (T)
69 P + a1(9 1) (3) [og(T)
3aq
lim sup
27


Reviews:
[{"review_id": "a4ymcR64Ri", "reviewer": "Reviewer_ddV8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The organization of the paper is good.\n2. A few motivating scenarios are discussed.\n3. Many relevant works are discussed.\n\n**Weaknesses:** some more description and explanation are appreciated.\n1. Why are the total time and reward defined as in equations (3) and (4)?\n1. I am a bit confused by the dynamic. Does the k+1-th trial start only after the termination of the k-th trial?\n1. How are the motivating examples described in Section 3 related to the formulated problem?\n1. What is the key challenge of this work?\n1. Why is the reward rate applied as the quantity to evaluate algorithms?\n1. When will the agent know the statistics?\n1. In numerical experiments, the benchmark algorithm is form a work published in 1993. Is there any more recent work/algorithm? How about those works mentioned in Section 1.1?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studied the bandit problem with controlled restarts. It proposed a few algorithms and derived the corresponding upper bounds on the reward rate. It also evaluated the algorithms with numerical experiments.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to 'Weaknesses' in the part of *Strengths and Weaknesses*.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "OFrh32JvJD", "reviewer": "Reviewer_GTM7", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThis is a well written paper, with well presented and sound theoretical results. The formulation is novel and allows for several benefits, like being versatile for diverse domains with a unified frameworkk. It also promises robustness to unknown statistics, which increases practical utility, unlike Cayci et al. (2019). Exploiting right-censored feedback also reduces regret dependence on $L$, a novel contribution over BwK.\n\nWeaknesses\n- Discretization: UCB-RC’s fixed $\\delta$ may not adapt to local reward rate variations potentially leading to sub-optimal performance. Dynamic discretization could be beneficial.\n- The assumption of deterministic and known resetting costs raise the question of how realistic it is. Stochastic or unknown costs would be common in practice - how would this affect the results?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work presents an extension to the multi-armed bandit (MAB) problem by integrating controlled restarts within a budgeted framework. Unlike traditional MAB settings that focus solely on arm selection to maximize rewards over a fixed number of rounds, this work considers a continuous-time scenario where each arm pull incurs a random completion time and yields a random, possibly correlated reward, both subject to a strict time budget $\\tau$. The decision-maker can interrupt an ongoing task, forgoing its potential reward, to restart with a potentially more rewarding arm, incurring a deterministic resetting cost $C_k(t)$. The work is motivated with real-world applications such as algorithm portfolios for SAT solvers, task scheduling in distributed servers , and hyperparameter tuning in neural network training, where tasks exhibit stochastic durations and restarts can optimize performance. The authors propose three algorithms: UCB-RM and UCB-RB for finite restart time sets, achieving $O(\\log \\tau)$ regret, and UCB-RC for continuous restart time sets, achieving $O(\\sqrt{\\tau \\log \\tau})$ regret. These algorithms operate without prior knowledge of system statistics, leveraging right-censored feedback to inform decisions. Theoretical guarantees are substantiated with empirical evaluations with superior performance over baseline restart strategies like the Luby scheme.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weaknesses, a brief justification would be appreciated.\n\nGenerally the theoretical framework for this work is robust, and the results well presented and well justified.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2aspHrVktj", "reviewer": "Reviewer_wmWz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The work studies an interesting problem.\n2. Both theoretical results and experimental results are provided.\n\nWeaknesses:\n1. The major weakness of this work is the readability. It is very hard to read this work and understand the main results of this work.\n\n1.1 What do you mean by \"continuous-time\" bandit? From you regret definition in Eqn. (5), (6), (7), it seems that you are still consider discrete time for bandit pulling. Then what is the difference if the delay is 5 vs 5.5?\n\n1.2 I do not understand Eqn. (5). Why is that the number of pulls under policy $\\pi$? I understand that $U_{I_i,i}(\\nu_i)$ is the total time for the $i$th pull? Is $i$th pull happening at time $i$?\n\n1.3 The motivating examples should be moved to the introduction. I am very confused about the intuition and use case of your method at the beginning, which definitely dampers my reading.\n\n1.4 Why do you need the bounded forth moment? In bandit literature, a random variable is heavy-tailed if it has finite $1+\\epsilon$ moment where $\\epsilon \\in (0,1]$.\n\n1.5 What is the assumption on the resetting time function? Why is there a resetting time intuitively?\n\n1.6 More explanation on the reward rate will be appreciated.\n\n2. Lower bounds should at least be discussed.\n3. Some related literature should be discussed in the related work section.\n4. Running time is recommended to be reported in the paper to showcase the efficiency of your method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces a continuous-time bandit framework with controlled restarts, addressing the challenge of balancing task completion time and reward optimization under uncertainty. It proposes near-optimal offline policies with provable guarantees and efficient online learning algorithms that leverage problem structure, outperforming the Luby restart strategy without requiring prior information. Theoretical results and experimental results are provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Besides the weaknesses in the above section for more clear presentation, the authors should cite more bandit work that is related with your problem setting and techniques.\n\n1. This paper should cite some work on delayed feedback bandits, which is a hot topic and similar with the setting in this work. For delayed feedback bandit, the reward will not be immediately revealed to the agent when it is pulled, but it will be observed after some stochastic delay. Seminal works on this problem are \n\na. Linear bandits with stochastic delayed feedback, Vernade et al. ICML 2020;\n\nb. Online learning with delayed feedback, Joulani et al. ICML 2013;\n\nc. Delay and cooperation in nonstochastic bandit, Cesa-Bianchi et al. JMLR.\n\n2. This paper should more work using restarting technique to handle the non-stochastic environment for the stochastic bandit problem. The restarting method in bandit community mostly refers to a specific technique that can handle non-stochastic bandit problems where the rewards are not sampled from a fixed distribution all the time. Seminal works using this restarting trick are:\n\na. A simple approach for non-stationary linear bandits, Zhao et al. AISTATS 2020;\n\nb. Online continuous hyperparameter optimization for generalized linear contextual bandits, Kang et al. TMLR;\n\nc. Learning to optimize under non-stationarity, Cheung et al. AISTATS 2019.\n\nPlease have a discussion on the related works mentioned in above 1, 2 in the revision (related works).", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)