Making the most of your day: online learning for optimal allocation of time

We study online learning for optimal allocation when the resource to be allocated is time . An agent receives task proposals sequentially according to a Poisson process and can either accept or reject a proposed task . If she accepts the proposal , she is busy for the duration of the task and obtains a reward that depends on the task duration . If she rejects it , she remains on hold until a new task proposal arrives . We study the regret incurred by the agent , first when she knows her reward function but does not know the distribution of the task duration , and then when she does not know her reward function , either . This natural setting bears similarities with contextual ( one-armed ) bandits , but with the crucial difference that the normalized reward associated to a context depends on the whole distribution of contexts . 1 Introduction . Motivation . A driver filling her shift with rides , a landlord renting an estate short-term , an independent deliveryman , a single server that can make computations online , a communication system receiving a large number of calls , etc . all face the same trade-off . There is a unique resource that can be allocated to some tasks/clients for some duration . The main constraint is that , once it is allocated , the resource becomes unavailable for the whole duration . As a consequence , if a “ better ” request arrived during this time , it could not be accepted and would be lost . Allocating the resource for some duration has some cost but generates some rewards – possibly both unknown and random . For instance , an estate must be cleaned up after each rental , thus generating some fixed costs ; on the other hand , guests might break something , which explains why these costs are both random and unknown beforehand . Similarly , the relevance of a call is unknown beforehand . Concerning duration , the shorter the request the better ( if the net reward is the same ) . Indeed , the resource could be allocated twice in the same amount of time . The ideal request would therefore be of short duration and large reward ; this maximizes the revenue per time . A possible policy could be to wait for this kind of request , declining the other ones ( too long and/or less profitable ) . On the other hand , such a request could be very rare . So it might be more rewarding in the long run to accept any request , at the risk of “ missing ” the ideal one . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Some clear trade-offs arise . The first one is between a greedy policy that accepts only the highest profitable requests – at the risk of staying idle quite often – and a safe policy that accepts every request – but unfortunately also the non-profitable ones . The second trade-off concerns the learning phase ; indeed , at first and because of the randomness , the actual net reward of a request is unknown and must be learned on the fly . The safe policy will gather a lot of information ( possibly at a high cost ) while the greedy one might lose some possible valuable information for the long run ( in trying to optimize the short term revenue ) . We adopt the perspective of an agent seeking to optimize her earned income for some large duration . The agent receives task proposals sequentially , following a Poisson process . When a task is proposed , the agent observes its expected duration and can then either accept or reject it . If she accepts it , she can not receive any new proposals for the whole duration of the task . At the end of the task she observes her reward , which is a function of the duration of the task . If , on the contrary , she rejects the task , she remains on hold until she receives a new task proposal . The agent ’ s policies are evaluated in terms of their expected regret , which is the difference between the cumulative rewards obtained until T under the optimal policy and under the implemented agent policy ( as usual the total length could also be random or unknown ( Degenne and Perchet , 2016 ) ) . In this setting , the “ optimal ” policy is within the class of policies that accept – or not – tasks whose length belongs to some given acceptance set ( say , larger than some threshold , or in some specific Borel subset , depending on the regularity of the reward function ) . Organization and main contributions . In Section 2 , we formally introduce the model and the problem faced by an oracle who knows the distribution of task durations as well as the reward function ( quite importantly , we emphasize again that the oracle policy must be independent of the realized rewards ) . Using continuous-time dynamic programming principles , we construct a quasi-optimal policy in terms of accepted and rejected tasks : this translates into a single optimal threshold for the ratio of the reward to the duration , called the profitability function . Any task with a profitability above this threshold is accepted , and the other ones are declined . As a benchmark , we first assume in Section 3 that the agent knows the reward function r ( · ) , but ignores the distribution of task durations . The introduced techniques can be generalized , in the following sections , to further incorporate estimations of r ( · ) . In that case , our base algorithm has a regret scaling as O ( √ T ) ; obviously , this can not be improved without additional assumptions , ensuring minimax optimality . In Section 4 , the reward function is not known to the agent anymore and the reward realizations are assumed to be noisy . To get non-trivial estimation rates , regularity – i.e. , ( L , β ) -Hölder – of the reward function is assumed . Modifying the basic algorithm to incorporate non-parametric estimation of r yields a regret scaling as O ( T 1−η √ lnT ) where η = β/ ( 2β + 1 ) . As this is the standard error rate in classification ( Tsybakov , 2006 ) , minimax optimality ( up to log-term ) is achieved again . Finally , our different algorithms are empirically evaluated on simple toy examples in Section 5 . Due to space constraints , all the proofs are deferred to the Appendix . Related work . As data are gathered sequentially , our problem bears similarities with online learning and multi-armed bandit ( Bubeck and Cesa-Bianchi , 2012 ) . The main difference with multi-armed bandit or resource allocation problems ( Fontaine et al. , 2020 ) is that the agent ’ s only resource is her time , which has to be spent wisely and , most importantly , saving it actually has some unknown value . The problem of time allocation actually goes way back . In economic theory , Becker ’ s seminal paper ( Becker , 1965 ) evaluates the full costs of non-working activities as the sum of their market prices and the forgone value of the time used to enjoy the activities . Various empirical works have followed ( see , e.g. , Juster and Stafford , 1991 ; Chabris et al. , 2009 ) . The problem of online scheduling has been studied in several possible variations ( see , e.g. , Pruhs et al. , 2004 , for a survey ) . Various subsequent papers consider scheduling models that combines online and stochastic aspects ( see , e.g. , Megow et al. , 2006 ; Chou et al. , 2006 ; Vredeveld , 2012 ; Marbán et al. , 2012 ; Skutella et al. , 2016 ) . Yet these models are different as they aim at minimizing the makespan of all received jobs , while actions here consist in accepting/declining tasks . In online admission control , a controller receives requests and decides on the fly to accept/reject them . However , the problem of admission control is more intricate since a request is served through some path , chosen by the controller , in a graph . Even solving it offline thus remains a great challenge ( Wu and Bertsekas , 2001 ; Leguay et al. , 2016 ) . An online model of reward maximization over time budget has been proposed by Cayci et al . ( 2019 , 2020 ) , where the agent does not observe the duration of a task before accepting it . As a consequence , the optimal policy always chooses the same action , which is maximizing the profitability . Since the duration of a task is observed before taking the decision in our problem , the decision of the optimal policy depends on this duration , used as a covariate . Cayci et al . ( 2019 ) thus aim at determining the arm with the largest profitability , while our main goal is here to estimate the profitability threshold from which we start accepting the tasks . The considered problems and proposed solutions thus largely differ in these two settings . Our problem is strongly related to contextual multi-armed bandits , where each arm produces a noisy reward that depends on an observable context . Indeed , the agent faces a one arm contextual bandit problem ( Sarkar , 1991 ) , with the crucial difference that the normalized reward associated to a context depends on the whole distribution of contexts ( and not just the current context ) . The literature on contextual bandits , also known as bandits with covariates , actually goes back to Woodroofe ( 1979 ) and has seen extensive contributions in different settings ( see , e.g. , Yang and Zhu , 2002 ; Wang et al. , 2005 ; Rigollet and Zeevi , 2010 ; Goldenshluger and Zeevi , 2009 ; Perchet and Rigollet , 2013 ) . This problem is also related to bandits with knapsacks ( see Slivkins , 2019 , Chapter 10 ) introduced by Badanidiyuru et al . ( 2013 ) and even more specifically to contextual bandits with knapsacks ( Badanidiyuru et al. , 2014 ; Agrawal et al. , 2016 ) , where pulling an arm consumes a limited resource . Time is the resource of the agent here , while both time and resource are well separated quantities in bandits with knapsacks . Especially , bandits with knapsacks assume the existence of a null arm , which does not consume any resource . This ensures the feasibility of the linear program giving the optimal fixed strategy . Here , the null arm ( declining the task ) still consumes the waiting time before receiving the next task proposal . Bandits with knapsacks strategies are thus not adapted to our problem , which remains solvable thanks to a particular problem structure . The problem of online knapsacks is also worth mentioning ( Noga and Sarbua , 2005 ; Chakrabarty et al. , 2008 ; Han and Makino , 2009 ; Böckenhauer et al. , 2014 ) , where the resources consumed by each action are observed beforehand . This is less related to our work , as online knapsacks consider a competitive analysis , whereas we here aim at minimizing the regret with stochastic contexts/rewards . 2 Model and benchmark . 2.1 The problem : allocation of time . All the different notations used in the paper are summarized at the beginning of the Appendix to help the reader . We consider an agent who sequentially receives task proposals and decides whether to accept or decline them on the fly . The durations of the proposed tasks are assumed to be i.i.d . with an unknown law , and Xi denotes the duration of the i-th task . If this task is accepted , the agent earns some reward Yi with expectation r ( Xi ) . The profitability is then defined as the function x 7→ r ( x ) /x . We emphasize here that Yi is not observed before accepting ( or actually completing ) the i-th task and that the expected reward function r ( · ) is unknown to the agent at first . If task i is accepted , the agent can not accept any new proposals for the whole duration Xi . We assume in the following that the function r is bounded in [ E , D ] with E ≤ 0 ≤ D , and that the durations Xi are upper bounded by C. After completing the i-th task , or after declining it , the agent is on hold , waiting for a new proposal . We assume that idling times – denoted by Si – are also i.i.d. , following some exponential law of parameter λ . This parameter is supposed to be known beforehand as it has a small impact on the learning cost ( it can be quickly estimated , independently of the agent ’ s policy ) . An equivalent formulation of the task arrival process is that proposals follow a Poisson process ( with intensity λ ) and the agent does not observe task proposals while occupied . This is a mere consequence of the memoryless property of Poisson processes . The agent ’ s objective is to maximize the expected sum of rewards obtained by choosing an appropriate acceptance policy . Given the decisions ( ai ) i≥1 in { 0 , 1 } ( decline/accept ) , the total reward accumulated by the agent after the first n proposals is equal to ∑n i=1 Yiai and the required amount of time for this is Tn : = ∑n i=1 Si +Xiai . This amount of time is random and strongly depends on the policy . As a consequence , we consider that the agent optimizes the cumulative reward up to time T , so that the number of received tasks proposals is random and equal to θ : = min { n ∈ N | Tn > T } . Mathematically , a policy of the agent is a function π – taking as input a proposed task X , the time t and the history of observations Ht : = ( Xi , Yiai ) i : Ti < t – returning the decision a ∈ { 0 , 1 } ( decline/accept ) . In the following , the optimal policy refers to the strategy π maximizing the expected reward Uπ at time T , given by Uπ ( T ) = E [ θ∑ n=1 r ( Xn ) π ( Xn , tn , Htn ) ] , ( 2.1 ) where tn : = Sn + ∑n−1 i=1 Si +Xiai is the time at which the n-th task is proposed . We allow a slight boundary effect , i.e. , the ending time can slightly exceed T , because the last task may not end exactly at time T . A first alternative would be to compute the cumulative revenue over completed tasks only , and another alternative would be to attribute the relative amount of time before T spent on the last task . Either way , the boundary effect is of the order of a constant and has a negligible impact as T increases .

Summary: This paper studies a decision making problem of accepting or rejecting of tasks that arrive randomly (wrt to Poisson process) and take random (unknown) durations to complete. By accepting/denying a task, the controller tries to minimizing the expected regret over time. Under this setup, the paper proposed threshold-based decision algorithms first with known reward function then with bandit feedback.