Making the most of your day: online learning for optimal allocation of time

We study online learning for optimal allocation when the resource to be allocated is time . An agent receives task proposals sequentially according to a Poisson process and can either accept or reject a proposed task . If she accepts the proposal , she is busy for the duration of the task and obtains a reward that depends on the task duration . If she rejects it , she remains on hold until a new task proposal arrives . We study the regret incurred by the agent , first when she knows her reward function but does not know the distribution of the task duration , and then when she does not know her reward function , either . This natural setting bears similarities with contextual ( one-armed ) bandits , but with the crucial difference that the normalized reward associated to a context depends on the whole distribution of contexts . 1 Introduction . Motivation . A driver filling her shift with rides , a landlord renting an estate short-term , an independent deliveryman , a single server that can make computations online , a communication system receiving a large number of calls , etc . all face the same trade-off . There is a unique resource that can be allocated to some tasks/clients for some duration . The main constraint is that , once it is allocated , the resource becomes unavailable for the whole duration . As a consequence , if a “ better ” request arrived during this time , it could not be accepted and would be lost . Allocating the resource for some duration has some cost but generates some rewards – possibly both unknown and random . For instance , an estate must be cleaned up after each rental , thus generating some fixed costs ; on the other hand , guests might break something , which explains why these costs are both random and unknown beforehand . Similarly , the relevance of a call is unknown beforehand . Concerning duration , the shorter the request the better ( if the net reward is the same ) . Indeed , the resource could be allocated twice in the same amount of time . The ideal request would therefore be of short duration and large reward ; this maximizes the revenue per time . A possible policy could be to wait for this kind of request , declining the other ones ( too long and/or less profitable ) . On the other hand , such a request could be very rare . So it might be more rewarding in the long run to accept any request , at the risk of “ missing ” the ideal one . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Some clear trade-offs arise . The first one is between a greedy policy that accepts only the highest profitable requests – at the risk of staying idle quite often – and a safe policy that accepts every request – but unfortunately also the non-profitable ones . The second trade-off concerns the learning phase ; indeed , at first and because of the randomness , the actual net reward of a request is unknown and must be learned on the fly . The safe policy will gather a lot of information ( possibly at a high cost ) while the greedy one might lose some possible valuable information for the long run ( in trying to optimize the short term revenue ) . We adopt the perspective of an agent seeking to optimize her earned income for some large duration . The agent receives task proposals sequentially , following a Poisson process . When a task is proposed , the agent observes its expected duration and can then either accept or reject it . If she accepts it , she can not receive any new proposals for the whole duration of the task . At the end of the task she observes her reward , which is a function of the duration of the task . If , on the contrary , she rejects the task , she remains on hold until she receives a new task proposal . The agent ’ s policies are evaluated in terms of their expected regret , which is the difference between the cumulative rewards obtained until T under the optimal policy and under the implemented agent policy ( as usual the total length could also be random or unknown ( Degenne and Perchet , 2016 ) ) . In this setting , the “ optimal ” policy is within the class of policies that accept – or not – tasks whose length belongs to some given acceptance set ( say , larger than some threshold , or in some specific Borel subset , depending on the regularity of the reward function ) . Organization and main contributions . In Section 2 , we formally introduce the model and the problem faced by an oracle who knows the distribution of task durations as well as the reward function ( quite importantly , we emphasize again that the oracle policy must be independent of the realized rewards ) . Using continuous-time dynamic programming principles , we construct a quasi-optimal policy in terms of accepted and rejected tasks : this translates into a single optimal threshold for the ratio of the reward to the duration , called the profitability function . Any task with a profitability above this threshold is accepted , and the other ones are declined . As a benchmark , we first assume in Section 3 that the agent knows the reward function r ( · ) , but ignores the distribution of task durations . The introduced techniques can be generalized , in the following sections , to further incorporate estimations of r ( · ) . In that case , our base algorithm has a regret scaling as O ( √ T ) ; obviously , this can not be improved without additional assumptions , ensuring minimax optimality . In Section 4 , the reward function is not known to the agent anymore and the reward realizations are assumed to be noisy . To get non-trivial estimation rates , regularity – i.e. , ( L , β ) -Hölder – of the reward function is assumed . Modifying the basic algorithm to incorporate non-parametric estimation of r yields a regret scaling as O ( T 1−η √ lnT ) where η = β/ ( 2β + 1 ) . As this is the standard error rate in classification ( Tsybakov , 2006 ) , minimax optimality ( up to log-term ) is achieved again . Finally , our different algorithms are empirically evaluated on simple toy examples in Section 5 . Due to space constraints , all the proofs are deferred to the Appendix . Related work . As data are gathered sequentially , our problem bears similarities with online learning and multi-armed bandit ( Bubeck and Cesa-Bianchi , 2012 ) . The main difference with multi-armed bandit or resource allocation problems ( Fontaine et al. , 2020 ) is that the agent ’ s only resource is her time , which has to be spent wisely and , most importantly , saving it actually has some unknown value . The problem of time allocation actually goes way back . In economic theory , Becker ’ s seminal paper ( Becker , 1965 ) evaluates the full costs of non-working activities as the sum of their market prices and the forgone value of the time used to enjoy the activities . Various empirical works have followed ( see , e.g. , Juster and Stafford , 1991 ; Chabris et al. , 2009 ) . The problem of online scheduling has been studied in several possible variations ( see , e.g. , Pruhs et al. , 2004 , for a survey ) . Various subsequent papers consider scheduling models that combines online and stochastic aspects ( see , e.g. , Megow et al. , 2006 ; Chou et al. , 2006 ; Vredeveld , 2012 ; Marbán et al. , 2012 ; Skutella et al. , 2016 ) . Yet these models are different as they aim at minimizing the makespan of all received jobs , while actions here consist in accepting/declining tasks . In online admission control , a controller receives requests and decides on the fly to accept/reject them . However , the problem of admission control is more intricate since a request is served through some path , chosen by the controller , in a graph . Even solving it offline thus remains a great challenge ( Wu and Bertsekas , 2001 ; Leguay et al. , 2016 ) . An online model of reward maximization over time budget has been proposed by Cayci et al . ( 2019 , 2020 ) , where the agent does not observe the duration of a task before accepting it . As a consequence , the optimal policy always chooses the same action , which is maximizing the profitability . Since the duration of a task is observed before taking the decision in our problem , the decision of the optimal policy depends on this duration , used as a covariate . Cayci et al . ( 2019 ) thus aim at determining the arm with the largest profitability , while our main goal is here to estimate the profitability threshold from which we start accepting the tasks . The considered problems and proposed solutions thus largely differ in these two settings . Our problem is strongly related to contextual multi-armed bandits , where each arm produces a noisy reward that depends on an observable context . Indeed , the agent faces a one arm contextual bandit problem ( Sarkar , 1991 ) , with the crucial difference that the normalized reward associated to a context depends on the whole distribution of contexts ( and not just the current context ) . The literature on contextual bandits , also known as bandits with covariates , actually goes back to Woodroofe ( 1979 ) and has seen extensive contributions in different settings ( see , e.g. , Yang and Zhu , 2002 ; Wang et al. , 2005 ; Rigollet and Zeevi , 2010 ; Goldenshluger and Zeevi , 2009 ; Perchet and Rigollet , 2013 ) . This problem is also related to bandits with knapsacks ( see Slivkins , 2019 , Chapter 10 ) introduced by Badanidiyuru et al . ( 2013 ) and even more specifically to contextual bandits with knapsacks ( Badanidiyuru et al. , 2014 ; Agrawal et al. , 2016 ) , where pulling an arm consumes a limited resource . Time is the resource of the agent here , while both time and resource are well separated quantities in bandits with knapsacks . Especially , bandits with knapsacks assume the existence of a null arm , which does not consume any resource . This ensures the feasibility of the linear program giving the optimal fixed strategy . Here , the null arm ( declining the task ) still consumes the waiting time before receiving the next task proposal . Bandits with knapsacks strategies are thus not adapted to our problem , which remains solvable thanks to a particular problem structure . The problem of online knapsacks is also worth mentioning ( Noga and Sarbua , 2005 ; Chakrabarty et al. , 2008 ; Han and Makino , 2009 ; Böckenhauer et al. , 2014 ) , where the resources consumed by each action are observed beforehand . This is less related to our work , as online knapsacks consider a competitive analysis , whereas we here aim at minimizing the regret with stochastic contexts/rewards . 2 Model and benchmark . 2.1 The problem : allocation of time . All the different notations used in the paper are summarized at the beginning of the Appendix to help the reader . We consider an agent who sequentially receives task proposals and decides whether to accept or decline them on the fly . The durations of the proposed tasks are assumed to be i.i.d . with an unknown law , and Xi denotes the duration of the i-th task . If this task is accepted , the agent earns some reward Yi with expectation r ( Xi ) . The profitability is then defined as the function x 7→ r ( x ) /x . We emphasize here that Yi is not observed before accepting ( or actually completing ) the i-th task and that the expected reward function r ( · ) is unknown to the agent at first . If task i is accepted , the agent can not accept any new proposals for the whole duration Xi . We assume in the following that the function r is bounded in [ E , D ] with E ≤ 0 ≤ D , and that the durations Xi are upper bounded by C. After completing the i-th task , or after declining it , the agent is on hold , waiting for a new proposal . We assume that idling times – denoted by Si – are also i.i.d. , following some exponential law of parameter λ . This parameter is supposed to be known beforehand as it has a small impact on the learning cost ( it can be quickly estimated , independently of the agent ’ s policy ) . An equivalent formulation of the task arrival process is that proposals follow a Poisson process ( with intensity λ ) and the agent does not observe task proposals while occupied . This is a mere consequence of the memoryless property of Poisson processes . The agent ’ s objective is to maximize the expected sum of rewards obtained by choosing an appropriate acceptance policy . Given the decisions ( ai ) i≥1 in { 0 , 1 } ( decline/accept ) , the total reward accumulated by the agent after the first n proposals is equal to ∑n i=1 Yiai and the required amount of time for this is Tn : = ∑n i=1 Si +Xiai . This amount of time is random and strongly depends on the policy . As a consequence , we consider that the agent optimizes the cumulative reward up to time T , so that the number of received tasks proposals is random and equal to θ : = min { n ∈ N | Tn > T } . Mathematically , a policy of the agent is a function π – taking as input a proposed task X , the time t and the history of observations Ht : = ( Xi , Yiai ) i : Ti < t – returning the decision a ∈ { 0 , 1 } ( decline/accept ) . In the following , the optimal policy refers to the strategy π maximizing the expected reward Uπ at time T , given by Uπ ( T ) = E [ θ∑ n=1 r ( Xn ) π ( Xn , tn , Htn ) ] , ( 2.1 ) where tn : = Sn + ∑n−1 i=1 Si +Xiai is the time at which the n-th task is proposed . We allow a slight boundary effect , i.e. , the ending time can slightly exceed T , because the last task may not end exactly at time T . A first alternative would be to compute the cumulative revenue over completed tasks only , and another alternative would be to attribute the relative amount of time before T spent on the last task . Either way , the boundary effect is of the order of a constant and has a negligible impact as T increases .

Summary: The paper studies a setting where requests arrive according to a Poisson process and an agents must decide to process them or not. If she decides to process the request, she gets busy for an amount of time shown in the request (and cannot process or see the processing times of any requests arriving in this time period) and gets a reward which is a function of the processing time; if not, she remains idle and ready to process new requests. It is assumed that the arrival rate is known but the probability distribution of processing times is not.   The contributions of the paper are as follows:  1) A myopic algorithm with bounded reward difference with respect to the optimal one is derived: it is essentially a threshold policy where the agent accepts a request if the ratio of reward over processing time (profitability) is greater than an appropriately computed threshold.  2)Learning algorithms with regret (with respect to the myopic algorithm) sublinear with the time horizon are derived, in the cases where the reward function is known or not.    The algorithms seem to be based in estimating fast the optimal profitability threshold $c^*$ and, in the bandit case, on estimating the reward function via splitting the interval of possible processing times into bins (and estimating a reward for each bin).