Robust Learning for Congestion-Aware Routing

t e ) + η t e , where x t e is the flow on edge e at time t , fe is the congestion function , and ηte is a noise sample drawn from an unknown distribution . The algorithm observes cte , and can use this observation in future routing decisions . The routing requests are supplied adversarially . We present an algorithm with cumulative regret Õ ( |E|t2/3 ) , where the regret on each time step is defined as the difference between the total cost incurred by our chosen path and the minimum cost among all valid paths . Our algorithm has space complexity O ( |E|t1/3 ) and time complexity O ( |E| log t ) . We also validate our algorithm empirically using graphs from New York City road networks . 1 INTRODUCTION . Modern navigation applications such as Google Maps and Apple Maps are critical tools in large scale mobility solutions that route billions of users from their source to their destination . In order to be effective , the application should be able to accurately estimate the time required to traverse each road segment ( edge ) along the user route in the road network ( graph ) : we call this the cost of the edge . In general , the cost of an edge also depends on the current traffic level ( the flow ) on the edge . Furthermore , costs may scale differently on different edges : a highway can tolerate more traffic than a small residential street . We model this using congestion functions that map traffic flows to edge costs . Usually , cost information is not readily available to the routing engine and can only be inferred indirectly . For example , this can be done using location pings from vehicles , or with loop detectors that record when vehicles cross a particular marker . All realistic methods of measuring the cost of an edge ( such as those above ) require the presence of a vehicle that reports this information back to the routing platform . In this paper , we assume that whenever a vehicle traverses an edge , we observe the time spent on the edge . We can then use this observation in future routing decisions . This induces a natural exploration/exploitation trade-off : we may wish to send vehicles on underexplored routes , even if those routes currently seem suboptimal . In this paper , we propose a learning model for congestion-aware routing , and present an algorithm which seeks to minimize the total driving time across all vehicles . Our algorithm applies to arbitrary networks and arbitrary ( Lipschitz-continuous ) congestion functions , even when observations are noisy . The algorithm is also robust to changes in traffic conditions in a strong sense : we show that even when request endpoints are chosen adversarially and even when traffic congestion on the edges is updated adversarially between requests , our algorithm learns an optimal routing policy . 1.1 MODEL . Consider a directed graph ( V , E ) . Each edge e has a deterministic and fixed ( but unknown ) congestion function fe : R≥0 → R≥0 . We assume each fe is L-Lipschitz continuous and nondecreasing , with L known . For simplicity , we also assume that fe ( 0 ) = 0 for all e ∈ E ( if not , we can simply translate and extend the function appropriately ) . We consider an infinite horizon of time steps , starting from t = 0 . At each time t , a new car arrives . An adversary tells us the current amount of flow on each edge , and the source and destination of the new car . Let xte be the flow on edge e at time t and let Pt be the set of paths between the source and destination for the time t arrival . Let xtmax = maxe , t x t e. We must choose how to route the car , i.e. , we must select a path pt ∈ Pt . Based on our choice of pt , we incur a cost based on the flow and the congestion functions on the edges in our chosen path : ct = ∑ e∈pt fe ( x t e ) For each e ∈ pt , we observe cte = fe ( xte ) + ηte , where ηte is a random variable with expectation 0 . The distribution of ηte is unknown , and can vary between edges and time steps . The distributions can be correlated across edges , as long as for a given edge , all the individual samples ( i.e. , η1e , η 2 e , . . . ) are independent . We assume that there exists β such that ηte ∈ [ −β/2 , β/2 ] for all edges e and times t , and that β ( or an upper bound on β ) is known . The optimal cost at time t is c∗t = min p∈Pt ∑ e∈p fe ( x t e ) so the regret of our algorithm over the first t time steps is Rt = t∑ r=1 E [ ct − c∗t ] where the expectation is over the randomness in the noise samples . Note also that we do not include the noise we observe in the objective function , since the noise has expectation 0 . Any algorithm with sublinear regret can be said to learn an optimal routing policy : if Rt = o ( t ) , then E [ ct− c∗t ] must shrink as t goes to infinity , i.e. , the difference between our algorithm ’ s cost and the optimal cost must go to 0 on average.1 In this paper , we give an algorithm with regret Õ ( t2/3 ) . 1.2 OUR CONTRIBUTION . Our main result is the following theorem : Theorem 1.1 . The expected regret of Algorithm 1 after t time steps is Rt = O ( t2/3 · |E| log xtmax ( β √ log t+ xtmaxL ) ) The space complexity and time complexity on time step t areO ( |E|t1/3 log xtmax ) andO ( |E| ( log t+ log log xtmax ) + SP ( |E| , |V | ) ) , respectively . Here SP ( |E| , |V | ) denotes the time complexity of computing the shortest path between two vertices in a graph with nonnegative weights . For example , this can be done by Dijkstra ’ s algorithm in time O ( |E|+ V log V ) . We also validate our algorithm ’ s performance using graphs from New York City road networks . 1.3 RELATED WORK . Comparison with multi-armed bandits . Perhaps the simplest model of exploration vs exploitation is the classical multi-armed bandit ( MAB ) problem ( Slivkins , 2019 ) . A MAB instance consists of n arms , each with an unknown but fixed distribution . At each time step , the algorithm selects an arm and observes a reward drawn randomly from that arm ’ s distribution . Several algorithms obtaining regret Õ ( √ t ) are known , and this is also known to be the best possible , up to logarithmic factors . 1Note that E [ ct − c∗t ] may not monotonically decrease : a sublinear regret algorithm can still incur large errors sporadically , as long as the the frequency of the large errors goes to 0 . Our routing model generalizes the MAB problem . In particular , when the graph consists of n parallel edges , the congestion functions are constant , and each edge has a single fixed noise distribution , our model reduces to the MAB problem : each edge is an arm , and the reward distribution for each edge is ( the negative of ) the congestion constant plus the noise distribution . Our model can be thought of as extending the MAB problem to the case where ( 1 ) each reward distribution has a parameter ( in our case , the flow ) , and ( 2 ) the arms are edges in an arbitrary graph and their usage is constrained according to paths in the graph . We believe that routing is the most natural application of this model , but there could be other applications as well . Since our problem is strictly harder than the MAB problem , we know that regret o ( √ t ) is impossible . We conjecture that even regret Θ ( √ t ) is impossible for our problem , but leave this as an open question . Comparison with other routing work . Our work is also related to the body of literature on shortest paths under uncertainty , in which it is typically assumed that edge lengths follow known independent distributions . A canonical problem is to find an s–t path whose length is below a threshold L with highest probability . When the edge length distributions are Gaussian , Nikolova et al . ( 2006 ) present a quasi-polynomial time algorithm for this problem via connections to quasi-convex maximization . A related probing problem is the Canadian Traveler Problem ( Nikolova & Karger , 2008 ; Papadimitriou & Yannakakis , 1991 ) , where the length of an edge is revealed when we reach one of its end-points and the goal is to minimize the expected cost . There are no efficient algorithms known for this problem , except under special assumptions such as no backtracking ( Bnaya et al. , 2009 ) . Awerbuch & Kleinberg ( 2004 ) study a conceptually similar adaptive routing problem . In their version , there is no noise in the observations , but the algorithm observes only the total cost of the path , and not the cost of each individual edge . Their model also does not have the same notion of a parametrized congestion function . Although their model is technically distinct from ours , it is noteworthy that they also obtain a regret bound of Õ ( t2/3 ) . A number of subsequent works have also sought to apply bandit algorithms to shortest path selection problems in the domain of network signal routing ( Chen & Ji , 2005 ; György et al. , 2007 ; Liu & Zhao , 2012 ; Zou et al. , 2014 ; Talebi et al. , 2018 ) . Moreover , there is a long line of other works broadly dealing with regret minimization in Internet routing and congestion control settings ( Dong et al. , 2015 ; 2018 ; Jiang et al. , 2016 ; 2017 ; Talebi et al. , 2018 ) . Similar works apply bandit algorithms to shortest path selection have also been applied to traffic route planning ( Chorus , 2010 ; de Oliveira Ramos et al. , 2017 ) . However , these works differ from ours in that they do not consider the more general congestion functions that we consider in our model . A more recent work ( Zhou et al. , 2019 ) considers a similar routing problem , labeled the Multi-Armed Bandit On-Time Arrival Problem . In this setting , regret is quantified in terms of ontime arrival reliability , and the arms are joint route and departure times . Kveton et al . ( 2014 ) also study ( among other applications ) a routing setting in the context of matroid bandits , but this work also assumes static edge costs and does not factor in congestion . Routing in the presence of congestion functions has been studied in the context of selfish routing Roughgarden & Tardos ( 2002 ) and in the atomic congestion games setting Awerbuch et al . ( 2005 ) ; Christodoulou & Koutsoupias ( 2005 ) . The difference from our setting is that the congestion games model and selfish routing models assume drivers have full information about the costs in the network and route themselves on an optimal path . In our setting costs are learned as time progresses and there is a centralized routing control as given by navigation applications .

Summary: The paper uses the bandit learning framework to study the online learning problem for routing in a city network . After each routing decision, the learning agent observes the actual delay on each edge, which is given by the congestion function on the given flow plus a random noise, and the reward is the total delay on all edges. The paper proposes a learning algorithm similar to the UCB approach, provide the regret bound result, and conduct simulations on the New York City network to verify performance of the algorithm.