Conditional Expectation based Value Decomposition for Scalable On-Demand Ride Pooling

1 INTRODUCTION . Taxi/car on Demand ( ToD ) services ( e.g. , UberX , Lyft , Grab ) not only provide a comfortable means of transport for customers , but also are good for the environment by enabling sharing of vehicles over time ( while being used to serve one request at any one point in time ) . A further improvement of ToD is on-demand ride pooling ( e.g. , UberPool , LyftLine , GrabShare etc . ) , where vehicles are shared not only over time but also in space ( on the taxi/car ) . On-demand ride pooling reduces the number of vehicles required , thereby reducing emissions and traffic congestion compared to Taxi/car onDemand ( ToD ) services . This is achieved while providing benefits to all the stakeholders involved : ( a ) Individual passengers have reduced costs due to sharing of space ; ( b ) Drivers make more money per trip as multiple passengers ( or passenger groups ) are present ; ( c ) For the aggregation company more customer requests can be satisfied with the same number of vehicles . In this paper , we focus on this on-demand ride pooling problem at city scale , referred to as RidePool Matching Problem ( RMP ) [ Alonso-Mora et al . ( 2017 ) ; Bei & Zhang ( 2018 ) ; Lowalekar et al . ( 2019 ) ] . The goal in an RMP is to assign combinations of user requests to vehicles ( of arbitrary capacity ) online such that quality constraints ( e.g. , delay in reaching destination due to sharing is not more than 10 minutes ) and matching constraints ( one request can be assigned at most one vehicle , one vehicle must be assigned at most one request combination ) are satisfied while maximizing an overall objective ( e.g. , number of requests , revenue ) . Unlike the ToD problem that requires solving a bipartite matching problem between vehicles and customers , RMP requires effective matching on a tripartite graph of requests , trips ( combinations of requests ) and vehicles . This matching on tripartite graph significantly increases the complexity of solving RMP online , especially at city scale where there are hundreds or thousands of vehicles , hundreds of requests arriving every minute and request combinations have to be computed for each vehicle . Due to this complexity and the need to make decisions online , most existing work related to solving RMP has focused on computing best greedy assignments [ Ma et al . ( 2013 ) ; Tong et al . ( 2018 ) ; Huang et al . ( 2014 ) ; Lowalekar et al . ( 2019 ) ; Alonso-Mora et al . ( 2017 ) ] . While these scale well , they are myopic and , as a result , do not consider the impact of a given assignment on future assignments . The closest works of relevance to this paper are by Shah et al . [ Shah et al . ( 2020 ) ] and Lowalekar et al . [ Lowalekar et al . ( 2021 ) ] . We specifically focus on the work by Shah et al. , as it has the best performance , while being scalable . That work considers future impact of current assignment from an individual agents ’ perspective without sacrificing on scalability ( to city scale ) . However , a key limitation of that work is that they do not consider the impact of other agents ( vehicles ) actions on an agents ’ ( vehicle ) future impact , which as we demonstrate in our experiments can have a major effect ( primarily because vehicles are competing for the common demand ) . To that end , we develop a conditional expectation based value decomposition approach that not only considers future impact of current assignments but also of other agents state and actions through the use of conditional probabilities and tighter estimates of individual impact . Due to these conditional probability based tighter estimates of individual value functions , we can scale the work by Guestrin et al . [ Guestrin & Parr ( 2002 ) ] and Li et al . [ Li & Kochenderfer ( 2021 ) ] to solve problems with no explicit coordination graphs and hundreds/thousands of homogeneous agents . Unlike value decomposition approaches [ Rashid & Whiteson ( 2018 ) ; Sunehag & Graepel ( 2018 ) ] developed for solving cooperative Multi-Agent Reinforcement Learning ( MARL ) with tens of agents and under centralized training and decentralized execution set up , we focus on problems with hundreds or thousands of agents with centralized training and centralized execution ( e.g. , Uber , Lyft , Grab ) . In this application domain of taxi on demand services , where improving 0.5 % -1 % is a major achievement [ Lin et al . ( 2018 ) ] , we demonstrate that our approach easily outperforms the existing best approach , NeurADP [ Shah et al . ( 2020 ) ] by at least 3.8 % and up to 9.76 % on a wide variety of settings for the benchmark real world taxi dataset [ NYYellowTaxi ( 2016 ) ] . 2 BACKGROUND . In this section , we formally describe the RMP problem and also provide details of an existing approach for on-demand ride pooling called NeurADP , which we improve over . Ride-pool Matching Problem ( RMP ) : We consider a fleet of vehicles/resources R with random initial locations , travelling on a predefined road network G with intersections : L as nodes , road segments : E as edges and weights on edges indicate the travel time on the road segment . Passengers that want to travel from one location to another send requests to a central entity that collects these requests over a time-window called the decision epoch ∆ . The goal of the RMP is to match these collected requests U t to empty or partially filled vehicles that can serve them such that an objective J is maximised subject to constraints on the delay D. We upperbound D and consider the objective J to be the number of requests served . Thus , RMP is defined using the tuple [ G , U , R , D , ∆ , J ] 1 . Please refer Appendix A.1 for a detailed description . Delay constraints : D consider two delays , { τ , 2τ } . τ denotes the maximum allowed pick-up delay which is the difference between the arrival time of a request and the time at which a vehicle picks the user up . 2τ denotes the maximum allowed detour delay which is the difference between the time at which the user arrived at their destination in a shared cab and the time at which they would have arrived if they had taken a single-passenger cab . Neural Approximate Dynamic Programming ( NeurADP ) for Solving RMP : Figure 1 provides the overall approach . In this paper , there are two NeurADP [ Shah et al . ( 2020 ) ] contributions of relevance : FV : To estimate Future Value of current actions , a method for solving the underlying Approximate Dynamic Program ( ADP ) [ Powell ( 2007 ) ] by considering neural network representations of value functions . 1Everywhere in the paper [ , ] is used as the concatenation operator DJV : To ensure scalability , Decomposing the Joint Value function into individual vehicle value functions by extending on the work of Russell et al . [ Russell & Zimdars ( 2003 ) ] . Future Value ( FV ) : ADP is similar to a Markov Decision Problem ( MDP ) with the key difference that the transition uncertainty is extrinsic to the system and not dependent on the action . The ADP problem for RMP is formulated using the tuple 〈S , A , ξ , T , J 〉 , where : S : The state of the system is represented as st = ( rt , ut ) where rt is the state of all vehicles and ut contains all the requests waiting to be served . The state is obtained in Step A of Figure 1 . A : At each time step there are a large number of requests arriving to the taxi service provider , however for an individual vehicle only a small number of such requests are reachable . The feasible set of request combinations for each vehicle i at time t , F it is computed in Step B of Figure 1 : F it = { f i|f i ∈ ∪c i c′=1 [ U ] c ′ , PickUpDelay ( f i , i ) ≤ τ , DetourDelay ( f i , i ) ≤ 2τ } ( 1 ) ai , ft is the decision variable that indicates whether vehicle i takes action f ( a combination of requests ) at a decision epoch t. Joint actions across vehicles have to satisfy matching constraints : ( i ) each vehicle , i can only be assigned at most one request combination , f ; ( ii ) at most one vehicle , i can be assigned to a request j ; and ( iii ) a vehicle , i can be either assigned or not assigned to a request combination.∑ f∈Fit ai , ft = 1 : : : ∀i ∈ R ∑ i∈R ∑ f∈Fit ; j∈f ai , ft ≤ 1 : : : ∀j ∈ Ut a i , f t ∈ { 0 , 1 } : : : ∀i , f ( 2 ) ξ : denotes the exogenous information – the source of randomness in the system . This would correspond to the user requests or demand . ξt denotes the exogenous information at time t. T : denotes the transitions of system state . In an ADP , the system evolution happens as ( s0 , a0 , s a 0 , ξ1 , s1 , a1 , s a 1 , · · · , st , at , sat , · · · ) , where st denotes the pre-decision state at decision epoch t and sat denotes the post-decision state [ Powell ( 2007 ) ] . The transition from state st to st+1 depends on the action vector at and the exogenous information ξt+1 . Therefore , st+1 = T ( st , at , ξt+1 ) ; s a t = T a ( st , at ) ; st+1 = T ξ ( sat , ξt+1 ) It should be noted that T a ( . , . ) is deterministic as uncertainty is extrinsic to the system . J : denotes the reward function and in RMP , this will be the revenue from a trip . Let V ( st ) denotes the value of being in state st at decision epoch t , then using Bellman equation : V ( st ) = max at∈At ( J ( st , at ) + γE [ V ( st+1 ) |st , at , ξt+1 ] ) ( 3 ) where γ is the discount factor . Using post-decision state , this expression breaks down nicely : V ( st ) = max at∈At ( J ( st , at ) + γV a ( sat ) ) ; V a ( sat ) = E [ V ( st+1 ) |sat , ξt+1 ] ( 4 ) The advantage of this two step value estimation is that the maximization problem in Equation 4 can be solved using a Integer Linear Program ( ILP ) with matching constraints indicated in expression 2 . Step D of Figure 1 ) provides this aspect of the overall algorithm . The value function approximation around post-decision state , V a ( sat ) is a neural network and is updated ( Step E of Figure 1 ) by stepping forward through time using sample realizations of exogenous information ( i.e . demand observed in data ) . However , as we describe next , maintaining a joint value function is not scalable and hence we decompose and maintain individual value functions . Decomposing Joint Value ( DJV ) : Non-linear value functions , unlike their linear counterparts , can not be directly integrated into the ILP mentioned above . One way to incorporate them is to evaluate the value function for all possible post-decision states and then add these values as constants . However , the number of post-decision states is exponential in the number of resources/vehicles . [ Shah et al . ( 2020 ) ] introduced a two-step decomposition of the joint value function that converts it into a linear combination over individual value functions associated with each vehicle . In the first step , following [ Russell & Zimdars ( 2003 ) ] , the joint value function is written as the sum over individual value functions : V ( sat ) = ∑ i V i ( sat ) . In the second step , the individual vehicles ’ value functions are approximated . They assumed that the long-term expected reward of a given vehicle is not significantly affected by the specific actions another vehicle makes in the current decision epoch and thereby completely neglect the impact of the actions taken by other vehicles at the current time step . Thus they model the value function using the pre-decision , rather than post-decision , state of other vehicles which gives : V i ( sat ) = V i ( [ si , at , s -i , a t ] ) ≈ V i ( [ s i , a t , s -i t ] ) where -i refers to all vehicles except vehicle i . This allows NeurADP to get around the combinatorial explosion of the post-decision state of all vehicles . NeurADP thus has the joint value function : V ( sat ) = ∑ i V i ( [ si , at , s -i t ] ) . They then evaluate these individual V i values ( Step C of Figure 1 ) for all possible si , at ( from the individual value neural network ) and then integrate the overall value function into the ILP as a linear function over these individual values . This reduces the number of evaluations of the non-linear value function from exponential to linear in the number of vehicles .

Summary: This paper studies an RPM problem (ride-pool matching problem) for on-demand transportation services. This problem is recently studied in various papers, but it is hard to choose a good matching by just using a bipartite graph matching due to the future demands, and it is an online decision-making problem.  A recent breakthrough, NeurADP by [Shah et al. 2020], has shown a good performance, but the proposed approach in the paper, CEVD, achieved much more performance gain (reported as 3.8%-9.76%), which has a significant impact on the ToD service. An essential technique of the proposed CEVD is considering the effect of other agents (i.e., other vehicles) when estimating the value of actions.