Triple-BERT: Do We Really Need MARL for Order Dispatch on Ride-Sharing Platforms?
Authors: Zijian Zhao
Sen Li
Venue: ICLR 2026 Oral (2026)
Keywords: Reinforcement Learning
Order Dispatching
Ride Sharing
Primary area: reinforcement learning
TL;DR: This paper proposes a novel centralized reinforcement learning framework for large-scale order dispatching tasks in ride-sharing scenarios, achieving better cooperation among workers compared to previous multi-agent methods.

Abstract: On-demand ride-sharing platforms, such as Uber and Lyft, face the intricate real-time challenge of bundling and matching passengers—each with distinct origins and destinations—to available vehicles, all while navigating significant system uncertainties. Due to the extensive observation space arising from the large number of drivers and orders, order dispatching, though fundamentally a centralized task, is often addressed using Multi-Agent Reinforcement Learning (MARL). However, independent MARL methods fail to capture global information and exhibit poor cooperation among workers, while Centralized Training Decentralized Execution (CTDE) MARL methods suffer from the curse of dimensionality. To overcome these challenges, we propose Triple-BERT, a centralized  Single Agent Reinforcement Learning (MARL) method designed specifically for large-scale order dispatching on ride-sharing platforms. Built on a variant TD3, our approach addresses the vast action space through an action decomposition strategy that breaks down the joint action probability into individual driver action probabilities. To handle the extensive observation space, we introduce a novel BERT-based network, where parameter reuse mitigates parameter growth as the number of drivers and orders increases, and the attention mechanism effectively captures the complex relationships among the large pool of driver and orders. We validate our method  using a real-world ride-hailing dataset from Manhattan. Triple-BERT achieves approximately an 11.95% improvement over current state-of-the-art methods, with a 4.26% increase in served orders and a 22.25% reduction in pickup times. Our code, trained model parameters, and processed data are publicly available at https://github.com/RS2002/Triple-BERT .

Published as a conference paper at ICLR 2026
TRIPLE-BERT: Do WE REALLY NEED MARL FOR OR- DER DISPATCH ON RIDE- PLATFORMS?
Zijian Zhao', Sen Li
'The Hong Kong University of Science and Technology
The Hong Kong University of Science and Technology (Guangzhou)
ABSTRACT
On-demand ride-sharing platforms, such as Uber and Lyft, face the intricate real- time challenge of bundling and matching passengers-each with distinct origins and destinations-to available vehicles, all while navigating significant system uncertainties. Due to the extensive observation space arising from the large number of drivers and orders, order dispatching, though fundamentally a centralized task, is often addressed using Multi-Agent Reinforcement Learning (MARL). However, independent MARL methods fail to capture global information and exhibit poor cooperation among workers, while Centralized Training Decentralized Execution (CTDE) MARL methods suffer from the Curse of Dimensionality (CoD). To overcome these challenges, we propose Triple-BERR a centralized framework designed specifically for large-scale order dispatching on ride-sharing platforms based on Single Agent Reinforcement Learning (SARL). Built on a variant TD3, approach addresses the vast action space through action decomposition strategy that breaks down the joint action probability into individual driver action probabilities. To handle the extensive observation space, we introduce a novel BERT-based network, where parameter reuse mitigates parameter growth as the number of drivers and orders increases, and the attention mechanism effectively captures the complex relationships among the large pool of driver and orders. We validate our method using a real-world ride-hailing dataset from Manhattan. Triple-BERT achieves approximately an 1 11.95% improvement over current state-of- the-art methods, with a 4.26% increase served orders and 22.25% reduction pickup times. Our code, trained model parameters, and processed data are publicly available htthtts::/ /gattu. wttp://ww..coom///o///////////// BERT
INTRODUCTION
Ride-sharing platforms, such as Uber and Lyft, face the complex challenge of dynamically matching passengers with distinct origins and destinations to available vehicles in real time. This task must account for significant system uncertainties, including fluctuating demand, varying traffic conditions, and the availability of drivers. As the volume of concurrent ride requests increases, these platforms must efficiently allocate resources to minimize detours, reduce waiting times, and maximize customer satisfaction and platform revenue. However, the inherently large and dynamically changing action and observation spaces make this problem highly challenging for the operation of ride-sharing platforms.
Recently, Reinforeement Learning (RL) methods have shown great potential in addressing the order dispatching problem in ride-sharing platforms. Model-free RL, in particular, enables agents to autonomously learn optimal dispatching policies by interacting with the environment, without requiring complex system modeling. This approach allows platforms to optimize multiple objectives, including platform income, driver payments, and customer satisfaction. Despite these advantages, applying RL to large-scale order dispatching introduces significant challenges. The vast action and observation spaces, stemming from the large number of drivers and orders, make sufficient
*Corresponding Author: Sen Li (cesli@ust.hkk) Do We Really Need MARL for Order Dispatch on Ride-Sharing Platforms? In summary: No, because we developed SARI. method that achieves hetter global planning. However, yes, our method still requires MARI. for pre-training establish strong starting point for SART..

---

Published as a conference paper at ICLR 2026
exploration and efficient training difficult. Multi-Agent Reinforcement Learning (MARL) methods have been widely adopted to address these challenges by decomposing the problem into smaller subproblems for individual agents (drivers). Independent MARL methods, such as Independent Double DQN (IDDQN) I 29 53 and Independent SAC (ISAC) 64 are computationally efficient but fail to capture global information and exhibit limited cooperation among agents. Graph Neural Networks (GNNs) have been introduced to enable the network to capture neighboring information for each agent, alleviating this issue to certain extent (23] 57.. Meanwhile, Centralized Training with Decentralized Execution (CTDE) methods, such as QMIX 16 and Coordinated Policy Optimization (CoPO) (4, struggle with the Curse of Dimensionality (CoD) when applied to large-scale scenarios with thousands of agents, resulting in slow convergence and suboptimal performance. (Due to page limitations, we provide a detailed review of ride-sharing methods and cooperative MARL, approaches in Appendix [F]
To address these limitations, this paper proposes a centralized Single-Agent Reinforcement Learning (SARL) method, named Triple-BERT, tailored for large-scale order dispatching in ride-sharing platforms. Triple-BERT introduces an action decomposition method that simplifies the joint action probability into individual driver action probabilities, enabling each driver to make independent decisions while maintaining global coordination. The method leverages TD3 [2] for optimization, with modifications to the actor optimization process via policy gradient 446 to better suit the ride- sharing context. To handle the extensive observation space, we design a novel BERT-based [ neural network architecture. This network employs bi-directional self-attention to effectively capture complex relationships between drivers and orders, while its parameter reuse mechanism prevents parameter explosion as the number of drivers and orders increases. Additionally, compared to MARL, SARL faces unique challenge of sample scarcity, as the records of multiple agents are merged into a single training stream. To address this, we propose a two-stage training strategy, where feature extractors are pre-trained using MARL approach to learn general embedding capabilities, followed by centralized fine-tuning. The main contributions of this paper can be summarized as follows:
We introduce TriPle-BERT, which is the first centralized framework for large-scale order dispatching side-sharing platforms based a variant centralized TD3. This framework addresses the limitations of the observation space and the inefficiencies in cooperation among agents present in conventional MARL-based methods. To tackle the large action space inherent in the matching problem of order dispatching tasks, we propose an action decomposition method that breaks down the joint action probability into individual driver action probabilities. Additionally, we propose a two-stage training method to address the sample scarcity issue in SARL, where the feature extractors are first trained using MARL approach.
To support the proposed RL framework in a large observation space, we develop novel neural network architecture based on BERT. This design leverages self-attention mechanisms to effectively capture the relationships between drivers and orders. Furthermore, we incorporate a QK -tttttentoon module to reduce computational complexity from multiplication to addition in the order dispatching task, along with a positive normalization method to mitigate parameter redundancy issues.
We validate the proposed method in the ride sharing scenario, using a real-world dataset of ride- hailing trip records from Manhattan. Our method outperforms the MARL methods reported in previous works, demonstrating approximately a 1.95% improvement over current state-of-the-arn methods, with a 4.26% increase in served orders and reductions of about 22.25% in pickup time.
2 PROBLEM SETUP
In this paper, we address the order dispatching task within on-demand ride sharing platform. We consider a platform managing n drivers (hereafter referred to as workers). represented by the state W. = [W1.t, W2.t..... ). where wi,t denotes the state of worker at time t. At each time step, the platform processes set of orders, including newly arrived orders and any previously unassigned orders, denoted as O, = {o1,e: O2. +.. where m, is the total number of orders at time t. Since real-time performance is crucial in on-demand systems, the platform aims to bundle and assign orders in a way that minimizes delivery time while maximizing the number of served orders. Customers assumed to be impatient; if an order is not acknowledged within specified time frame, customers will decline it. Moreover, late deliveries beyond the scheduled time may result in customer complaints, potentially causing losses for the platform. The overall workflow is illustrated
2

---

Published as a conference paper at ICLR 2026
Update State
Decentralized IDDQN
Assignment
W
BiPartite
Q-Matrix
Matching
Worker Pool
Q
P
global Dranlse)
W
Centralized TD3
Order Pool
w
BiPartite
Action Log
Matching
Probability Matrix
Add Newly Arrived Orders
Delete
Overtime
probabiltt)
Orders
Figure I: Workflow: At each time step, the worker and order pools update their states based on the assignments made in the previous time step. Specifically, the order pool adds newly arrived orders and removes overdue ones. For IDDQN, the Q-value each worker-order pair is calculated, and ILP is applied maximize the global Q-value, For TD3, the probability of each worker-order pair is computed, followed by the application of ILP to maximize the global assignment probability.
in Fig. | and the Markov Decision Process (MDP) is formulated as < S, R. P.P. encompassing state, action, reward, and transition function, which will be detailed below:
(i) State: At timestep i, the state or observation can be represented S. = [W, Or). consisting of the states of workers and orders. For the order j to be assigned, the state Os,; includes the order's origin and destination, pickup time, and scheduled arrival time. For each worker the state wi, consists the onboard orders Hy, that are still unfinished, the current location, the residual capacity, and the estimated time when he/she will be available to accept new order. (Note that assume if a worker is en route to pick up a new order or if his/her capacity is full, he/she cannot serve a new order.) Specifically, H.,, is sequence of orders H;, [hi,1.t h4,2,1 hi,kk,,.th s.th. where k is the number of onboard orders for worker at time / and each order hi,k. contains the same information the orders to be assigned 0j.t-
(ii) Action: At each time f, the action can be represented as A, {a1.1, 02,1, d2,1.-.-, an,t). where each ag, an mn-dimensionall vector with
one element to indicating which order is assigned to worker i. The order dispatching task is particularly challenging due to two main factors: (i) the size of the action space keeps changing over time because the number of orders m, varies dynamically as new orders arrive and old orders are completed or canceled; (ii) the size of the action space is extremely large for real systems. For instance, considering n 1000 workers and m, = 10 orders, the action space can reach approximately 1030 (A detailed proof is provided in Appendix A This combination of an enormous action space and its continuously changing size significantly complicates sufficient exploration and stable network convergence for standard RL methods.
(iii) Reward Function: We split the reward function for each worker, meaning each worker will receive reward ri.++1 at time step 6, and the global reward is the sum of each worker's reward: Rit1 Ei Fi,+1- The reward ri,++ can be calculated according to the following function:
11 Bart Bap.. 194X. AsPiL 0
|a |ar,1 =0
Ti = R(s,,,
(1)
where 31 to As are non-negative weights representing the platform's valuation of each term, p and p,. represent the income from customers and the payout to workers, respectively. The variables Xi.t and Pı, represent the number of en-route orders that will exceed their scheduled time and the additional travel time of all en-route orders when the assigned order is added to the scheduled route of worker time I, respectively. This reward function is designed to comprehensively consider
3

---

Published as a conference paper at ICLR 2026
the interests of the platform, workers, and customers, mimicking the operation of sharing platform. It is important to emphasize that p.A and p.r are calculated based on the order distance and the additional travel distance for the worker, respectively. When calculating travel time, we will utilize the Traveling Salesman Problem (TSP) to optimize the worker's rout..
ride
(iv) Transition Function: In our system, the reward is deterministic given the current state and action. Therefore, the transition function is represented by P(Sr+1|SA A₁). In this study, the transition probabilities are not explicitly modeled; instead, they are inferred through the model-free RL.
METHODOLOGY
Valuel, Values
BERT
X
Lineer Laper
Critic MLP
Critic BERT
Layer Norm
Multi-head
Aetf-tntntion
by
Action Probabiitty
QK- Attention
MatMul
Rused Sequarece
Wile Gibbediing
Order Embedding
12-Norm
Actor QK- Attention
Query
Softflus
MIP
MLP
Actor BERT
MLP
w peo tronal smmbeeding
Order Encoder
Norm
ARL
Encoder
Ra BaiWWightin::
Adaptive
Laper
Norm
Norm
Pickedl Oncior Information
Information
Worker
Warkeer Information
Order Information
New Order Information
The network consists of three main components: the feature extractor, the actor sub-network, and the critic sub-network. First, a worker encoder and an order encoder are used to extract features from individual worker and order information, respectively. Then an Actor BERT model captures the relationships between them and a QK-Attention module calculates the selection probabilities for each worker-order pair. Finally, the fused features of the selected worker-order pairs are input into two separate Critic BERT models for further information extraction, and two Critic MIPs compute the Q-values, TD3 requires sequence (input to Critic-BERT) represents workers 1, 3, 6, and n selecting orders 2, 3, 4, and m, respectively.)
Figure 2:
Network Architecture:
critics. (In this figure, the fused
3.1 OVERVIEW
In this work, we aim to utilize centralized SARL to address the large-scale order dispatching task, with the goal of enabling the model fully leverage global information enhance cooperation among workers. To tackle the challenges of large action and observation spaces, we propose a novel network architecture, as illustrated in Fig. 2 This architecture employs the BERT model [8] to effectively extract the relationships between workers and orders using the self-attention mechanism. Additionally, an improved QK -aatttentoon [88 is implemented reduce the computational complexity associated with the order dispatching task. Furthermore, we introduce an action decomposition method that breaks down the choice probability of each action within the vast action space into individual action probabilities for each worker selecting each order. Finally, to address the data scarcity challenge in MARL, we propose a two-stage training method, as shown Fig. [ In the first stage, train the upstream layers of the network using the IDDQN approach, allowing them to develop general feature extraction capabilities. Subsequently, we train the entire neural network using centralized TD33 to realize better cooperation between workers.
4

---

Published as a conference paper at ICLR 2026
3.2 NETWORK ARCHITECTURE
The proposed network structure is shown as Fig. 2 which constists of three parts: encoders (embed the worker and order information common feature space), actor sub-network (a BERT to extract the relationship between different workers and orders and a QK-Attention to generate the utility/probability of each worker-order pair), and critic sub-network (two BERT taking output of actor BERT as input and output the Q-value respectively).
3.2. FEATURE EXTRACTORS
At each time step, the network takes the entire state S. = [W,, Or] as input. We consider this as a combination of two sequences: W and Oz. For each element wi,t and o we employ two distinct encoders, referred to as the "Worker Encoder" and the "Order Encoder", to embed them separately into a feature space of the same dimension, allowing them to be input into single BERT model.
Each worker state w.e consists two parts: an on-board order sequence and other non-sequence information. For the order sequence, bi-directional LSTM [17] is utilized to extract its features. This approach effectively encodes variable-lengee sequences into a uniform dimensional feature space, addressing the CoD associated with conventional MLP encoders, where the number of parameters increases with sequence length. For non-sequence information, an MLP is employed for feature extraction. Finally, the two features are combined into primary feature Wi,t. For the orders to be assigned Oj.e. an MLP is also used to extract the feature oz.t. Notably, the dimensions of Wi. and 0J,1 are identical, and their information is concatenated into sequence represented as S, = [ W2.t..... Wn 01 ;2,
Additionally, to facilitate network convergence and enhance the extraction of input features, incorporate a normalization layer and an Adaptive Re-whtttting Layer (ARL) 4 Given that different parts of the input may have varying magnitudes, which can impede model training, the normalization layer effectively addresses this issue. Furthermore, since different parts of the input carry different utilize the ARL to enable the model learn these variations, represented as: = x o Ω2 where x denotes the input, Q represents the weight vector, calculated by Ω = MLP(z), and indicates the element-wise product.
levels of importance,
3.2.2 ACTOR SUB-NETWORKS
The Actor sub-network consists of BERT X model for feature extraction and -Katttentoon module [68] for action decomposition and generation, which we will introduce in turn. In the feature extractors, we have already extracted the primary features from each worker and order state separately. To further explore the relationships between workers and orders, we utilize the BERT model, where the self-attention mechanism can effectively capture these relationships: S, Actor-BERT(S (5.). Specifically, due to the permutation invariance of our input sequence, we omit the positional embedding in BERT, ensuring that the order in S does not influence the encoding result. In contrast to conventional MARL methods like [23] 16), which encode each worker with its neighboring states to gain broader perspective, our Actor-BERT directly aggregates global worker information, facilitating more effective cooperative dispatching between workers.
[W1.e: W2, Wn,t: 0 02 .... om
In conventional order dispatching tasks, the typical approach address the dynamic action space (related to the number of orders) involves evaluating each worker-order pair separately and finding the optimal dispatching solution based on these evaluations. However, this approach has two significant shortcomings. First, it neglects the relationships between orders, which we address through the self-attention mechanism BERT, capturing not only the relationships between workers but also between orders and between orders and workers. Second, evaluating each worker-order pair is time-consuming and resource-intensive: F(W.,, Tz.r; 0r) E Ik=,, where is the network and 0r represents its parameters. The complexity can be represented O(F n m2), where F] denotes the complexity the neural network. To mitigate this issue, we employ a QK-attention module (68), represented as:
QK QK-Attention (T.. Ty.c) = f( . 0y) g(D,, O,)) F(W,,,,
(2)
where and g are two smaller networks, and o and 9, are their parameters. The intuition behind QK- attention is to use smaller networks approximate larger network, similar the motivation
5

---

Published as a conference paper at ICLR 2026
behind LoRA 20 In this way, the complexity of computing all worker-order pairs becomes O[ n + g m, + d- n m&), where [f] and lgl are the complexities of the two neural networks, d is their output dimension, and d n m, is the complexity of matrix multiplication. Here, d is very small, making d. n. my much smaller than the neural network computation complexity, i.e., d n. my < [f] ig < |F] Thus, have O[f [[F n- m.), indicating that the QK-attention successfully transforms the multiplication complexity of evaluating each worker-order pair into addition complexity.
[gl m, d m4) < O(F)) (n +m))) <
However, we observe a parameter redundancy issue in Equation 2 which can lead to potential instability during training. This redundancy arises because there are actually infinite solutions for and g. as f = a and g = is also valid solution for any non-zero real vector a. Inspired by Dueling DQN 58
we propose positive normalization method:
Softphus(g((,s500))
CiatenntionNNorm( Wy.e: 03,1) == f(warr 0,)
(3)
[Sooiplus(g(0..i 0g)ll
This normalization ensures that the elements in [Soofplussgtt,,,:1.. " are always non-negative, with an L2 norm of 1. Although this approach does not guarantee a unique solution, it significantly improves training stability, as demonstrated by our experimental results Section our task, the output of the QK-attention is matrix M, E Rr.m, representing the utility of each worker choosing each order, which will detailed in Section 3.3.2
3.2.3 CRITIC SUB-NETWORKS
The role of the critic is to evaluate the quality of actions, with the detailed action generation method We first define an action function A:
introduced in Section 3.3.2
A(wa,) = (103, 05.)) if no order is assigned to worker at time t
order j is assigned to worker i at time
(4)
where Wie and 0z, are the outputs of Actor-BERT, and (Wa,t, 0,.)) represents the combination of the two vectors into a single feature vector. We then construct a new sequence: Š; [A(w1,); A(.,,,,(u...) A(w2,)...... A(ws,))) Another BERT network, referred as "Critic-BERT" is used to further extract features from Š,, represented as Š, = Critic-BERT (Š:). A self-attention mecha- nism and a linear layer (collectively named Critic-MLP)
then utilized to estimate the Q-value from Š, (for detailed processing methods, refer to (6). Furthermore, as TD3 [12) requires two critics, we employ two distinct Critic-BERT and Critic-MLP networks. These share the input features from but process them separately.
3.3
TRAINING PROCESS
3.3.1 STAGE 1: DECENTRALIZED IDDQN TRAINING
In this stage, we aim to first train the feature-extracting capacity of the worker encoder and order encoder using substantial number of samples. To obtain sufficient samples, we view the dispatching problem as a multi-agent scenario, where at each time step, each agent can access its own record. We adopt the independent assumption that all agents share the same policy, allowing for the sharing of records between agents and leading to a large experience replay buffer.
Since our goal in this stage is not to train powerful model but rather enable the feature extractor to learn its general feature-extractin capabilities, we select the simplest yet efficient method for order dispatching, namely, the IDDQN. Each worker is treated as an independent agent with the state defined as 8;, [We,,, Or] at time t. We employ neural network to estimate the Q-value at each step as Q DON (5.,,.r,). where & represents the network parameters and T.. denotes the strategy.
To construct the network, utilize QK-attention to process the outputs of the worker en- coder and order encoders to estimate the Q-value for each worker-order pair, represented as Ec--AteentionNNomffee (1 (j.)) (denoted as yi.,,). Although the state space encompasses the entire

---

Published as a conference paper at ICLR 2026
order state from 01,1 Om,,t focus single order 0j.4 when computing the Q-value for choos- ing order j. This approach aligns with previous work such as [23 21], as the entire order state can be excessively large for simple network to learn (our Triple-BERT effectively addresses this issue) and many networks struggle to process variable dimensional inputs (with order amounts varying at each can compute Q-matrix Y2 IR where the element in the i-th row and j-th column, Vi.j.e. represents the Q-value of assigning order j to worker at time t. The core strategy of IDDQN is to maximize the global Q-value, expressed as Q(S,, Ad مك= Q(5 aia)) at each time step. To achieve this, we construct a bipartite graph where each worker and order is represented as a node. An arbitrary worker and order j are linked by an edge weighted by the Q-value of this worker selecting this order at the current time, i.e., Vi.j.t- We then utilize Integer Linear Programming (ILP) to solve this maximizing bipartite matching problem. (To avoid assigning orders to unavailable workers-those at full capacity on their way to pick up an assigned order-we set the Q-value of all actions for such workers in the Q-matrix Y, to - -c.) A detailed construction of the problem is provided in Appendix [B.I] For the training of IDDQN, it follows the same process of Due to page limitation, we detailed it in Appendix D.I
time step). Consequently,
previous work 23
Here, We want to make some explanations about the independent assumption. We acknowledge that the independent assumption can be unreliable, which indeed hinders the performance of conventional independent MARL-based methods. However, previous works (40 TO- (56] have shown the efficacy and simplicity of independent MARI. methods. As a result, even if the performance of independent MARL not satisfactory, it does provide a good starting point for our centralized SARL method. In our approach, the independent assumption is only utilized during the pre-training stage to warm up the model. After pre-trot pre-training our centralized SARL framework no longer relies on the independence i
assumption.
3.3.2 STAGE 2: CENTRALIZED TD3 TRAINING
In the standard AC framework, the process can be summarized as follows: an actor network generates actions based on the current state, represented as A, Actor(S,; 0A). while critic network evaluates these actions using Q = (r, Ae; 0c, =TA)) ). Here, AA and 6c are the parameters of the actor and critic networks, respectively, and m denotes the strategy of AC. During training, the critic network is updated using TD-error, similar to Q-learning, and the actor network is updated to maximize Q However, challenge mentioned in Section is that the action space is too large for the order dispatching scenario. Additionally, the actions in order dispatching are discrete, complicating optimization using TD3. To address these issues, we propose action decomposition method along with a policy gradient-styy style optimization method.
Before delving into the details, we denote both A and Oc with the parameters θ, in our network (Fig. 2 the actor and critic share the same architecture. The trained network parameters from Stage 1, ②b are part of O. Moreover, the policy of TD3 is represented To.
(i) Actor: In the standard AC framework for discrete action problems, the policy network generates probabilities for each action, from which actions are sampled. However, in the ride-sharing task, this approach encounters two significant challenges: (i) First, the action space is exceedingly large. As shown in Appendix A a typical scenario with 1,000 couriers and 10 orders can yield nearly 10000 possible actions. (ii) Second, because the orders vary at each step (including both order volume and content), it is impossible generate a fixed-dimension action probability vector is customary in the standard AC framework. (iii) Third, due to the dependency among drivers (an order cannot be assigned to multiple workers concurrently). treating drivers as independent individuals for separate action sampling is impractical.
To address these issues, we impose structural assumptions on the policy function to facilitate its derivation (i.e. the proposed action decomposition strategy). We define Piy.t as the probability that worker i chooses order j at time t, based on the logit model [33]. Specifically, we first expand the utility matrix M. generated by the Actor QK-Attention to M, = [M,, NN.] E [l. where N, is an n-dimensional vector representing the utility of each worker choosing no order. This vector is obtained by processing the output of Actor-BERT with MLP, expressed as N, MLP( [71,e, W2....... Wn,l)). This allows us to compute the probability of each worker choosing each action using the logit model, yielding P. Softmax(MM).
7

---

Published as a conference paper at ICLR 2026
Furthermore, we assume that the aggregate policy is derived from the product of these probabilities:
Te (A())) =
Pise (.EEh(Ae)
II
(5)
where z(-) is increasing function that also depends on the current state St (which we omit for simplicity), and h() is defined as h(Ar) = {i..j) Jaij = 1}. This is a reasonable simplification, as implies that if action Ar has higher value of II.JEMA,)) Pijsts it has a greater probability of being chosen. We wish to emphasize that the probability S does not exist in reality but serves as virtual construct defined by us. We connect the output of the network Me to the policy Th through this defined probability and the mapping function z(.). Essentially, we restrict the policy space to smaller class defined by Eq. to facilitate optimization and application, as follows.
However, defining and computing such a function z(-) is challenging due to the vast action space, complicating the sampling of an action from the strategy T5(A,, We define efficient approach to address this. First, during inference, we can greedily select the action with the maximum probability, as this action should theoretically have the highest utility:
arg max (A,(A))) = arg max Z A,E((S.)
II (.EE((A))
Pigst = arg max A.E)(S.) i.EE((A))
Σ
log Pi5s
A.EV(S.)
(6)
where (S,) is the set of all possible actions under the current state St. This holds because both z(.) and log(-) are increasing functions. We can construct bipartite graph similar to Stage 1, where each available worker and order is represented as a node, and the link between each worker i and order at time is weighted by their log probability log P.t.t. t. By utilizing ILP, we can find the action A, that maximizes 75 (A,) S.). The bipartite graph construction process is detailed in Appendix B.2 During training, we introduce random noise to the probability matrix P, and the model selects actions using the same method as in Eq. 6 When the noise is sufficiently large, the policy degrades totally random policy, and when the noise is zero, the policy converges to greedy strategy. Although we cannot directly express the function z(-), it must ensure that the function (since the noise is totally random). More details about the noise can be found at Appendix C
increasing function
Optimizing this probability using vanilla TD3 is challenging due to the variable action space and the gap between action probabilities and the selected action (the gradient cannot propagate through them). address this, we employ an approximate policy gradient optimization method [46]:
VeJ(O) B,g (0 (S.
B
Σ log P iisht
(.EE((A.)
where I(e) is the optimization objective (long-term cumulative reward), B is a baseline independent of state (we simplify by setting it to 0), and Q TDD3 (St, Ar) is the Q-value under the policy T. which can be estimated by Q TDD3 (S., (S,,, Ar; O) using our proposed network (i = 1, as there are two estimated Q-values in TD3). Detailed derivations can be found in Appendix C We then use gradient
ascent to maximize J ((), thus the loss function for the actor can be expressed as LA = - ()))).
(ii) Critic: For the critic, it can be updated in manner similar to vanilla TD3, where the loss function can be expressed as:
Le
Σ Egg T
R+t1; 0-) (S (s, Ac:
1,2
(8)
OTP' 1++1;07) R +7 min O 1 Actor(Si+1:0 .{); 07)
where QTPs is the learning target function, O represents the parameters of the target network, which updates more slowly than the policy network θ to provide a stable target, and & is a small random noise applied in the probability matrix Φ. More details can be viewed in Appendix D.2
8

---

Published as a conference paper at ICLR 2026
Table Comparison of Different Ride Sharing Methods: Bold entries represent the best results.
Method
DeepPool
BMG-Q
HIVES
Enders
CEVD
Triple-BERT
Type
Independent
CTDE
Centralized
RL Algorithm
IDDQN
48
IDDQN 48
QMIX 38
MASAC [4
VT 45
IDDQN 48
TD3
Multi-Agent
Reward Type
Local MI.F
Local
Global
Local
Global
Local Global
Network Backbone
GAT [5]
GRU
MLP+Autentio
MLP
BERT X
Model Size
20K
117K
16M
118K
23K
16M
GPU Occupation (GB)
3,97
4.28
6.01
8.19
21.45
8.03
Average Reward (10")
12.72
13.04
12.37
12.04
13.16
14.73
EXPERIMENT
To validate the proposed method, we evaluate its performance in the ride sharing dispatching task using real-world yellow ride-hailing data from Manhattan, New York City 147 To illustrate the efficiency and superiority of our proposed Triple-BERT, we compare it with several previous ride sharing methods of different types, including Independent MARL, CTDE MARL, and Centralized Detailed information regarding our experiment configuration, simulator
MARL, as shown in Table [ setup, and a comprehensive description of the comparative experiment can be found Appendix
Neward
Served Order
Ce Velivery Time
3160
22000
13000
3400
Tipk-rt"
Tipa-ER" stagel
1890-9
2000
Hits
11000
Delcon Timm
Pikkup Timm
Darii Time
1.0
0.0
22
7.0
2.0
2.0
Figure 3: Training Process: Each method is trained five times, and the curve is smoothed using Exponential Moving Average (EMA) with a = 0.1. The shaded area represents the standard deviation.
As shown in Fig. we first illustrate the training process of different models by evaluating their performance in the training scenario every 10 episodes. The six sub-figures depict the cumulative reward, the number of orders served, and the average delivery time, detour time, pickup time, and confirmation time for each order. is evident that our method outperforms the other models in most
"The original VD is CTDE method. However, the CEVD variant modifics it to centralized version. 'In original paper DecpPool, the authors used CNN. However, due to differences the ohservation space of our task, we replaced it with ML.. 'tttps: wwtp///wwww..oo/ nyc.
page

---

Published as a conference paper at ICLR 2026
metrics, with the cumulative reward exceeding that of the best alternative method by approximately 15%. The highest number of served orders indicates that our method achieves better cooperation among workers. We then evaluate these methods over different periods, and the average rewards are shown in Table T where our method also demonstrates the best performance. More details about the experimental results can be found in Appendix F.44
To further illustrate the generalization capacity of our method, we evaluate it in additional trials, specifically utilizing all-day data from July 18, 2024, drawn from High Volume For-Hire Vehicle (FHV) trip data 47 This dataset features a different order distribution compared to the yellow-taxi data used for training. The 24-hour data generates 48 episodes, each lasting 30 minutes, during which the order volume varies from 734 to 5,989 in each episode. The experimental results are presented in Table _ The findings remain consistent with previous results: our Triple- BERT model achieves the highest reward by optimizing pickup time to improve the service rate, even if this leads to higher delivery and detour times due to increased order bundling. Additionally, we observe that our method exhibits a higher standard deviation in rewards compared to others. This is because, in low-order volume scenarios, performance among different methods does not substantially differ, as all orders be effectively served. However, in high-order volume scenarios, our Triple-BER' model demonstrates a significant advantage, resulting in higher rewards. Consequently, this contributes to greater standard deviation when aggregating all scenarios. More expanded experiment could be found at Appendix H
Table 2: Performance of Different Methods in Manhattan FHV Data
Method
Reward
Service Rate
Delivery Time
Detour Time
Pickup Time
Confirmation Time
DeepPool
(6538.6612578.6
0,7620.18
13.9320.93
2.54
10.1910.78
0.0910.07
BMG-Q
088993942804.65
0.78=0.17
13.27=1.00
1.44
9.3920.43
0.15+0.11
HIVES [6
11183.124458.22
0.7820.18
13.72a1.48
2.87+1.90
9.7720.68
0.05+0.03
Enders ct al.
10512.65+274.33
0,78=0.17
14.12:0.48
3.06+0.43
9,92=0.55
1.37+0.99
CEVD
3
12556.7423303.93
0.80=0.13
12.33=0.72
2.25+1.33
8.02+1.27
0.0910.08
Triple-BERT
14329.74:4627.26
0.88g0.11
13.07#0.61
2.78+0.92
7.02a00.88
0.34+0.32
Then to demonstrate the model's efficiency, we conduct a series of ablation studies. In terms of model training, we compare the performance of the model with and without stage I pre-training. Regarding the network structure, primarily compare the QK-Attention mechanism with and without the proposed positive normalization module. The detailed results are shown in Fig. 3 We observe that without stage pre-training, the model fails converge and exhibits significant fluctuations. Particularly in the later stages, the reward begins to decrease, which can be attributed to the lack of samples. Additionally, without the proposed normalization in QK-Attention, the model performs poorly, underperforming compared to all other methods. This is due to parameter redundancy, which leads to substantial fluctuations and hinders efficient learning.
CONCLUSION
In this work, we propose the first centralized SARL method, Triple-BERT, for large-scale order dispatching in ride-hailing platforms. Our method successfully addresses the challenge of large action spaces through an action decomposition technique and tackles the issue of sample scarcity with a proposed two-stage training method. The novel network also addresses the large observation space challenge by leveraging the self-attention mechanism of BERT. Additionally, we introduce an improved QK-Attention mechanism to reduce the computational complexity of order dispatching. Through experiments on real-world ride sharing data, we demonstrate that our method significantly outperforms conventional MARL methods, achieving better cooperation among drivers.
However, compared to traditional MARL-based ride-sharing methods, Triple-BERT is more sensitive to single points of failure, as its decisions depend on comprehensive information from all drivers and orders. Efficient strategies to address this dilemma warrant exploration in future work. Additionally, while this study represents the first centralized SARL-based approach to ride-sharing, we view it as merely the starting point for this new paradigm. Future research could focus on identifying more efficient SARL frameworks enhancing our existing method, such as exploring importance sampling within our off ool-colicy policy gradient- based actor optimization method, or investigating the use of offline training to replace our pre-training phase.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work adheres to the principles outlined in the ICLR Code of Ethics.
Efficient ride- saade-shring plays a crucial role in promoting convenient and sustainable urban transportation services. By enabling greater sharing among passengers, our method not only increases platform profitability and operational efficiency but also helps reduce total vehicle miles traveled and per-capita carbon emissions compared to solo rides. This. in turn, supports environmental sustainability goals. Moreover, our centralized reinforcement learning framework improves coordination among drivers, reduces delivery and detour times, and enables the platform to time frame. As a result, both platform income and customer satisfaction are enhanced, while also contributing greener and more efficient transportation system. We believe these contributions highlight the practical significance and societal value of our research.
orders within the same
However, the issue of algorithmic discrimination has received widespread attention over time. Closed- box management algorithms, including those for order dispatching, have been shown to create discriminatory scenarios for workers, as reinforcement learning methods primarily aim maximize rewards without considering ethical implications. For example, algorithms may set different payment structures or order assignment preferences based on individual features or geographical locations of workers. We hope that our method will not exacerbate these issues and can be further developed to include constraints that promote fairness. Our goal is to strike a balance between profit and ethics, fostering win-win situation for platforms, workers, and customers.
REPRODUCIBILITY STATEMENT
The source code, trained parameters, and processed dataset are available at hattts::/githuut com com/ 200///220022//::1e-BBBR11 The appendix also includes detailed description of our method-
ology.
REFERENCES
[1] Abubakr Al-Abbasi, Arob Ghosh, and Vaneet Aggarwal. Deeppool: Distributed model-free algorithm for ride-sharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 20(12):4714 2019.
[2] Javier Alonso-Mora, Samitha Samaranayake, Alex Wallar, Emilio Frazzoli, and Daniela Rus. On-demand high-capacity ride-sharing via dynamic trip-vehicle assignment. Proceedings of the National Academy of Sciences, 114(3)462-467,, 2017.
[3] Avinandan Bose, Hao Jiang, Pradeep Varakantham, and Zichang Ge. On sustainable ride pooling through conditional expected value decomposition. In ECA/ 2023, pp. 295-302. IOS Press, 2023.
[4] Tingwei Chen, Yantao Wang, Hanzhi Chen, Zijian Zhao, Xinhao Li, Nicola Piovesan, Guangxu Zhu, and Qingjiang Shi. Modelling the 5g energy consumption using real-world data: Energy fingerprint is all you need. arXiv preprint arXiv:2406.16923 2024.
[5] Kyunghyun Cho. B van Merrienboer, Caglar Gulcehre, F Bougares, H Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), 2014.
[6] Yi-Hui Chou, I-Chun Chen, Joann Ching, Chin-Jui Chang, and Yi-Hsuan Yang. Midibert-piano: Large-scale pre-training for symbolic music classification tasks. Journal of Creative Music Systems, 8(1), 2024.
[7] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. International Conference on machine learning, pp. 1538-155 1546. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2026
[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4188 2019.
[9] Tobias Enders, James Harrison, Marco Pavone, and Maximilian Schiffer. Hybrid multi-agent deep reinforcement learning for autonomous mobility on demand systems. In Learning for Dynamics and Control Conference, pp- 1284-1296. PMLR, 2023.
[10] Siyuan Feng, Peibo Duan, Jintao Ke, and Hai Yang. Coordinating ride-soureing and public transport services with a reinforcement learning approach. Transportation Research Part C: Emerging Technologies, 138: 103611, 2022.
[11] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[12] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
[13] Shuxin Ge, Xiaobo Zhou, and Tie Qiu. Marl-based pricing strategy via mutual attention for mod systems with ridesharing and repositioning. In JEEE INFOCOM 2025-IEEE Conference on Computer Communications, pp. 1-10. IEEE, 2025.
[14] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1861-1870. Pmir, 2018.
[15] Marina Haliem, Ganapathy Mani, Vaneet Aggarwal, and Bharat Bhargava. A distributed model- free ride-sharing approach for joint matching, pricing, and dispatching using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 22(12):7931-7MM 2021.
[16] Jiang Hao and Pradeep Varakantham, Hierarchical value decomposition for effective on-demand ride-pooling. In Proceedings of the 21 International Conference Autonomous Agents and Multiagent Systems, pp. 580-587, 2022.
[17] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, (8):1735-1780, 1997.
[18] Joshua Holder, Natasha Jaques, and Mehran Mesbahi. Multi agent reinforcement learning for sequential satellite assignment problems. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 26516 26516-26524 2025.
[19] Heiko Hoppe, Tobias Enders, Quentin Cappart, and Maximilian Schiffer. Global rewards in multi-agent deep reinforcement learning for autonomous mobility on demand systems. In 6th Annual Learning for Dynamics & Control Conference, pp. 260-272. PMLR, 2024.
[20] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, volume 1, pp. 3, 2022.
[21] Yulong Hu, Tingting Dong, and Sen Li. Coordinating ride dool-pooing with public transit using reward-guided conservative q-learning: An offline training and online fine-tuning reinforcement learning framework. arXiv preprint arXiv:250J. 2025.
[22] Yulong Hu, Yali Du, and Sen Li. Real-time coordination of human couriers and drones for on- demand food-delivery platforms: A multi-stage risk-aware multi-agent reinforcement learning framework. Transportation Research Part C: Emerging Technologies, 181: 181:105381, 2025.
[23] Yulong Hu, Siyuan Feng, and Sen Li. Bmg-q: Localized bipartite match graph attention q-learning for ride-pooling order dispatch. arXiv preprint 2xi:25501.1444 2025.
12

---

Published as a conference paper at ICLR 2026
[24] Weiqiang Jin, Hongyang Du, Biao Zhao, Xingwu Tian, Bohang Shi, and Guang Yang. A comprehensive survey on multi- agent cooperative decision-making making: Scenarios, approaches, challenges and perspectives. arXiv preprint arXiv: 10550...111,,, 2025.
[25] Jiahui Li. Kun Kuang, Baoxiang Wang, Furui Liu, Long Chen, Fei Wu, and Jun Xiao. Shapley counterfactual credits for multi-agent reinforcement learning. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 934-994-999 942, 2021.
[26] Minne Li, Zhiwei Qin, Yan Jiao, Yaodong Yang, Jun Wang, Chenxi Wang, Guobin Wu, and Jieping Ye. Efficient ridesharing order dispatching with mean field multi-agent reinforcement learning. In The world wide web conference, pp. 983-994, 2019.
[27] Xinran Li and Jun Zhang. Context-aware communication for multi-agent reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pp. 1156-1166 2024.
[28] Xinran Li, Xiaolu Wang, Chenjia Bai, and Jun Zhang. Exponential topology-enables scal- able communication multi-agent reinforcement learning. In The Thirteenth International Conference Learning Representations.
[29] Yang Liu, Fanyou Wu, Cheng Lyu, Shen Li, Jieping Ye, and Xiaobo Qu. Deep dispatching: A deep reinforcement learning approach for vehicle dispatching on online ride-hailing platform. Transportation Research Part E: Logistics and Transportation Review, 161:102694, 2022.
[30] Dennis Luxen and Christian Vetter. Real-time routing with openstreetmap data. Proceedings of the 19th ACM SIGSPATIAL International Conference Advances in Geographic Information Systems, GIS '11, pp. 513-516, New York, NY. USA, 2011 ACM. ISBN 978- 178-1-4503-1031- URL http:http://eoo.....ooor////.... httpp///..11.....2ee.
4. doi: 10.1145/2093373 2094062..
2094062]
[31] Hangyu Mao, Zhengchao Zhang, Zhen Xiao, and Zhibo Gong. Modelling the dynamic joint policy of teammates with attention multi-agent ddpg. arXiv preprint arXiv: 1.17,29 2018.
[32] Laëtitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agen teams. In 2007 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 64-69 IEEE, 2007.
[33] Daniel McFadden. Conditional logit analysis of qualitative choice behavior. 1972.
[34] G Ayorkor Mills-Tettey, Anthony Stentz, and M Bernardine Dias. The dynamic hungarian algorithm for the assignment problem with changing costs. Robotics Institute, Pittsburgh, PA, Tech. Rep.
07-277, 7, 2007.
[35] Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforce- ment learning. Applied Intelligence, 53(11))13677-13722, 2023.
[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performant deep learning library. Advances in Neural Information Processing Systems, 32, 2019.
[37] Zhiwei Qin, Xiaocheng Tang, Qingyang Li, Hongtu Zhu, and Jieping Ye. Reinforcement Learning in the Ridesharing Marketplace. Springer, 2025.
[38] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Leurning Reseurch, (111))):1-51, 2020.
[39] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
13

---

Published as a conference paper at ICLR 2026
[40] Jie Shi, Yuanqi Gao, Wei Wang, Nanpeng Yu, and Petros A Ioannou. Operating electric vehicle fleet for ride-hailing services with reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 21(11):4822 4834, 2019.
[41] Ashutosh Singh, Abubakr O Al-Abbasi, and Vaneet Aggarwal. A distributed model-free algorithm for multi-hop ride-sharing using deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, 23(7):8555-8605 2021.
[42] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pp. 5887-5896. PMLR, 2019.
[43] Jianyu Su, Stephen Adams, and Peter Beling. Value-decompooittion multi-agent actor-critics. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1 11352-1133 11360, 2021.
[44] Sainbayar Sukhbaatar, Rob Fergus, al. Learning multiagent communication with backpropa- gation. Advances in neural information processing systems, 29, 2016.
[45] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanetot, Nicolas Sonnerat, Joel Leibo, Karl Tuyls, al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Pro- ceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 2085-2087 2018.
[46] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
[47] New York City Taxi and Limousine Commission. Nyc taxi and limousine commission-trip record data nyc. URL
nyc. http://wwwww.c.eee///ee/e//l////
flc-trip-record--ee page
[48] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[49] Paulina Varshavskaya, Leslie Pack Kaelbling, and Daniela Rus. Efficient distributed reinforce- ment learning through agreement. In Distributed Autonomous Robotic Systems 8, pp. 367-378. Springer, 2009.
[50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[51] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, al. Graph attention networks. stat, 1050(20):10 10-48550, 2017.
[52] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3)2611272, 2020.
[53] Dujuan Wang, Qi Wang, Yunqiang Yin, and TCE Cheng. Optimization of ride-sharing with passenger transfer via deep reinforcement learning. Transportation Research Part E: Logistics and Transportation Review, 103000 2023.
[54] Jingwei Wang, Qianyue Hao, Wenzhen Huang, Xiaochen Fan, Zhentao Tang, Bin Wang. Jianye Hao, and Yong Li. Dyps: Dynamic parameter sharing in multi-agent reinforcement learning for spatio-temporal resource allocation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 3128-3 1132-339, 2024.
[55] Xuefeng Wang, Xinran Li, Jiawei Shao, and Jun Zhang. Ac2c: Adaptively controlled two-hop communication for aulti-agent reinforcement learning. In Proceedings of the 2023 International Conference Autonomous Agents and Multiagent Systems, pp. 425- 2023.
14

---

Published as a conference paper at ICLR 2026
[56] Yinquan Wang, Jianjun Wu, Huijun Sun, Ying Lv, and Guangtong Xu. Reassignment algorithm of the ride-sourcing market based on reinforcement learning. IEEE Transactions on Intelligent Transportation Systems, (1110))002331092000 2023.
[57] Yinquan Wang, Jianjun Wu, Huijun Sun, Ying Lv, and Junyi Zhang. Promoting collaborative dispatching in the ride-sourcing market with a third-party integrator. IEEE Transactions on Intelligent Transportation Systems, 25(7):6889- 6107)99,, 2024.
[58] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pp. 1995-2003, PMLR, 2016.
[59] Shangyao Yan, Chun- Ying Chen, and Yu-Fang Lin. A model with a heuristic algorithm for solving the long-term many-to-many pooling problem. IEEE Transactions Intelligent Transportation Systems, 12(4):1332-1373, 2011.
[60] Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multi-agent reinforcement learning. International conference on machine learning, pp- 5571-5580. PMLR, 2018.
[61] Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress cooperative multi-agent reinforcement learning in open environment. arXiv preprint arXiv:2312.00 01.0588, 2023.
[62] Xianjie Zhang, Jiahao Sun, Chen Gong, Kai Wang, Yifei Cao, Hao Chen, and Yu Liu. Mutual information as intrinsic reward of reinforcement learning agents for on-demand ride pooling. arXiv preprint arXiv:2312.15195 2023.
[63] Xianjie Zhang, Pradeep Varakantham, and Hao Jiang. Future aware pricing and matching for sustainable on-demand ride pooling. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 14626, 2023.
[64] Zhongyun Zhang, Lei Yang, Jiajun Yao, Chao Ma, and Jianguo Wang. Joint optimization of pricing, dispatching and repositioning in ride-hailing with multiple models interplayed reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 2024.
[65] Zijian Zhao and Sen Li. The impacts of data privacy regulations on food food-delivery platforms. Transportation Research Part C: Emerging Technologies, 181:105364, 2025.
[66] Zijian Zhao and Sen Li. One step is enough: Multi-agent reinforcement learning based on one-step policy optimization for order dispatch on ride-sharing platforms. arXiv preprint arXiv:2507.15351, 2025.
[67] Zijian Zhao and Sen Li. Discriminatory order assignment and payment-setting of on-demand food-delivery platforms: A multi-action and multi-agent reinforcement learning framework. Transportation Research Part E: Logistics and Transportation Review, 208: 104653, 2026.
[68] Zijian Zhao, Tingwei Chen, Zhijie Cai, Xiaoyang Li, Hang Li, Qimei Chen, and Guangxu Zhu. Crossfi: cross domain wi-fi sensing framework based on siamese network. IEEE Internet of Things Journal, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
A Action Space Size
17
BiParite Graph Construction
17
B.1 IDDON Bipartite Graph
17
B.2 TD3 Bipartite Graph
17
Policy Gradient Proof
18
D Training Process
19
D.I Stage I: IDDQN Algorithm
19
D.2 Stage 2: TD3 Algorithm
20
Experiment Details
E.
Experiment Configurations
21
Simulation Setup
22
E.3
Introduction of Comparative Methods
22
Additional Experiment Result
23
E.5
Expanded Generalization and Extensibility Experiment
23
E.6
Expanded Ablation Study
24
Related Work
26
Order Dispatch Ride- Side-Sharing Task
26
F2
Cooperative Multi-Agent Reinforcement Learning
26
16

---

Published as a conference paper at ICLR 2026
A ACTION SPACE SIZE
The action space in our order dispatching task given by:
C(m,, k))((, k) Σ m n!
AAe
(9)
kl(m, k)! (n k)! k=
A===
where P(n, k) represents the permutations of assigming k orders to workers and C(m,, k) represents the combinations of selecting k orders from the total m, orders. This equation based on two assumptions: (i) the platform will assign an arbitrary number of orders at each step (some orders yielding negative income will be declined by the platform) and (ii) the number of orders my less than the number of workers which can always be satisfied since me represents the order count at only one timestep. Then we can derive the lower bound of Ar] as:
n2!
AA
Σ k=
C(m2,
k)
C(m,: k) - k + 1)
n K)!
k=
(10)
C(m,, (n - m. + 1)* (n m, 2) 2 2T
(n 2 m 20.
k==
As a result, the action space has a lower bound with the exponent to m. Consider the example in Section 2 where the number of workers n is 1000 and the number of orders m, is 10. In this case, the expression (n m, + 2) evaluates to 992 x 1030
B BIPARITE GRAPH CONSTRUCTION
B.1 IDDQN BIPARTITE GRAPH
The bipartite graph in the IDDQN-based order dispatching method is constructed as follows:
max A
Oijt
Digie:
(ll
iE7
S.t.
Oiii 11, Vj Jr,
(11b)
IEI
Σ
Oidh 1
ViEI,
(11c)
jEZ,
ai.j E (0,1), WEI,EJJ
(11d)
where arjjtt is the action representing whether worker assigned order j at time (with indicating assignment and 0 indicating at time t (with Vi.j.t = oo for all unavailable workers at time t), I is defined as (1, 2,.... n), and the set is defined as 21, m.). Constraint lb ensures that order can be assigned to most one worker, while constraint T1c guarantees that each worker is assigned at most one order.
assignment), Vi.j.t denotes the Q-value of worker choosing order
B.2 TD3 BIPARTITE GRAPH
The bipartite graph in our proposed TD3-based order dispatching method is constructed follows:
max
Σ
Fiih
log P P..j.t
(12a)
Xx
1EIN"
s.t.
Σ
Vj E Jr, Fiiit 11,
(12b)
IET
Σ
Fiiit = 1,
Vi E I,"
(12c)
JEJ,
Fi.j E (0,1),
Vi I,j E J.U U (m+ 1),
(12d)
17

---

Published as a conference paper at ICLR 2026
where Zí" represents the set of available workers time t. Here, constraint 12b does not apply in the last column, as it represents declining all orders, an action that can be chosen by any worker. Constraint [ requires each row equal ensuring that each worker must either take an order or reject all, without other choices. We can then convert X to action Ag as follows:
Fi.j.
if I and +1, = 0
di,t
0
otherwise
(13)
C POLICY GRADIENT PROOF
According to the policy gradient theory 46). we have:
VeJ(O)
x E
1 (S A.) - B) Ve log (A(A)))))
(QTP* (St,A) BB Ve logz
II
Pisie
(JEh(A,)
Q (St, A1) At) -B) dz(lI dZ(TII)EMAA)) Pijae) IIL.EEAA))) Pisit F
Vo log II iJEh(A,)
Pistt
a[IiSEM(AN) Piys Z(Teeenna) Pigie)
PT As) B) Ed Aie Ve G
Σ
log P
i,j.
(JEh(A.)
(14)
where & denotes elasticity, which measures the sensitivity of one variable changes in another, and is defined as:
dlogy dy
Ey. dlogæ
(15)
dr
As introduced in Section 3.3.2 the probability 3 and the mapping function z(-) do not exist in reality but serve as virtual constructs used to connect the output to the policy Ti for simplified optimization and application. This means we can define them any format we choose, and they restrict the policy space to structural class defined by Eq. 5
To further simplify optimization, we define the format of z(-) as x(x) = ar where a, b > 0. This formulation has the advantage that the elasticity of z(-) is positive constant, i.c. Edt,),r ab. Thus,
B) Ve Ei.jEN(A)) log P..aa corresponding Eq. 7 This format of z(-) aligns with assumption that z(-) should be increasing function, which implies that if a driver-order pair has a higher probability of being chosen, it should also have a higher utility, resulting in greater likelihood of being selected in the joint action.
we have: (e) x E, [ (o (S,,
As mentioned in Section 3.3.2. during training, we add random noise to P, and then choose the action that maximizes Σi.ee(AA) log P i.j.t. Currently, the mapping from EijEM(A,) log P k.j.t to the choosing probability Th corresponds to z(.). To further illustrate the robustness of our method, compare the performance of our model using Gaussian noise [18]. uniform noise, and binary symmetric channel (BSC) noise, where the noise follows Bernoulli distribution and has been widely utilized in previous work 21). During training, we gradually reduce the noise to make the policy more deterministic. The experimental results are shown in Fig. g where we observe that, despite certain performance differences between the various types of noise, they all outperform conventional MARL methods. This suggests the efficiency and high robustness of our proposed method, indicating that the detailed expression of z(xr) does not significantly influence the validation of the method based on Eq. 7 even if it may cause some performance gaps. The optimal noise for our task may require further exploration. For fairness, we choose to use BSC noise when comparing with other methods, even though it appears to perform the worst among the three rypes of noise. We aim demonstrate that our results are robust and superior, not relying on a particular choice of experiment scenarios.
or
18

---

Published as a conference paper at ICLR 2026
Reward
Served Order
Delivery Time
15800
3720
12.5
Unform Gaussian
15000
3710
15400
Order
Delivery Time
12.9
3700
Reward
15200
15000
3630
11.5
2680
200
400
800
1000
Epoch
Epoch
Epoch
Detour Time
Pickup Time
Confirmation Time
3.6
6.50
Confirmation Time
Detour Time
Pickup Time
0.10
6.00
2.0
0.06
2.6
0.04
5.25
0.00
2.0
200
400
000
800
1000
200
000
1000
200
400
000
800
1000
Epoch
Epoch
Epoch
Figure 4: Comparison Between Different Noise Methods: Each method trained three times, and the curve is smoothed using EMA with 0.1. The shaded area represents the range of fluctuations, while the solid line indicates the average value.
D TRAINING PROCESS
D.1 STAGE 1: IDDQN ALGORITHM
Q-value Matrix
Actor QK-
Attention
Worker Encoder
Order Encoder
m
Worker Information
Order Information
Figure 5: Network Structure in Stage
In stage 1, the network structure is shown as Fig.
which is consisted by the encoders and the Although the model takes the entire worker and
OK-Attention module of proposed network in Fig.
19

---

Published as a conference paper at ICLR 2026
order sequence input, it primarity aims utilize parallel computation to enhance computational efficiency. In the encoders, each worker and order's 's information is processed separately. Similarly, in the QK-Attention module, the Q-value for each worker-order pair is computed independently. It is also feasible to imput only a single worker-order pair into this network, computing the Q-value exclusively for that pair; however, this would increase the computation time.
During IDDQN training, we need to introduce some noise into the Q-matrix Y, to facilitate sufficient exploration. Specifically, for the e-greedy strategy, we randomly select proportion of non--oo elements in Y, and set them to a large positive number to enhance their likelihood of being selected. We then update the neural network by minimizing the TD-error, expressed as:
LQ = Egg
O
DQN 9
- -Q DON (8i, aizz Q)
Q DQN
7i,0111 +
-i,t+1
DQN
(i,441,
(16)
D
DQN (8,,411,
61,41 = arg
max 1
where Q is the learning target function, y is the discount factor, 2,,c+1 is the possible action -
space for worker at time + 1, and Φ represents the parameters of the target network, which updated at a slower pace compared to the policy network to provide a stable target for training. After
each training iteration, the target network updated in soft manner: 4 := T (1 7) where 7 is the update rate.
The detailed process is illustrated in Algorithm ! where 1, represents the vector that only the jth position is and other positions are O.
Algorithm IDDQN Training Process
Require: Number of training episodes E, number of training steps T, mini-batch size m, target
update rate exploration noise final exploration ey, exploration decay 5, discount factor model parameters Q
1: Initialize target networks +- + O
2: Initialize replay buffer B
3: for k 1 to F do
4: for
7 do 5: Calculate Q-value matrix Y;: Vig.t Q
DQN
(st,1, 1,i Φ)
6: Select action with exploration noise:A, ILL,(Y,,))
7: Observe reward ri,4+1 and new state 81,4+1 for each worker
8:
Store transition (si,s, Oi,t, F2,44,, 81,4+1 in B
9:
Sample mini-batch of m transitions (s, (,,, from B
Compute target Q-value:
y - + 2ODO0
DQN (si,441, arg max
DQN
DQN
(i,,44,,, 4):
12:
Update N-Network: +- arg ming 1 E
DQN
(s, a; 4))²
m
13:
Update target networks: 4 + T + (1 T
end for
15:
Decay exploration: c < max(c;, (d)
end for
D.2 STAGE 2: TD3 ALGORITHM
The process of our Stage TD3 training is illustrated in Algorithm? 2 In experiment, we follow the vanilla TD3 approach of updating the actor once after updating the critic twice.
20

---

Published as a conference paper at ICLR 2026
Algorithm 2 TD3 Training Process
Require: Number of training episodes E, number of training steps T, mini-batch size policy delay d, target update rate т, exploration noise e, final exploration ef. exploration decay 6, target policy smoothing noise ç, discount factor 7, model parameters θ
1: Initialize target networks è t-6
2: Initialize replay buffer B
3: for = to E do
4:
for f= 1 to T do
5:
Select action with exploration A, Oc)
6:
Observe reward R4+1 and state S
7:
Store transition (St, Ar, S141, in B
8:
Sample mini-batch of N transitions (S, A, R, S') from B
9:
Compute target action with smoothing noise: A' + Actor(S;
10:
Compute target Q-value: f- + min Q TD3 (S', 0T)
Update critics:
arg argmine m Σl(y Q T
(S, A;))) (v (2
12:
if & mod d = 0 then
Update actor using deterministic policy gradient:
14:
J(( mi [ A; Q) Ve log (
(A Actor Actor(S; (S;
15:
Update target networks: θ + - + (1- end if
17:
end for
18:
Decay exploration: 6 + max(ef,
19:
end for
E EXPERIMENT DETAILS
E.1 EXPERIMENT CONFIGURATIONS
Our model was trained using the PyTorch framework [36] on workstation running Windows 11, equipped with an Intel(R) Core(TMM) i7-14700KF processor and an NVIDIA RTX 4080 graphics card. The detailed model configurations are shown as Table During the training phase, the model utilized approximately 8.03 GB of GPU memory. For optimization, an initial learning rate of 10 -4 and a decay rate of 0.99. Stage I, the batch size was set to 256, while in Stage 2, it reduced to 16, due to sharp decrease in sample amount. Additionally, optimization was performed once every 4 time steps, and in Stage 2, the actor was updated once for every two updates of the critic.
employed the Adam optimizer with
Table 3: Model Configurations
Configuration
Our Setting
Hidden Dimension
64 (Actor) 128 (Critic)
Attention Heads
BERT Layers
for Each 0.1
Dropout Rate
Optimizer
Adam 10-4
Learning Rate
Scheduler
ExponentialL.R
Learning Rate Decay
0.99
Batch Size
256 (Stage 1) 16 (Stage 2)
Exploration Rate Updating Rate of Target Network Discount Factor
0.99 - 0.0005
0.005
0.99
21

---

Published as a conference paper at ICLR 2026
E.2 SIMULATION SETUP
In the simulation, we follow previous works [23 I. setting the total number of drivers to 1,000, with each car having capacity of 3 passengers. The maximum waiting time for customers is set to 5 minutes. Fach episode lasts 30 minutes, divided into 30 time steps, where each step determines the operations for the subsequent minute. For the TSP route optimization and time estimation, we utilize the OSRM simulator [30), with a default traveling speed of 60 km/h. For solving the bipartite matching, we use the Hungarian algorithm [34,, provided SciPy [52].
We train the model using data from 19:00 to 19:30 on July 17, 2024, which includes 3,726 valid orders, and we test the trained model during other time periods on July 17, 2024, including 14:00-14:30 (2,850 valid orders), 17:00-17:30 (3,577 valid orders), 20:00-20:30 (3,114 valid orders), 21:00-21:30 (4,264 valid orders), and 22:00-22:30 (4,910 valid orders). where the order amount range from 2,850 to 4,264.
The evaluation metrics include:
Served Rate: the rate of confirmed trips relative the total trip requests initiated by customers.
Delivery Time: the time taken to serve a trip from origin to destination.
Detour Time: the extra time spent on delivery beyond the minimum delivery time (i.e., the time if
the vehicle only serves this trip without bundling other trips).
Pickup Time: the waiting time for customers between the trip confirmation and the arrival of the vehicle at the trip origin.
Confirmation Time: the waiting time for customers from when they initiate the trip request to when the platform assigns the trip to vehicle.
E.3 INTRODUCTION OF COMPARATIVE METHOOSS
The methods using in our comparative experiment can be mainly divided into three categories:
Independent MARL: The DeepPool [ and BMG-Q [23 utilize similar IDDQN method as described in Section 3.3. I with BMG-Q employing GAT (5) to capture the relationships among neighboring agents. Additionally.in the original paper fo DeepPool, the authors used CNN. However, due to differences in the observation space of our task, we replaced with MLP.
Centralized Training Decentralized Execution (CTDE): The HIVES (38) framework introduces a QMIX 38) based method to address the shortcomings of IDDQN, specifically the inadequacy of treating the global Q-value as simple summation of the individual Q-values of each agent. Enders et al. (9) propose a MASAC [4] based approach, allowing each driver to choose whether to accept order, thereby preventing low-profit orders from negatively impacting the global income. Centralized Training and Centralized Execution (CTCE): CEVD (3. hased on VD (45), innovatively combines the Q-values of each agent with those of their neighbors to create new type of Q-value, akin to the motivation behind BMG-Q.
Overall, most of these methods attempt various strategies to enhance each agent's awareness of the global state, facilitating better cooperation. In contrast, our method directly transforms the formulation into a centralized singlc-agent reinforcement learning approach.
It is noteworthy that these Independent and CTDE MARL dispatching methods differ slightly from general MARL methods. In order dispatching, one order cannot be assigned to multiple workers, making it necessary to employ some centralized mechanism achieve this. We refer to them as independent MARL and CTDE methods because they can directly calculate their own Q-values or action probabilities using their own or neighboring states. Conversely. CEVD must calculate the primary Q-value of each agent separately and then combine those primary Q-values with their neighbors to obtain a final Q-value for each agent.
Through the experimental results in Fig. 3 we observe that DeepPool 11. serving as one of the earliest benchmarks, demonstrates relatively stable and good performance, suggesting the simplicity and effectiveness of IDDQN features. In contrast, BMG-Q 23 significantly improves performance by utilizing FAT to capture neighboring information. As for HIVES (38] and CEVD B], while they exhibit relatively good performance in the early stages of training likely due to their hierarchical structure and centralized training methods thheodd--ehirrro rewards even starting to decrease. This instability may stem from the hierarchical approach not
their performance becomes unstable in later stages, with
22

---

Published as a conference paper at ICLR 2026
adequately addressing the large network input of the mixture network in QMIX and the lazy agent problem in VD. Additionally, their centralized training approach faces the same data searcity issues as method, making convergence more challenging. For Enders et al. (9, note that their method shows worse performance than others. This may be related to their state processing method during training. where they replace the next state in the replay buffer with the request state from the current state to maintain a consistent agent count across two successive time steps, which appears be strong assumption. Finally, for the last three methods, their original papers primarily focus their reward functions on the serving order amount, without incorporating additional terms like ours (which also considers income, outcome, and user satisfaction levels). This makes our scenario more complex and may further reduce the performance of their methods in our setting.
E.4 ADDITIONAL EXPERIMENT RESULT
The detailed experimental results across different time periods are shown in Fig. 6 while the weighted average numerical results are presented in Table For each model in each scenario, we repeat the experiment three times, and the error bars in the figure represent the standard deviation. We observe that our Triple-BERT achieves the highest reward across all scenarios, with the advantage becoming more pronounced the order volume increases. Triple-BERT primarily optimizes the service rate and pickup time, significantly outperforming other methods.
For delivery time and detour time, the figures only account for completed orders, as the status of unfinished orders is uncertain, which may introduce some bias in the detailed values. In terms of these two metries, Triple-BERS clearly performs better in high order volume scenarios, but not in low order volume scenarios. This may be due to the relatively low conflict caused by MARL in low order scenarios, while in high order scenarios, both the observation and action spaces increase sharply, making it challenging for MARL to find optimal solutions.
Lastly, we note that our method and the approach by Enders et al. g exhibit higher confirmation times. This may be attributed to both methods having an explicit rejection action (i.e., choosing no order), unlike the other methods. While this mechanism can lead to higher confirmation times, it also enables the model discard negative profit orders and reserve some orders for currently unavailable workers.
Table 4: Average Performance under Multiple Periods: Bold entries represent the best results.
Method
Reward
Service Rate
Delivery Time
Detour Time
Pickup Time
Confirmation Time
DeepPool 1
12723.85
0,91
11.53
2.47
7.77
0.06
BMG-Q
13036.29
0,92
10.57
7.61
HIVES [16
12365.11
0.89
11.04
2.28
7,99
0,03
Enders el al.
12041.62
0.90
12.28
2,90
7.94
0.80
CEVD [3
13157.96
0.94
11.36
2.31
7.37
0.06
Triple-BERT
14730.48
0.98
11.53
2.52
5.73
0.13
w/o stage 1
10665.02
0.87
11.92
9.36
0.68
w/o normalization
10839.33
0.88
12.50
2.85
9.10
0.24
E.5 EXPANDED GENERALIZATION AND EXTENSIBILITY EXPERIMENT
In this section, we first further prove the generalization capacity of our method in more testing scenarios, using the scenarios of other days, not just the other periods in the same day as the last section. The result is shown in Table 5 Specifically, we compared our Triple-BERT with other methods using the testing scenario from July 16 to July 18, 2024, at 6 PM. The results indicate that our Triple-BERT achieved the highest reward among all scenarios, highlighting its strong generalization capacity.
Then, we aim to prove the extensibility and scalability of our method. As a result, we compared its performance against other approaches as the driver count increased to 1,500 and 2,000 during high concurrency period. In this scenario, the order volume reached 6,775, which we synthesized by combining orders from two different periods. The result is shown in Table 6 The results indicate that our Triple-BERT model achieves the highest reward across various scenarios, without the need for retraining.
23

---

Published as a conference paper at ICLR 2026
Table Reward of Different Methods Under Different Days
Scenario
DeepPool
I
BMG-Q 23
HIVES [16]
Enders et al. [
CEVD R
Triple-BERT
7.16 (4,451 orders)
13,473
14,121
12,070
12, 142142
14,226
16,831
7.17 (4,125 orders)
13,204
13,424
12,232
12,208 12,268
14,145
16,145
7.18 (3,635 orders)
12,679
13,067
12,397
13,336
14,819
Table 6: Reward of Different Methods During High Concurrency Period among Different Driver Amounts
Driver Amount
DeepPool [1
BMG-Q 23
HIVES [6
Enders et al. 9
CEVD 3
Triple-BERT
1,500
21,090
22,333
19,587 25,713
19,546 25,394
23,092
27,458
2,000
25,316
25,650
26,207
28,273
Additionally, to prove the practical application potential of our method, we test the computation time with driver amounts ranging from 1,000 2,000 and order amounts from 300 500 at a single time step, shown in Table (In our real-world dataset, we typically observe that the order amount does not exceed 200 at any single step.) The results indicate that the decision time remains consistently under 0.2 seconds across all scenarios. Additionally, the order and driver counts have minimal impact on computation time. This suggests that, for the current simulation, most of the processing time is allocated to the simulator's operations rather than to decision computation cost.
Table Decision Time of Different Driver and Order Amounts (unit: seconds)
Order Amount
300
400
500
Driver Amount
1,000
0.1801
0.1839
0.1870
1,500
0.1806
0.1820
0.1830
2,000
0.1789
0.1829
0.1809
To further demonstrate the generalizability of our method, conducted an expanded experiment using High Volume For-Hire Vehicle (FHV) trip data from Queens, New York City [47]. We chose continue with the yellow taxi data, its primary operational area is Manhattan. Unlike the capital-intensin region of Manhattan, Queens has significantly lower trip volume, despite its area being about five times larger. Consequently, the data distribution in Queens presents markedly different challenge: while the number of orders decreases, the distances between them tend to increase, leading to greater difficulties for ride-hailing services. To adapt to this scenario, we set the driver count to 500 while keeping all other settings consistent.
The detailed experimental results are presented in Table using data from 19:00 19:30 on July 17, 2024, which includes 2,024 valid orders. We observe that our BERT maintains its SOTA performance, primarily optimizing the assignment and improving rewards by serving more orders and reducing pickup times. However, this inevitably leads to slight increase in delivery and detour times due to the bundling of more orders.
Additionally, compared to the results in Manhattan, shown in Table 4 we noted that the average pickup time Queens is significantly longer. While the decrease in the number of drivers may contribute to this, it is not the main factor, the order volume has also declined significantly. The primary reason lies in the larger area and more dispersed order distribution in Queens. This leads to a substantial penalty in the pickup time term of the reward function, resulting in minimal differences in the rewards among different policies. Such challenging scenarios further hinder the efficient exploration and learning of centralized MARL methods like HIVES and CEVD.
E.6 EXPANDED ABLATION STUDY
In this section, we aim to further illustrate the efficacy of removing the positional embedding in BERT and the utilization of ARL style attention-based encoder for better feature extraction.
24

---

Published as a conference paper at ICLR 2026
Table 8: Performance of Different Methods in Queens, New York City
47
Method
Reward
Service Rate
Delivery Time
Detour Time
Pickup Time
Confirmation Time
DeepPool
5222.85
0.64
11.24
1.84
12.30
0.21
BMG-Q 23
$362.00
0.66
9.63
1.08
12.98
0.27
HIVES
3560.80
0.41
Enders
4543.68
0.61
10.41
0.85
13.39
2.25
CEVD
4388.83
0.62
11.61
1.33
13.74
0.29
Triple-BERT
5577.83
0.72
9.07
0.90
11.32
0.23
For the positional embedding. the rationale behind eliminating the positional embedding includes:
Incorporated Position Information in State Space: The coordinates of each vehicle and order are included as part of the state input, making additional positional embeddings unnecessary. Nature of Vehicle and Order Dispatching: In ride-sharing tasks, the optimal assignment should independent the input sequence order. By removing positional embedding, we ensure that all positions homogeneous, thus realizing this property.
Scalability Considerations: Utilizing positional embeddings in BERT requires a predefined maximum input length before model training, which cannot be altered later. In scenarios of extreme concurrency, the number of orders may exceed this maximum length, potentially compromising model efficacy.
To further prove it, we compare the model performance with and without the positional embedding as shown in Table 9 The results indicate that positional embedding introduces additional interference, resulting in lower performance. This is particularly evident in generalization problems: when using positional embeddings, Triple-BERT only outperforms previous MARL SOTA in training scenarios, but not in testing scenarios. Additionally, when the order amount exceeds the training scenario, the maximum length restriction hinders the model's effectiivness..
Table 9: Reward Tripe-BERT w/ and w/o Positional Embedding (PE)
Order Amount
3,726 (train)
2,850
3,114
3,577
3,910
4,264
PE
w/
14,092
10,679
11,431
12,841
w/o
15,388
11,148
13,483
14,477
16,335
17,366
To illustrate the efficiency of the encoder, compare methods and others when using the attention-based encoder and vanilla MLP-based encoder. The result is shown in Table [O] The results indicate that the ARL-based encoder significantly enhances the performance of our Triple-BERT alongside independent MARL methods such as DeepPool and BMG-Q. However, this improvement does not extend CTDE and centralized MARL approaches like HIVES, Enders et al., and CEVD. The underlying reason is that CTDE and centralized MARL methods in ride-sharing primarily suffer from the CoD in the critic network, an issue that cannot be mitigated by a more powerful feature extractor. We believe that the improvements observed in DeepPool, BMG-Q and our Triple- BERT effectively enhance the efficiency our designed encoder. Furthermore, even with the ARL-based encoder, our TriPle-BERT consistently outperforms all previous methods, underscoring the superiority of our approach. Additionally, even with the simple MLP-based encoder, Triple-BERS still outperforms previous MARL methods, illustrating the robustness of centralized SARL method.
Table 10: Reward of Different Methods Under Different Encoder
Encoder
DeepPool
I
BMG-Q 23
HIVES [6
Enders al.
CEVD
Triple-BERT BERT
Vanilla MLP
13,332 14,570
13,539 13,879
13,126 13,095
12,670 12,706
13,746 13,724
14,967
ARL-based
15,388
25

---

Published as a conference paper at ICLR 2026
F RELATED WORK
F.1
ORDER DISPATCH IN RIDE-SHARE SHARING TASK
Order dispatch methods in ride-sharing can be primarily categorized into model-based and RL-bbasdd approaches. Model-based methods often rely on early assumptions that all order information is known in advance [59] or neglect potential future dynamics, resulting impractical or myopic solutions. Later research focused modeling and capturing environmental uncertainties for practical applications (2]. However, accurately characterizing these complexities in the ride-sharing market remains challenging task.
In contrast, model-free RL methods free researchers from these constraints, allowing agents to interact with and learn from the environment independently. Given the vast action and observation spaces, most studies adopt a MARL paradigm (some referring to their methods as decentralized RL) to effectively manage these challenges (37]. Similar standard MARL, the methods adapted for ride-sharing can be divided into Decentralized Execution (DTDE, also known as independent MARL), Centralized Training with Decentralized Execution (CTDE), and Centralized Training with Centralized Execution (CTCE) (24). DTDE methods, while widely used in early studies [[] IO], suffer from low cooperation since each agent perceives others merely as part of the environment, often leading to instability during training. Hu et al. (3 introduced the GAT [5T] to enable each agent to consider the information of neighbors, thereby reducing overestimation issues and improving system performance. Other studies have shifted towards CTDE and CTCE paradigms to promote effective cooperation. For instance, Enders et al. [9] proposed delayed matching method based on MASAC [14), allowing agents to decide whether to accept assigned orders at each step. Bose et al. 3 developed a VD-based method [45) that utilizes a global reward to foster cooperation, although faces the challenge of lazy agents. Furthermore, Hao et al. [16] applied QMIX (38), while Hoppe al. [19 employed COMA [I]] alongside a mix of global and local rewards to address existing issues. However, many of these centralized methods require a centralized critic, reintroducing the challenge of CoD. To address this, Li et al. [26] proposed a Mean Field MARI. framework [60), and Zhao et al. (66) developed the GRPO 39 and OSPO methods based on the homogencous properties among agents, although these assumptions are not always met in practice.
Additionally, several studies began explore more practical scenarios that integrate order dispatch with other tasks, such as repositioning, price setting, and multi- transportation 51 For example, Zhang et al. [63] considered the joint optimization of order dispatch and price setting, which can significantly influence customer demand. Similarly, Haliem et al. [15] and Ge et al. [3 examined repositioning strategies based on these factors. Hu al. 21 22 analyzed order assignment in joint delivery scenarios involving cars, subways, and Unmanned Aerial Vehicles (UAVs), while other works, such as Singh et al. [41]. have studied multi-hop transportation. Furthermore, some studies have examined fairness considerations. For instance, Zhang al. (62) introduced mutual information in the reward function to address challenges related to unusual order distributions and improve platform income. Zhao et al. 67 investigated algorithmic discrimination by considering joint order assignments and payment settings for workers.
Overall, current MARL-based ride-sharing methods face a fundamental dilemma: centralized training paradigms foster cooperation at the cost of exacerbating the CoD, whereas decentralized approaches mitigate CoD but suffer from non-stationarity and poor coordination, ultimately limiting system performance. To navigate this dilemma, our paper offers a novel perspective by directly utilizing centralized SARL framework leverage complete global information, where we then develop efficient techniques, such as action decomposition, to directly address the ensuing CoD challenge.
F.2 COOPERATIVE MULTI-AGENT REINFORCEMENT LEARNING
Cooperative MARL has a wide range of applications, including power control, robot fleet man- agement, and ride- side-sharing tasks [61]. According recent survey B55, these methods can be categorized into five types: independent learning, centralized critic, value decomposition, consensus, and communication.
'To avoid confusion, we would like to clarify that the concept of multi-modal in transportation differs from that in computer science. In transportation, "multi-modal" refers to travel model that utilizes multiple vehicles, such as taxis, buses, and subways, regardless of the model input.
26

---

Published as a conference paper at ICLR 2026
staly-stage works primarily focused independent learners, which represent the simplest realization of MARL. By viewing each agent as an independent entity and others as part of the environment, standard SARL methods can be easily transferred to MARL scenarios. However, this approach can lead to unstable environments, as the policies of other agents continuously change. Additionally, it often results in local optima, each agent aims to maximize its own reward, neglecting the cooperation necessary to optimize global reward. Despite efforts such as Hysteretic Q-Learning [32] challenges remain in large-scale environments and sparse reward situations.
To overcome these challenges, the CTDE and CTCE paradigms have been widely explored in centralized critic and value decomposition MARL methods. In the case of centralized critics, early research focused on adapting actor-critic methods like PPO, SAC, and DDPG by replacing the critic with a centralized one. During training. the critic receives input from all agents, thus addressing the instability problem, although it is not needed during execution, allowing actors to function independently. To further enhance this paradigm, attention-based critics were introduced to capture the relationships among agents BII inspired by the Transformer architecture [50]. Foerster et al. M proposed COMA to resolve the credit assignment problem by introducing a counterfactual baseline, which underwent further improvements in subsequent research (25). Additionally, value decomposition methods aim to effectively distribute global rewards among agents, enabling them to optimize the overall reward rather than just their individual rewards. Starting from VD (45), which struggles with lazy agents due to the simple addition of Q-values, many efforts have been made to enhance the representation capacity of functions that combine global and individual Q- values. Notable examples include QMIX (38), VDAC [43), and QTRAN [42). However, most of these methods encounter the challenge of CoD when the number of agents increases, particularly in scenarios like our ride-sharing task, where the number of agents can exceed hundreds or even thousands.
Consensus and communication methods, developed later, strive to find a balance between low co- operation and the CoD challenge. Here, agents only exchange information with their neighbors or selected agents. Consensus based methods utilize sparse communication to achieve policy consensus among agents, often with convergence guarantees under linear approximators [49]. However, many of these methods often rely on multiple rounds of communication, leading to practical challenges in the ride-sharing task, which has high demands for real-time performance. contrast, communication- based methods focus on designing efficient mechanisms for determining what information send and to whom. For instance, Sukhbaatar et al. 44 proposed CommNet, which broadcasts each agent's hidden features derived from their observations. However, similar Mean Field approaches [60) in MARL, this method considers only the average influence of others, neglecting the specific relationships among agents. Subsequent attention-
methods were introduced evaluate the significance of information from different agents [7] [27]. However, the training difficulties associated with communication based methods are relatively high, particularly as etrly-stage communication of- ten yields little meaningful information. Furthermore, many communication methods face a trade-off between cooperation performance, communication costs, and the content of communication-issuce closely related to CoD. Even recent methods focusing on efficient communication mechanism design (55) still encounter challenges our ride-sharing scenario with extremely high agent volumes. For example, [88] proposed novel communication protocol based exponential graphs, guaranteeing global message exchange within [log2 (N 1)] steps. However, this number can reach 10 in our experiment with 1,000 agents, which is relatively high compared to the total time step horizon of 30. Additionally, communication- based methods represent an ongoing challenge, as determining what content to communicate and whom poses risks associated with credit assignment. Despite these advancements, most consensus and communication methods still encounter challenges such as credit assignment and a lack of convergence theory. In contrast, the SARL-based method circunvents these problems by directly capturing global information. Given that ride-sharing is fundamentally a centralized decision task, we consider the based method to be a promising solution that overcomes the limitations of previous MARI- approuches.
27

---

Published as a conference paper at ICLR 2026
Triple-BERT
BMG-Q
Triple-BRRT w/o stagel
HIVES
Triple-BERT w/o normalization
Enders et al.
DeepPool
CEVD
(a) Legend
Comparison of Reward Across Models
15000
Reward
10000
5000
2850
3114
3577 Order
3910
4264
(b) Accumulative Reward
Comparison of Served Rate Across Models
1.0-
0.8
Served Rate
0.6
0.4
0.2-
2850
3114
3577
3910
4264
Order
(c) Service Rate
Comparison of Delivery Time Across Models
12-
Delivery Time
10
4-
2850
3114
3577 Order
3910
4264
(d) Delivery Time
Figure 6: Detailed Evaluation Results
28

---

Published as a conference paper at ICLR 2026
Comparison Detour Time Across Models
Detour Time
2-
2850
3114
3577 Order
3910
4264
(c) Detour Time
Comparison of Pickup Time Across Models
10
Pickup Time
6
2
2850
3114
3577 Order
3910
4264
(1) Pickup Time
Comparison of Confirmation Time Across Models
Confirmation Time
1.5
1.0
0.5
0.0
2850
3114
3577 Order
3910
4264
(g) Confirmation Time
Figure 6: Detailed Evaluation Results
29


Reviews:
[{"review_id": "PZ0KRqi4gD", "reviewer": "Reviewer_vF6k", "summary": "This paper proposes Triple-BERT, a centralized single-agent reinforcement learning (SARL) framework for large-scale ride-sharing order dispatch. It addresses MARL limitations (e.g., poor cooperation, high dimensionality) via a BERT-based architecture with self-attention to model driver-order relationships globally, QK-attention to reduce computational complexity, and action decomposition to simplify joint decisions. A two-stage training strategy (decentralized pre-training + centralized fine-tuning) mitigates SARL sample scarcity. Experiments show Triple-BERT outperforms MARL baselines by ~15% in reward, serving more orders with lower latency (<0.2s), and better scalability to thousands of agents.", "questions": "What challenges might be encountered when deploying this framework in practical applications?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- BERT’s self-attention effectively captures global driver-order interactions, enhancing cooperation.\n- QK-attention reduces complexity from multiplicative to additive, enabling real-time dispatch.\n- Handles 1,500–2,000 drivers/orders without retraining, outperforming MARL in high-concurrency scenarios.\n- Two-stage training resolves SARL data scarcity via MARL pre-training.", "weaknesses": "- Action decomposition relies on an approximate \"independent action probability\" assumption, which may not fully reflect inter-dependencies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vc1XXQRG6U", "reviewer": "Reviewer_TpDZ", "summary": "This paper addresses the real-time order dispatching problem in large-scale ride-hailing platforms, where numerous drivers and passengers must be matched dynamically under uncertainty. The authors propose Triple-BERT, a centralized single-agent reinforcement learning approach designed to overcome the scalability and coordination limitations of multi-agent RL (MARL) frameworks. The method extends the TD3 algorithm with (i) an action decomposition strategy that factorizes joint action probabilities and (ii) a BERT-based attention architecture to handle large observation spaces efficiently via parameter reuse and contextual embedding of driver–order relationships.\nExperiments on a real-world Manhattan ride-hailing dataset show improvements of roughly 12% in total performance metrics, with additional gains in served orders (+4.3%) and pickup times (–22%). Code and models are made publicly available.", "questions": "1. How does Triple-BERT compare to MARL approaches that use reward decomposition or credit assignment to improve coordination?\n\n2. Could the authors elaborate on the role of the final bipartite matching step — is it part of the learning policy or a post-processing stabilization?\n\n3. How sensitive are the results to hyperparameters and random seeds? Would results remain consistent under higher variance conditions?\n\n4. Does the model generalize to other cities or datasets beyond Manhattan?\n\n5. What are the main computational trade-offs of the BERT-based encoder compared to simpler attention or graph-based alternatives?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-written and clearly structured: The paper is technically sound, logically organized, and easy to follow. The motivation for replacing MARL with a single-agent formulation is well articulated.\n\n2. Practical relevance: Large-scale order dispatching remains a key benchmark problem in reinforcement learning for logistics and mobility; the paper contributes to this ongoing line of research.\n\n3. Architectural innovation: The integration of a BERT-based network to manage relational structure and parameter reuse is elegant and practically valuable.", "weaknesses": "1. Limited discussion of related MARL coordination strategies: The paper does not engage with recent advances in local/global reward allocation or communication mechanisms that also address coordination challenges in MARL. Without such comparisons, it remains unclear whether the proposed single-agent framework is superior conceptually or primarily an engineering simplification.\n\n2. Scope of novelty: While the combination of TD3 and BERT-based embedding is technically solid, the conceptual contribution lies mainly in neural architecture design rather than introducing a fundamentally new RL principle. The innovation should be framed more as an architectural contribution.\n\n3. Statistical robustness: Results are averaged over only three random seeds, which limits statistical reliability. Given the inherent stochasticity of dispatching and RL training, more repetitions would strengthen confidence in the reported improvements.\n\n4. Clarification on matching procedure: The algorithm ultimately uses a bipartite matching layer for stabilization, suggesting that global coordination is still enforced externally. The authors should clarify to what extent Triple-BERT learns full end-to-end dispatching policies versus providing improved matching scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EMPITr1M5O", "reviewer": "Reviewer_TZpd", "summary": "This paper presents a centralized single agent reinforcement learning method for large-scale order dispatching on ride-sharing platforms. In particular, the authors provide a two-stage training strategy to achieve good performance, i.e., 1) the first stage uses IDDQN for MARL pre-training that yields good feature extraction; 2) the second stage employs TD3 for SARL fine-tuning to get the final order dispatch result. In addition, the network backbone design consists of BERT, MLP and QK-attention. Experimental results validated the effectiveness of the proposed method.", "questions": "No.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The motivation is well presented of using a two-stage training strategy for order dispatch on ride-sharing platforms.\n\n(2) The explanations and illustrations are mostly clear and intuitive of the action space, the actor sub-network, the critic sub-network, and the solving algorithms.", "weaknesses": "(1) The main contribution is the two-stage training strategy but some other possible approaches are not analyzed. In particular, the title claims MARL is not required but used as pre-training, why not directly consider SARL without MARL pre-training.\n\n(2) On Page 5, Line 236-237, the positional embedding is discarded due to permutation invariance of the input sequence. This is not a typical practice in the literature, it could be better if an ablation study is conducted for the difference. Besides, if permutation invariance is important for the whole framework, using GNN to model the input could be a better choice as done in [16,34].\n\n(3) On Page 5, Line 269, this guarantees a unique solution. No proof or explanation on this point.\n\n(4) On Page 6, Line 323-324, the authors assume that unsatisfactory independent MARL can provide a good starting point. This point is not proved or analyzed to some extent.\n\n(5) On Page 6, Line 299-300, the authors claim that IDDQN is chosen for the stage-1 pre-training. However, in Table 1, the authors present the proposed Triple-BERT uses TD3 as a difference to comparison methods. It is a bit confusing since the whole framework is a two-stage approach and it is inappropriate to only highlight the stage-2 as a contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFHd77YHnw", "reviewer": "Reviewer_PcM6", "summary": "This paper proposes Triple-BERT, a centralized single-agent reinforcement learning approach tailored for the order dispatch problem on ride-sharing platforms, aiming to address the challenges faced by traditional multi-agent reinforcement learning (MARL) methods in handling large-scale order dispatch scenarios. The paper introduces an innovative network architecture and training strategy that significantly enhances the efficiency of order dispatch and platform performance. Overall, the research is innovative, and the experimental results are impressive, offering substantial theoretical and practical contributions to the ride-sharing domain.", "questions": "No", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Clear Problem Definition\n2. Triple-BERT innovatively addresses the large action and observation spaces through an action decomposition strategy and BERT network  The incorporation of the QK-Attention mechanism significantly reduces computational complexity.", "weaknesses": "While the experimental results are significant, the theoretical analysis in the paper is relatively weak. For instance, a deeper exploration of why the action decomposition strategy works effectively and its theoretical foundations could be further elaborated.\nIn the paper, the literature review on related work appears to be somewhat limited. For instance, significant contributions such as \"Future Aware Pricing and Matching for Sustainable On-Demand Ride Pooling\" , and studies exploring \"MARL-Based Pricing Strategy via Mutual Attention for Mobility-on-Demand (MoD) Systems with Ridesharing and Repositioning,\" as well as \"Mutual Information as Intrinsic Reward of Reinforcement Learning Agents for On-Demand Ride Pooling,\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)