SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yiheng Zhu $^{1}$ , Yang Zhan $^{2,3*}$ , Xuankun Huang $^{1}$ , Yuwei Chen $^{1}$ , Yujie Chen $^{1}$ , Jiangwen Wei $^{1}$ , Wei Feng $^{1}$ , Yinzhi Zhou $^{1}$ , Haoyuan Hu $^{1}$ , Jieping Ye $^{4}$
<sup>1</sup>Department of Artificial Intelligence, Cainiao Network.
<sup>2</sup>Zhejiang University of Technology. <sup>3</sup>Hailiang Group Co., Ltd. <sup>4</sup> DAMO Academy.
{zyh171911,huangxuankun.hxk,chenyuwei.chenyuwe,aisling.cyj,
jiangwen.wjw,fw222241,yinzhi.zyz,haoyuan.huhy}@cainiao.com,
zhanyang@zjut.edu.cn, yejieping.ye@alibaba-inc.com
The dramatic growth of global e-commerce has led to a surge in demand for efficient and cost-effective order fulfillment which can increase customers' service levels and sellers' competitiveness. However, managing order fulfillment is challenging due to a series of interdependent online sequential decision-making problems. To clear this hurdle, rather than solving the problems separately as attempted in some recent researches, this paper proposes a method based on multi-agent reinforcement learning to integratively solve the series of interconnected problems, encompassing order handling, packing and pickup, storage, order consolidation, and last-mile delivery. In particular, we model the integrated problem as a Markov game, wherein a team of agents learns a joint policy via interacting with a simulated environment. Since no simulated environment supporting the complete order fulfillment problem exists, we devise Order Fulfillment COoperative mUlti-agent Reinforcement learning Scalable Environment (OFCOURSE) in the OpenAI Gym style, which allows reproduction and re-utilization to build customized applications. By constructing the fulfillment system in OFCOURSE, we optimize a joint policy that solves the integrated problem, facilitating sequential order-wise operations across all fulfillment units and minimizing the total cost of fulfilling all orders within the promised time. With OFCOURSE, we also demonstrate that the joint policy learned by multi-agent reinforcement learning outperforms the combination of locally optimal policies. The source code of OFCOURSE is available at: https://github.com/GitYiheng/ofcourse.
The growth of e-commerce has increased the need for efficient and cost-effective Order Fulfillment (OF) services which are essential for upgrading customers' service levels and enhancing the sellers' competitiveness [Houde et al., 2017]. OF covers the entire process from order placement to order delivery, involving a series of online sequential decision-making problems across different fulfillment stages, including order handling, packing and pickup, storage, order consolidation, and last-mile delivery [Lin and Shaw, 1998, Croxton, 2003]. To reduce the overall cost of the complete OF problem, it is pivotal to approach it holistically, which poses two challenges. Firstly, order information is revealed progressively over time, while decisions must be made once the requests arrive without future information. Secondly, as OF involves managing numerous orders over multiple interconnected fulfillment units, all constituent decision-making stages are interdependent and require collective
* Yang Zhan (zhanyang@zjut.edu.cn) is the corresponding author.
consideration. Recent works have revealed that Multi-Agent Reinforcement Learning (MARL) [Zhang et al., 2021] is a well-established paradigm for handling these challenges with interacting constituents. However, due to the lack of a standard MARL framework for the complete OF problem, previous works often focused on solving individual OF subproblems, as reviewed in Section 2 and compared in Appendix C.
To enable the customization for various OF problems and facilitate the deployment of future MARL algorithms, this paper proposes a MARL framework for the complete OF problem. Specifically, we formulate the complete OF problem as a Markov Game (MG) where a team of fulfillment agents collectively learns to minimize the global fulfillment cost by interacting with an environment [Busoniu et al., 2008], as detailed in Section 3. To support this, we develop a simulated environment called OFCOURSE (Order Fulfillment COoperative mUlti-agent Reinforcement learning Scalable Environment). As described in Section 4, OFCOURSE conforms to the OpenAI Gym format [Brockman et al., 2016, Koul, 2019, Terry et al., 2021] and takes into account the heterogeneity, scalability, and variability in OF problems. In Section 5, we show how OFCOURSE can be used to customize OF problems in diverse contexts. Moreover, by conducting computational experiments, we demonstrate that the joint policy learned via MARL outperforms the combination of locally optimal policies.
Order Fulfillment (OF) problems have been extensively studied in the fields of operations management [Liu et al., 2022a, Gopalakrishnan et al., 2023], supply chain management [Yang et al., 2021], operations research [Bayram and Cesaret, 2021, Jiang et al., 2022], and a recent one — machine learning. Here we focus on reviewing the stream that applies Reinforcement Learning (RL) techniques to solve OF problems, which are closely related to this paper.
A large number of works have proposed RL-based methods to solve decision-making problems in the complete OF. For example, to determine the processing sequences of orders (known as scheduling) in different processing units (warehouse pickers) in the order handling stage of OF, Zhang et al. [2020] used graph isomorphism networks for feature embedding and proximal policy optimization for policy learning to solve the problem in the static context. Liu et al. [2022b] solved the dynamic job shop scheduling problem using double deep Q-networks. To select a container (box) and determine the positions of items within the containers in the packing and pickup stage, Hu et al. [2017] employed a pointer network to learn the optimal sequence of items, and Duan et al. [2019] extended this approach by learning the orientations in a multi-task fashion. To ensure the appropriate inventory levels to meet the demand (known as the inventory management problem) in the storage stage of OF, Oroojlooyjadid et al. [2022] learned to optimize the solution using the deep Q-network. Later, De Moor et al. [2022] augmented this deep Q-network approach by reward shaping using existing well-performing heuristics to solve perishable inventory problems. To solve the well-known vehicle routing problem, which assigns orders to vehicles and determines the optimal route of each vehicle in the last-mile delivery stage of OF, Nazari et al. [2018] developed a policy network using a recurrent neural network decoder with the attention mechanism. James et al. [2019] proposed a model based on a graph convolutional network encoder and a recurrent neural network decoder to solve the online version of the vehicle routing problem, where the delivery requests were not static but arrived progressively. Additionally, Duan et al. [2020] constructed a model based on the graph convolutional network and trained the model using a hybrid learning method.
The above pioneering studies have demonstrated that RL is a promising approach to handle the complexity of OF problems. However, they only focused on addressing individual OF subproblems (refer to Appendix C for a comparison between precedent formulations and our contribution), which may yield suboptimal solutions when viewed from a holistic perspective. Additionally, they did not construct RL environments in standard formats, such as the OpenAI Gym style [Brockman et al., 2016, Koul, 2019, Terry et al., 2021], making it difficult to reproduce the results and reuse them to build customized applications.
Recently, some works have open-sourced standard RL format environments that covered a portion of the subproblems of OF. For example, Balaji et al. [2020] benchmarked online bin-packing, news vendor, and vehicle routing problems. Later, Hubbs et al. [2020] enriched this benchmark to incorporate knapsack, multi-period asset allocation, multi-echelon supply chain management, and
virtual machine assignment problems. However, they only supported single-agent RL, which cannot jointly handle multiple decision-making problems in OF. This paper contributes to the literature by filling the research gap of collectively solving the series of decision problems in OF and creating a standard OF environment in the OpenAI Gym format, which facilitates the deployment of future MARL algorithms in solving various OF problems.
3.1 Order Fulfillment Problem
Order fulfillment involves delivering ordered items over a fulfillment system, with the aim to minimize the overall fulfillment cost within the promised time [Lin and Shaw, 1998, Croxton, 2003].
For an individual order, once a customer places it online, a record is created on the e-commerce platform server. Right after that, the order progresses to the order handling stage where it accumulates processing time and incurs costs related to server maintenance and customer service. Next, the order information arrives at the seller who packs the ordered items where a packing scheme is chosen from various options with different prices and processing time. Starting from there, the parcel is picked up by an agency, with multiple agencies available at various prices and time, and a decision is made on which to choose. Then, at the consolidation stage, the parcel can wait to be consolidated with others going to the same destination, taking longer but saving downstream costs. After that, the parcel undergoes warehouse storage, where a decision is made regarding whether to delay the delivery in order to free up logistics partner capacities at the next stage for other urgent parcels. At the last-mile delivery stage, a decision is made to select the most suitable logistics partner for delivery from multiple candidates offering different prices and delivery time. There are numerous such orders in a complete OF system.
In reality, OF systems vary significantly depending on the context, and their characteristics can be summarized as follows.
(i) Heterogeneity: Fulfillment stages are heterogeneous, as the aims and operations vary across different fulfillment stages. For example, the operations in the consolidation stage involve either waiting or consolidating existing orders, while the operations in the last-mile delivery stage correspond to different logistics partners to choose from. It is worth noting that even within the same stage, the fulfillment units from different fulfillment agents may not be identical. Taking the delivery stage for example, one fulfillment agent offers air transport as a candidate operation, while another fulfillment agent offers sea transport, resulting in significant differences in terms of cost and time.
(ii) Scalability: Fulfillment systems come in various sizes depending on the business contexts. For example, when it comes to domestic fulfillment processes that fulfill orders within a country, there are typically three fulfillment stages involved: local customers placing orders online, the e-commerce platform notifying the sellers, and the sellers delivering the ordered items directly to the customers' doorstep. On the other hand, a cross-border fulfillment process requires additional fulfillment stages such as the collection warehouse, distribution center, consolidation warehouse, line-haul warehouse, and more.
(iii) Variability: The fulfillment system is subject to change due to the constant variation of fulfillment resources. Fulfillment resources may become unavailable, as exemplified by cases where order processing servers go offline owing to policy or technical issues. Additionally, the set of available actions can also vary dynamically. For instance, last-mile delivery may be expanded to include an additional logistics partner after the onboarding of a new company.
To effectively model diverse OF tasks, a desirable MARL framework needs to account for the characteristics of heterogeneity, scalability, and variability. In Section 4, our proposed framework addresses these aspects, thereby enabling a comprehensive representation of diverse OF tasks.
Before elaborating on our approach, we first introduce key concepts and notations. In a fulfillment system, we define the fundamental building block as a fulfillment unit. For example, in Figure 1, the fulfillment unit in the order placement stage (the leftmost blue box) encapsulates the action of [orderCreation] and observation of order information for the 2 illustrated orders. As illustrated in Figure 1, a fulfillment agent,  $i \in \mathcal{I}$ , connects fulfillment units into a fulfillment route, with each fulfillment unit realizing a fulfillment stage. Its observation space (the red box in Figure 1),
Figure 1: An example order fulfillment agent. The observation is highlighted by a red box and the action is emphasized by a green box. All fulfillment stages are marked by blue boxes.
$o^i$ , combines order information across all of its fulfillment units. For example, this observation space can be represented by a feature matrix of size [num_unit, num_order, order_feat], where num_unit indicates the number of fulfillment units (8 fulfillment units in Figure 1), num_order signifies the current number of orders in the agent ([2, 1, 1, 2, 5, 9, 2, 6] in Figure 1) and order_feat denotes the feature dimension of each order. Its action space (the green box in Figure 1),  $a^i$ , concatenates the discrete actions across all of its fulfillment units. For example, in the pickup stage in Figure 1, the agent can select pickup_agency_B from the available operations of [pickup_agency_A, pickup_agency_B, pickup_agency_C]. Accordingly, at a decision step, the action of a fulfillment agent encompasses such immediate actions pertaining to all its carried orders (28 orders in Figure 1). A fulfillment system consists of  $|\mathcal{I}| = L$  such fulfillment agents, where each fulfillment agent is responsible for making decisions for a specific portion of the orders. Here, orders compete for recourses across multiple fulfillment stages, so agents interact accordingly. In order to solve the complete OF problem by  $L$  interacting agents, we need to find a joint policy, denoted as  $\pi(a_t|o_t) = \prod_{i=1}^{L}\pi^i(a_t^i|o_t^i)$  where  $t$  is the decision step, that makes a series of decisions along the fulfillment system as described above in the MARL framework.
Problem Definition: Order fulfillment involves transferring  $|\mathcal{N}|$  orders from source to target fulfillment units in  $T$  discrete steps. Order  $j \in \mathcal{N}$ , with promised fulfillment time  $\tau_j$ , is created at its source fulfillment unit at step  $t_j^{init}$  and arrives at its target fulfillment unit at step  $t_j^{trml}$ . The total fulfillment cost of order  $j$  is  $c_j = \sum_{t = t_j^{init}}^{t_j^{trml}}c_{j,t} + p_j$  where  $c_{j,t}$  is the step cost and  $p_j$  is the penalty if  $t_j^{trml} - t_j^{init} > \tau_j$ . The objective of order fulfillment is to find the optimal (joint) policy to minimize the fulfillment cost of these  $|\mathcal{N}|$  orders,  $\pi \gets \mathrm{argmin}_{\pi}\sum_{j = 1}^{|N|}c_j$ .
3.2 Markov Game
The complete OF problem can be formulated as a Markov Game (MG) [Littman, 1994] based on the enumerated characteristics and notations in Section 3.1. In the literature, similar concepts of MG have been referred to as stochastic game [Shapley, 1953], multi-agent Markov decision processes [Boutelier, 1996, Lauer and Riedmiller, 2000], and Markov teams [Ho, 1980, Wang and Sandholm, 2002]. There has been a significant amount of research on a specific type of Markov game (MG) that involves partial observability in a cooperative setting, which is known as the decentralized partially observable Markov decision process (Dec-POMDP) [Bernstein et al., 2002, Oliehoek and Amato, 2016]. In this paper, to cover various OF scenarios, we formulate it as a general MG that is denoted by a tuple  $\mathcal{M} = \langle \mathcal{I}, S, \rho_0, \mathcal{A}, P, R, \mathcal{O}, \Omega, h, \gamma \rangle$ , where  $\mathcal{I}$  is a set of  $L$  agents with a specific agent denoted by  $i \in \mathcal{I}$ ,  $\mathcal{S}$  signifies the state space,  $\rho_0 \subset \mathcal{S}$  represents the initial state distribution,  $\mathcal{A} = \prod_{i=1}^{L} \mathcal{A}^i$  denotes the joint action space,  $P: S \times \mathcal{A} \times S \to [0,1]$  is the transition function,  $R: S \times \mathcal{A} \to \mathbb{R}^L$  represents the reward function,  $\mathcal{O} = \prod_{i=1}^{L} \mathcal{O}^i$  denotes the joint observation space,  $\Omega: S \times \mathcal{O} \to [0,1]$  is the observation function,  $h \in \mathbb{N}^+$  is the time horizon, and  $\gamma \in [0,1)$  signifies the discount factor. For a given MG  $\mathcal{M}$ , the MARL objective is to find a joint policy  $\pi: \mathcal{O} \times \mathcal{A} \to [0,1]$  to maximize the objective function  $J(\pi) = \mathbb{E}_{s_{0:h} \sim \rho_{\pi}, a_{1:h} \sim \pi}[\sum_{t=1}^{h} \gamma^t r_t]$ , where  $\rho_{\pi}(s) = \sum_{t=0}^{h} \gamma^t P(s_t = s)$  represents the discounted visitation frequencies, and  $r_t = R(s_t, a_t | s_{t-1})$  denotes the joint reward
that is equivalent to the negative joint fulfillment cost (detailed calculation is provided in Section 4.5). To elaborate, an MG trajectory starts from initial state  $s_0 \in \rho_0$ . At step  $t \in [1,h]$ , facing state  $s_{t - 1} \in S$ , the team of agents jointly observe  $\pmb{o}_t = [o_t^1,\dots,o_t^L ] \in \mathcal{O}$  via  $\Omega (\pmb{o}_t|s_{t - 1})$ . Based on this joint observation  $\pmb{o}_t$ , the team of agents samples a joint action  $\pmb{a}_{t} = [a_{t}^{1},\dots,a_{t}^{L}] \in \pmb{A}$  according to the joint policy  $\pi (\pmb {a}_t|\pmb {o}_t) = \prod_{i = 1}^{L}\pi^i (a_t^i |o_t^i)$ . Simultaneously, the agent team receives a joint reward  $\pmb{r}_t = [r_t^1,\dots,r_t^L ] \in \mathbb{R}^L$  and transits to the next state  $s_t$  with probability  $P(s_{t}|s_{t - 1},\pmb{a}_{t})$ .
4 OFCOURSE
Figure 2: An example fulfillment system in OFCOURSE. A fulfillment unit (gray) is the basic building block, containing operations (green) and containers. Inventories (orange) and buffers (blue) are two types of containers. Orders are held in containers and transferred between containers via operations.
To address the complete OF problem by MARL, we are motivated to design an environment for the agent team to interact with. Considering the three characteristics described in Section 3.1, we devise Order Fulfillment COoperative mUlti-agent Reinforcement learning Scalable Environment (OFCOURSE) with three corresponding functionalities. (i) Time-price metrics: Different fulfillment units can have distinct properties. Accordingly, OFCOURSE focuses on recording the price and time, while neglecting other inconsistent details. Specifically, OFCOURSE maintains cumulative price and time for each order, updating them at every simulation step. (ii) Modular design: Fulfillment systems can vary significantly in scale. To accommodate this, OFCOURSE adopts a modular design approach, allowing the construction of fulfillment systems for different scenarios using basic building modules. (iii) Step-wise update: The status of fulfillment units is subject to constant change. To cope with this, OFCOURSE supports step-wise updates for module variables. The components of OFCOURSE will be described in detail in the remaining parts of this section.
4.1 Order
Figure 3: A typical order. The goal of order fulfillment is to minimize the overall cost of delivering all orders within promised time. Accordingly, OFCOURSE focuses on keeping track of the cumulative price and time, while neglecting other inconsistent details.
An order is the embodiment of a customer's demand, which can be either virtual or physical. For example, movies or music can be considered virtual orders, while cosmetics or electronics are physical orders. With the objective of minimizing the total cost of fulfilling all orders within the promised time, OFCOURSE keeps track of prices and time for all orders, as Figure 3 illustrates. As the time counter for an order increments at simulation step  $t$ , its step-wise financial cost is calculated by  $c_{t} = c_{t}^{\mathrm{ctr}} + c_{t}^{\mathrm{op}}$  where  $c_{t}^{\mathrm{ctr}}$  denotes the container-related cost and  $c_{t}^{\mathrm{op}}$  represents the operation-associated cost. With regard to the container cost, an order incurs charges for being held by a specific container, which
depends on the container's quote based on its occupancy status. An example of this cost is the storage fee at a warehouse. As for the operation cost, an order incurs charges for being transferred from one container to another. For example, an e-commerce platform may charge for order processing and customer service.
4.2 Container
A container serves as the carrier of orders. Each container is capable of holding multiple orders, but each individual order can only be held by one container. For example, a server can act as a container for virtual orders such as movies, while a warehouse shelf can serve as a container for tangible orders like laptops.
Resource: In reality, the resources associated with containers are not unlimited. For example, the storage space in a warehouse can be exhausted. However, it is possible to rent additional storage space at a higher cost. Considering this aspect, we assign a specific resource to each container, where the same resource can be shared among multiple containers. For instance, orders traveling along different routes can share the same vehicle for a portion of their transportation. At simulation step  $t$ , the quote of a container depends on its resource usage, denoted as  $c_t^{\mathrm{ctr}} = g(u_t^{\mathrm{ctr}})$ , where  $u_t^{\mathrm{ctr}}$  represents the container's usage, and  $g(\cdot)$  is a nonlinear function that calculates the quote based on the usage.
Figure 4: Buffer mechanism in OFCOURSE. A buffer consists of multiple chunks, where each chunk can hold multiple orders. At each simulation step, the orders in a chunk are transferred to its successor chunk.
Buffer and Inventory: There are two types of containers in OFCOURSE: buffers and inventories. Specifically, buffers carry geographically or procedurally dynamic orders, while inventories hold static ones. Conventionally, the estimated time of arrival (ETA) [Wang et al., 2018] is regarded as an order-wise feature. In OFCOURSE, the ETA of an order is indicated by its located chunk in a buffer. Figure 4 demonstrates the structure of a buffer, which consists of a sequence of chunks  $(B_{T},\ldots ,B_{T + n},B_{O})$ . At simulation step  $T = t$ , only the orders in chunk  $B_{t}$  are operable. The orders in chunk  $B_{t + n}$  reach  $B_{t}$  after  $n$  simulation steps. Meanwhile, outlier orders are temporarily stored in chunk  $B_{O}$ . Due to the heterogeneity of the fulfillment system, certain orders are processed in batches, some orders are handled sequentially, and others are handled individually. Using discrete buffers is a design compromise that ensures compatibility across heterogeneous fulfillment units. Additionally, using discrete buffers helps to preserve the morphological information of the fulfillment system. Another type of container is the inventory, which holds static orders. Orders stored in an inventory are subjected to recurring fees charged at each simulation step. For example, the storage fee in a warehouse is charged on a daily basis.
4.3 Operation
An operation is an abstract action of transferring orders between containers. For example, transporting an ordered item from a regional hub to a last-mile warehouse can be outsourced to different third-party logistics (3PL) companies. Consequently, the decision problem in OFCOURSE involves selecting a 3PL company from a pool of candidates, with each company represented as an individual operation.
Figure 5: An example fulfillment unit with three operations in OFCOURSE. Activated operations are highlighted with bold outlines, and destination containers are colored in purple. (Left) All available orders are transferred to the inventory. (Middle) All available orders are injected into the corresponding buffer chunk of its successor fulfillment unit. (Right) Same as the middle operation.