SlowGuess/NeurIPS_2023_OCR huggingface dataset:
4.4 Fulfillment Unit
A fulfillment unit consists of containers and operations. For instance, a fulfillment unit can be a physical warehouse or a virtual e-commerce platform. Figure 5 shows an example fulfillment unit with 3 containers and 3 operations. The top container serves as an inventory, while the middle and bottom containers function as buffers. At each simulation step, only one operation can be selected. The top operation receives orders from both buffers and then stores them in the inventory. The middle operation gathers orders from all 3 containers, consolidates them, and sends them to its connected buffer chunk. Similarly, the bottom operation collects orders from all 3 containers, combines them, and dispatches them to the associated buffer chunk.
4.5 Fulfillment Agent
Figure 6: An example fulfillment agent in OFCOURSE, consisting of 5 cascaded fulfillment units.
A fulfillment agent is the decision-making entity in OFCOURSE. In a fulfillment system, the concatenation of all fulfillment units along an order route is defined as a fulfillment agent.
Action Space: The action space of a fulfillment agent consists of a list of discrete actions from its constituent fulfillment units. Specifically, fulfillment unit  $i$  has a finite set of actions, denoted as  $\mathcal{A}_i = \{a_1, \dots, a_{n_i}\}$ , meaning it has a Discrete action space of size  $|\mathcal{A}_i| = n_i$ . For a fulfillment agent composed of  $m$  constituent fulfillment units, its action space is the Cartesian product of the action spaces of each constituent fulfillment unit, expressed as  $\mathcal{A}_1 \times \ldots \times \mathcal{A}_m$ . Consequently, this fulfillment agent has a MultiDiscrete [Brockman et al., 2016] action space of size  $|\mathcal{A}_1 \times \ldots \times \mathcal{A}_m| = \prod_{i=1}^{m} n_i$ . As an example, consider the fulfillment agent represented in Figure 6, whose action space has a shape of  $[1, 1, 2, 3, 1]$ . In this case, the fulfillment unit with 1 operation has a dummy action, indicating that it can only select this specific operation in this fulfillment unit. Meanwhile, the fulfillment unit with 2 operations and the fulfillment unit with 3 operations provide multiple candidates to choose from.
Observation Space: The observation space of a fulfillment agent combines the observation spaces of its constituent fulfillment units. This Box [Brockman et al., 2016] observation space can be represented as  $o^{\mathrm{fa}} = [o_1^{\mathrm{fu}} + \dots + o_m^{\mathrm{fu}}}$ , where  $o^{\mathrm{fa}}$  refers to the observation of the fulfillment agent,  $o_i^{\mathrm{fu}}$  represents the observation of the  $i^{th}$  fulfillment unit, and  $+$  is the concatenation operator. For the  $i^{th}$  fulfillment unit, its observation consists of the combined information from all the containers and operators it includes, which is expressed as  $o_i^{\mathrm{fu}} = [o_i^{\mathrm{ctr}} + o_i^{\mathrm{op}}}$ , where  $o_i^{\mathrm{ctr}}$  indicates the observation of the containers and  $o_i^{\mathrm{op}}$  denotes the observation of the operations for the  $i^{th}$  fulfillment unit. To ensure flexibility, the observation of an operation  $o_i^{\mathrm{op}}$  includes only its price and time. On the other hand, the observation of a container  $o_i^{\mathrm{ctr}}$  includes the status of all the orders it carries, capturing the cumulative prices and time.
Reward Function: The reward reflects the goal of OF, which is to minimize the total cost of fulfilling all orders within the promised time. At simulation step  $t$ , we calculated the instant financial cost of the  $k^{th}$  fulfillment agent as  $c_{t}^{k} = \sum_{i=1}^{m_{k}} \sum_{j=1}^{n_{ik}} c_{t}^{ijk}$ , where  $m_{k}$  represents the number of fulfillment units,  $n_{ik}$  signifies the number of orders in the  $i^{th}$  fulfillment unit, and  $c_{t}^{ijk}$  denotes the cost of the  $j^{th}$  order in the  $i^{th}$  fulfillment unit of the  $k^{th}$  fulfillment agent. Accordingly, we define the step reward for the  $k^{th}$  fulfillment agent as
$$
r _ {t} ^ {k} = \left\{ \begin{array}{l l} - c _ {t} ^ {k}, & \text {f o r} t \neq h, \\ - c _ {t} ^ {k} - \sum_ {i = 1} ^ {m _ {k}} \sum_ {j = 1} ^ {n _ {i k}} p ^ {i j k} d ^ {i j k}, & \text {f o r} t = h, \end{array} \right.
$$
where  $t = h$  represents the last simulation step,  $p^{ijk}$  is the penalty coefficient for an unfulfilled order, and  $d^{ijk}$  equals to 1 for a failed order and 0 for a successful order.
Transition Function: Following each simulation step, the states of all orders are updated based on the transition function. Orders that are currently being operated are transferred to their designated positions in the target containers. In buffers, orders are moved from the current chunk,  $B_{T + i}$ , to
the next one,  $B_{T + i - 1}$ . In inventories, orders are charged recurring fees in accordance with the quote. Meanwhile, failed orders are handled separately, either by being refunded or fulfilled with priority, with all associated costs being covered by the fulfillment company. Since failed orders cause significantly higher costs, fulfillment agents are incentivized to learn how to avoid them. The implementation of the state transition logic is elucidated in detail with code snippets in Appendix D.
4.6 Fulfillment System Overview
As illustrated in Figure 2, a fulfillment system can be conceptually represented as a directed graph, where vertices denote containers and directed edges represent operations. Containers and operations that are geographically or procedurally adjacent are grouped together into a single fulfillment unit. Based on this grouping, all fulfillment units along a fulfillment route are concatenated to form an individual fulfillment agent. The objective for the collaborative team of fulfillment agents is to transfer orders from source to destination containers at minimum cost while meeting promised timelines. At each simulation step, a fulfillment agent's action includes the operations of all its constituent fulfillment units. The joint action of all fulfillment agents thereby determines the allocation of resource in shared containers.
As a Simulator for Practical Order Fulfillment: In practical scenarios, the fulfillment of an individual order often requires multiple transfers between various locations, both geographically and procedurally. In the OFCOURSE framework, these locations are represented as containers, and the transfers themselves are represented as operations. To capture the synergies that exist across multiple locations, containers that are highly correlated are aggregated into fulfillment agents. To model the diverse range of real-world locations and transfer actions, we focus on recording the metrics of prices and time, allowing for a comprehensive representation of the operational details associated with different types of fulfillment units.
As a Benchmark for Studying MARL Algorithms: The OFCOURSE framework can serve as a benchmark for studying MARL in heterogeneous systems, while also providing insights potentially useful for real-world applications. In OFCOURSE, fulfillment agents with different observation and action spaces collaborate to minimize the overall cost. This heterogeneity allows for the evaluation of MARL algorithms in non-identical as well as decentralized settings. Alternatively, when agent homogeneity is imposed, OFCOURSE enables the modeling of order fulfillment as a mean-field game [Lasry and Lions, 2007, Guo et al., 2019, Subramanian and Mahajan, 2019]. The objective of cost minimization holds practical relevance since the findings from simulations can provide valuable information for real-world applications.
In this section, based on the computational experiments, we show the capability of OFCOURSE in customizing OF tasks in different contexts, as well as the effectiveness of collectively solving a series of decision problems using MARL based on OFCOURSE.
5.1 Various Order Fulfillment Tasks
As our modules cover all the stages and resources that an OF system may involve, various customized fulfillment systems can be constructed using our proposed OFCOURSE framework. We show two examples as follows.
Figure 7: Task 1 — fulfillment of physical and virtual orders in one system by OFCOURSE.
Task 1 — Fulfillment of Physical and Virtual Orders in One System. Orders can be physical or virtual. Taking music consumption for example, consumers can either order a vinyl record or download its digital version. As Figure 7 illustrates, agent 1 fulfills physical orders while agent 2
fulfills virtual orders. Both agents share the fulfillment stages of online order processing and seller notification. Excepting the shared stages, agent 1 has extra fulfillment stages for order transportation, indicating its orders have to go through the consolidation warehouse and last-mile warehouse. In the consolidation warehouse, the ordered items can be stored and consolidated with later ones. By performing order consolidation, multiple ordered items can be sent in a single shipment, which cuts overall cost because shipment fees are normally much higher than storage fees. With respect to the two extra stages of agent 1, it has to choose a 3PL company from a finite list of candidates, where each candidate is abstracted as an operation. Usually, no decision has to be made in fulfilling virtual orders. Consequently, each fulfillment unit in agent 2 only contains a single operation.
Figure 8: Task 2 — cross-border order fulfillment by OFCOURSE.
Task 2 — Cross-Border Order Fulfillment. Cross-border order fulfillment is characterized by its extended and diverse fulfillment stages. Figure 8 shows a cross-border fulfillment system consisting of 6 agents. A customer places an order on the online e-commerce platform, right after which the seller is notified for item collection. This ordered item is then gathered at the nearest collection warehouse and its next station is determined at its associated distribution center. The ordered item can either be sent directly to a line-haul warehouse or go through an extra stage of being consolidated with other items. Next, this ordered item is sent from the departure country to the destination country. Finally, the ordered item is delivered to the door of the customer, which might go through extra stages of storage for cost reduction.
5.2 Effectiveness of MARL with OFCOURSE
To show the effectiveness of MARL with OFCOURSE, we conduct experiments to compare an MARL algorithm with three baseline algorithms on the OF tasks built upon OFCOURSE in Section 5.1. In Section 5.2.1, we briefly describe the MARL method and three baseline methods. The experimental results are discussed in Section 5.2.2. Algorithm details can be found in Appendix A and the experimental setup is provided in Appendix B.
5.2.1 Baseline
Based on the OF environments defined in Section 5.1, we implement three baseline methods for comparison along with an MARL algorithm:
Combination of local optima (CLO) [Gendreau et al., 2010] is a non-learning method. It finds the optimal local action at each fulfillment unit based on the current local observation. Then it combines all these local actions into a global joint action.
Proximal policy optimization (PPO) [Schulman et al., 2017, Yu et al., 2022] is an RL method. A gentle introduction to PPO is provided in Appendix A.2. PPO treats the multi-agent system as a single agent, only considering the joint observation-action space. Then it applies the single-agent RL algorithm to this integrated agent.
Independent PPO (IPPO) [Tan, 1993, Foerster et al., 2018] is an RL method as well. The algorithmic implementation of IPPO is identical to PPO, while IPPO factors the joint observation-action space into local spaces then treats them as completely independent agents. Then it uses standalone neural networks for each individual agent.
Heterogeneous agent PPO (HAPPO) [Kuba et al., 2022] is a theoretically-justified MARL method that supports heterogeneous agents with non-shared policies. HAPPO follows the paradigm of Centralized Training and Decentralized Execution (CTDE) [Kraemer and Banerjee, 2016], allowing for collective learning and coordination during training while still maintaining flexibility and adaptability during execution. The details of HAPPO are elaborated in Appendix A.3.
5.2.2 Result
(a) Task 1 — fulfillment of physical and virtual orders in one system.
(b) Task 2 — cross-border order fulfillment.
Figure 9: Experimental results. In each subplot, the bar chart compares the converged performance, and the line chart illustrates the training curves of the learning-based methods.
Task 1 Results. The experimental results on Task 1 are illustrated in Figure 9a. In general, the converged performance of HAPPO stays in the vicinity of IPPO, both of which exceed the performance of PPO and CLO. The similar performance of HAPPO and IPPO can be attributed to the fact that agent 1 shares no resource with agent 2, so one agent's observation is redundant information for another agent. For PPO, agent 1 might mistake the coincidental correlation between its reward and the observation of agent 2 as causation, causing its lower return compared to HAPPO and IPPO. As for CLO, it always selects the local operation with the lowest cost without considering the overall operation time required, resulting in inferior performance.
Task 2 Results. Figure 9b shows the experimental results on Task 2. Overall, the results demonstrate that HAPPO outperforms the other three methods. Specifically, PPO achieves performance comparable to HAPPO upon convergence, surpassing both IPPO and CLO by a visible margin. It is worth noting that HAPPO and PPO have access to the global state during training, while IPPO and CLO are limited to local observations. The marginally lower performance of PPO compared to HAPPO can potentially be explained by PPO agents erroneously inferring spurious correlations between their own rewards and observations of unrelated agents. On the other hand, the converged performance of IPPO falls between HAPPO/PPO and CLO, which is likely because IPPO agents overlook useful information contained in observations from agents that share the same resources. Similarly, CLO exhibits the worst performance among all the approaches tested. As discussed for Task 1, this can be attributed to CLO optimizing locally without considering the global cost.
In this paper, we presented Order Fulfillment COoperative mUlti-agent Reinforcement learning Scalable Environment (OFCOURSE), which was the first simulated environment that facilitated solving the complete order fulfillment problem by multi-agent reinforcement learning algorithms. Our computational experiments using OFCOURSE indicated that the joint policy learned by multi-agent reinforcement learning algorithms outperformed the combination of locally optimal policies. We hope that our promising results could motivate more academics and industrial practitioners to approach the complete order fulfillment problem using multi-agent reinforcement learning techniques.
This work is supported by the National Natural Science Foundation of China (No. 72201245) and the Zhejiang Province Postdoctoral Research Fund for Excellent Candidates Project (No. ZJ2023148).
Jean-François Houde, Peter Newberry, and Katja Seim. Economies of density in e-commerce: A study of amazon's fulfillment center network. *National Bureau of Economic Research*, 2017.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 2022.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. International Conference on Machine Learning, pages 330-337, 1993.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. Association for the Advancement of Artificial Intelligence, 32(1), 2018.
Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. International Conference on Learning Representations, 2022.
Landon Kraemer and Bikramjit Banerjee. Multi-agent reinforcement learning as a rehearsal for decentralized planning. Neurocomputing, 190:82-94, 2016.
Csaba Szepesvári. Algorithms for reinforcement learning. Synthesis Lectures on Artificial Intelligence and Machine Learning, 4(1):1-103, 2010.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
Marc Peter Deisenroth, Gerhard Neumann, and Jan Peters. A survey on policy search for robotics. Foundations and Trends in Robotics, 2(1-2):1-142, 2013.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-256, 1992.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. International Conference on Machine Learning, pages 267-274, 2002.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. International Conference on Machine Learning, pages 1889-1897, 2015.
David JC MacKay. Information theory, inference and learning algorithms. Cambridge University Press, 2003.
Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3), 1997.
Jonathan Richard Shewchuk. An Introduction to the Conjugate Gradient Method without the Agonizing Pain. Carnegie-Mellon University., 1994.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. International Conference on Learning Representations, 2016.
Abien Fred Agarap. Deep learning using rectified linear units (relu). CoRR, 2018. URL http://arxiv.org/abs/1803.08375.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 2015.