Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a MARL style algorithm to solve the capacity modelling problem for the periodic review inventory management problem.
main_review: Overall, this is a nice paper that addresses a simplified version of a major issue faced by retailers (inventory management and capacity control) as a MARL problem. The approach identifies that in the IM problem - the shared context is only really the resource that needs to be divided between different agents. 

The paper has a nice explanation of the problem - though it suffers from weak baselines (other PPO methods), rather than trying out a few different approaches (dual SGD, admm etc). The data is rich in sales/demand - but seems to be a simple simulation for other relevant parameters (VLT, price, cost etc). So that would definitely require some improvement to see if the MARL approach can generalize to those scenarios

The reward given to the agent resembles a shadow price, but has a hard truncation to respect the capacity. This isnt very realistic and could probably be improved a bit. Finally, it seems to be only for one time period? So with arbitrary stochastic VLTs, how do the authors control for the fact that order arrivals can be later then expected and then blow the capacity?

Finally, why wouldn't one just treat every SKU as an instance from some joint stochastic process (with context of course) - and do a dual sgd loop at the batch level. Is the stochastic game formulation adding much more than that?

Overall I weakly recommend accepting the paper as it addresses a novel way to approach the capacity control problem.
summary_of_the_review: The paper is a nice MARL approach to the IM Capacity Problem. I like the experiments and the modification of the public data used. The baselines are not ideal and the reward function doesn't resemble a real capacity issue - but it is a good step towards a more general IM solution
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: A multi-agent RL algorithm is proposed to deal with the inventory management problem where there are several SKUs. Since items should compete for a limited budget and warehouse space, the replenishment problem is modeled as a multi-agent game and a multi-agent reinforcement learning algorithm is proposed to solve that. The agents use a shared piece of information which is the utilized warehouse space and other shared resources (if any) and rather than that there is no other shared information among them. So, each agent uses that shared information along with its local observation, and local reward and train decentralized agents in an independent learner scheme. The PPO algorithm is used to train the independent policies. The agents share the parameters of the actor and critic to mitigate the computation time and power. The action is selecting one of 15 possible values to multiply to the sum of sales over the last two weeks. The algorithm is evaluated on instances with 5 and 50 SKUs and compared with some benchmarks. 

main_review: Strengths: 
A new MARL algorithm to deal with joint replenishment problem in inventory management is proposed. 

Weaknesses:
The claims of the paper about the performance of the algorithm is not well supported with numerical experiments. Also, there are some questions in the paper that need to be addressed. See below comments for more details. 

Q1- With the introduced shared contextual information, still, the env is non-stationary from the viewpoint of each agent. In other words, if agent $i$ is in state $s_t^i$ and takes $a_t^i$, the distribution of next state $s_{t+1}^i$ and the reward $r_t^i$ heavily depend on the actions of other agents. How do you deal with that? How can one make sure that the training is not hindered with that? In other words, can you guarantee that the training is successful for any data/demand distribution, any number of SKUs, and other characteristics of the problem? 
Q1-1 Same issue exists when you do sampling $c^{-i}^t$ and $(s^i_t; a^i_t; c^i_t)$ from the old policy. 


Q2- Since in theory, the algorithms with independent learners may not work in practice, do you have any intuition why does CD-PPO work on the presented examples? Does the data have any given feature/characteristic? 
Q2-1 In a broader question, can you categorize the type of problems that CD-PPO work well for them? 


Minor comments:
Some of the references do not have all the required information to find the paper, like the journal, doi, etc, like:

Yaodong Yang, Rui Luo, Minne Li, Ming Zhou, Weinan Zhang, and Jun Wang. Mean field multiagent reinforcement learning, 2020. A.1

Georgios Papoudakis, Filippos Christianos, Lukas Sch¨afer, and Stefano V. Albrecht. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks, 2021. 4.1, E.1

Afshin Oroojlooyjadid, MohammadReza Nazari, Lawrence Snyder, and Martin Tak´aˇc. A deep qnetwork for the beer game: A deep reinforcement learning algorithm to solve inventory optimization problems, 2020a. 5.1

Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?, 2020. 4.1, 5.2

Please take check all the references and update them with the latest version. 


[1] Khouja, M., & Goyal, S. (2008). A review of the joint replenishment problem literature: 1989–2005. European Journal of Operational Research, 186(1), 1-16.
[2] Otero-Palencia, C., Amaya–Mier, R., & Yie-Pinedo, R. (2019). A stochastic joint replenishment problem considering transportation and warehouse constraints with gainsharing by Shapley Value allocation. International Journal of Production Research, 57(10), 3036-3059.
[3] Ongkunaruk, P., Wahab, M. I. M., & Chen, Y. (2016). A genetic algorithm for a joint replenishment problem with resource and shipment constraints and defective items. International Journal of Production Economics, 175, 142-152.
[4] Nguyen, T. T., Nguyen, N. D., & Nahavandi, S. (2020). Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE transactions on cybernetics, 50(9), 3826-3839.
[5] OroojlooyJadid, A., & Hajinezhad, D. (2019). A review of cooperative multi-agent deep reinforcement learning. arXiv preprint arXiv:1908.03963.
[6] Zhang, K., Yang, Z., & Başar, T. (2021). Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, 321-384.
[7] Minner, S., & Silver, E. A. (2005). Multi-product batch replenishment strategies under stochastic demand and a joint capacity constraint. IIE Transactions, 37(5), 469-479.
[8] Hoque, M. A. (2006). An optimal solution technique for the joint replenishment problem with storage and transport capacities and budget constraints. European journal of operational research, 175(2), 1033-1042.
summary_of_the_review: Same as above 
I'll update my recommendation based on the answers of the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper considers the multi-period inventory replenishment problem for a single store with multiple SKUs. The problem is formulated as a resource constrained stochastic game, where the store capacity is the resource constraint shared by all SKUs. The paper takes a multi-agent RL approach to solve this problem, with each SKU being modeled as an agent and having its own replenishment policy. To tackle the challenge presented by the potentially large number of SKUs (agents), a context-aware decentralized PPO method is proposed, where each agent is trained independently in its local simulator given a sample path of the shared resource state (context). Benchmarking were performed on two domains with 5 SKUs and 50 SKUs each against several model-free MARL baselines. Results show that the proposed method can achieve comparable results but with much higher sample efficiency.
main_review: Strengths
- The paper studies a very interesting and promising industrial application of RL. Improvement in inventory replenishment has profound real-world impact.
- A model-based MARL algorithm is proposed to tackle the challenge of large number of agents by exploiting the structure of the multi-SKU replenishment problem, that is, the replenishment of the SKUs are associated only through the global capacity constraint. 
- Empirical results appear to support that the proposed method achieves greater learning efficiency than the model-free baselines.

Weaknesses
- It's not clear how exploration is done in the proposed method. In order for the individual SKUs to train and update their policies independently, the global simulator has to run to obtain a sample path of the shared resource state (context) in each epoch. This basically allocates the capacity for each SKU using the previous policy, and each agent only learns within its allocated capacity trajectory. The inner loop of training is very restrictive in this way given that capacity constraint is a core element in inventory replenishment. How does the joint policy explore different capacity allocations?
- The objective function does not incorporate any penalty for inventory overflow, which is a serious issue in practice.
- The surrogate loss function is based on the assumption that each SKU has limited impact to its context (shared resource). When the number of SKUs is small, this would not hold. In the experiments, did you observe any difference in performance over the 5-SKU and 50-SKU cases? 
- The paper should add at least one classical non-RL replenishment algorithm as baseline. Otherwise, it is hard to judge the significance of the contribution.
Moreover, mean-field MARL [1] is designed for large number of agents. The idea of the context in this method is similar in principle to MF-MARL. 
[1] Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W. and Wang, J., 2018, July. Mean field multi-agent reinforcement learning. In International Conference on Machine Learning (pp. 5571-5580). PMLR.
- The results compare the number of environment interactions. I assume this is the interaction with the global environment. In the proposed method, each agent further interacts with its local environment for a number of steps, while the model-free methods do not. Hence, the comparison may not be entirely fair.
summary_of_the_review: This paper presents some novelty in addressing the issue of large number of SKUs (agents) in applying MARL to solve the multi-SKU inventory replenishment problem, by exploiting the structure of the problem and devising a model-based approach. The empirical results show that the proposed method achieves better sample efficiency in the global environment than the model-free baselines. However, there are several important algorithmic issues not investigated in this paper, and the important related method in the literature is not compared with.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors formulate an inventory optimisation problem as a stochastic game, where each SKU is a player. In order to reduce computational complexity, the game is formulated such that the only interactions among players are through the available storage space they have to share. The paper also proposes an algorithm that relies on local simulators (as opposed to joint simulation) and surrogate losses to improve efficiency. Simulations show that the proposed algorithm is able to outperform other MARL baselines for up to 50 SKUs.
main_review: Strengths:
- The paper tackles an important problem with impact in the real world.
- The formulation as a stochastic game where the SKUs are the players is novel and potentially useful.
- The approximations to make simulations more efficient are sensible. 
- Simulations show that the proposed approach is more applicable than the considered baselines.
- The paper is well written and easy to follow.

Weaknesses:
- Although the paper tackles a real world problem, the scenarios used for the experiments are too simple; and although the authors claim that the approach could be used for more complex settings, the fact is that the authors acknowledge scalability issues. Hence, it is difficult to know if this approach will actually work in the real world. I would have expected to see the algorithm working with more realistic settings, including at least hundreds of SKUs, a multi-echelon network and ideally stochastic lead times. The study of the response under disruptions would have been especially appreciated. 
- Moreover, the baselines chosen for the experiments aren't very relevant. First, they are not well suited for the problem formulation where there are many SKUs, but they could still work for other formulations of the same inventory management problem. Second, I would have expected comparison with well known operations research baselines.
- The authors also claim their approach could work for non-homogeneous SKUs, but it is not clear how. Would they consider the agent's ids (or features) as inputs to the neural networks?
- Another point for improvement is the comparison with related work. For example, I think the current approach is very related to potential games and to mean-field games. Comparison with the former could bring better theoretical understanding and if the current game is potential (as I suspect), the authors could present their approach as a distributed method and compare with the potential-based centralised approach. Comparison with the latter would be very interesting as they potentially could scale the current formulation to arbitrarily large number of SKUs by relying on a mean-field formulation. 

Minor comments
- Some simulation results are not clearly displayed. For example, what is the rationale for the bold font in Table 1? IPPO-IR (w/o context) performs almost as well as CD-PPO and with less variance in the N50-C2000 scenario. Also, why comparing with the worst baseline and not the median or the second best profit in Figure 7?
- Ref to A2C is not Tsitsiklis & Konda, 2000, but an post in the OpenAI blog that suggest a synchronous version of A3C.
summary_of_the_review: The introduces a potentially useful approach for an important real-world problem. But the benefit of tackling a real-world problem vanishes because the proposed approach has been evaluated in very simple environments and with baselines that are not very relevant for this problem. The paper would have higher impact with some of these improvements:
- The algorithm is evaluated in a realistic environment.
- The algorithm is compared with relevant IM baselines (operation research approaches actually used in the real life) even in this toy scenarios.
Also, the paper misses two important lines of related work, namely potential and mean-field games, which could offer new insights to the  proposed approach.


Meta-review
=========
I think the paper tackles an important real world problem with an interesting approach and, although the experiments were a bit lacking, the authors have included a sensible baseline and a more realistic scenario.

I still see some scalability issues and I would appreciate a deeper and insightful discussion with related approaches, like potential and mean-field games. But I think the paper is in better shape than before and it might be useful for other researchers in the field.

So I am happy to increase my score to 6 if the other reviews don't see major flaws.
review_title: null

Meta-review: This is a pretty nice paper, but it suffers a bit from being in an 'uncanny valley' between application and research.  The approach clearly has been made, and derives from, the application under consideration.  However, the application is not a real application, and rather is a simplified simulation. That's okay, but it means that the application here is not the real goal.  So, our attention should go to the solution technique.  Unfortunately, this seems rather specific, exploiting known structure for the specific problem at hand, and lacking other reasonable baselines one could imagine.

So, this is not really an application paper, as the application in the paper is a proxy.  But this is also not really an algorithm paper, as the algorithm is not clearly shown to be generalisable to other settings.  And this is also not a theory paper that tells us something general and meaningful.

These are just observations - this is not criticism per se.  But it means I struggle a little to find something meaningful to learn from this paper, that could be applied elsewhere.

This, in addition to the overall recommendations by the reviewers, unfortunately lead me to reject the paper in its current form.  I want to thank the authors for engaging with the discussion, and hope they have found it interesting and rewarding, despite the outcome this time around.
Decision: Reject