A Simulation Environment and Reinforcement Learning Method for Waste Reduction
Authors: Sami Jullien
Mozhdeh Ariannezhad
Paul Groth
Maarten de Rijke
Venue: Accepted by TMLR ()

Abstract: In retail (e.g., grocery stores, apparel shops, online retailers), inventory managers have to balance short-term risk (no items to sell) with long-term-risk (over ordering leading to product waste). This balancing task is made especially hard due to the lack of information about future customer purchases. In this paper, we study the problem of restocking a grocery store’s inventory with perishable items over time, from a distributional point of view. The objective is to maximize sales while minimizing waste, with uncertainty about the actual consumption by costumers. This problem is of a high relevance today, given the growing demand for food and the impact of food waste on the environment, the economy, and purchasing power. We frame inventory restocking as a new reinforcement learning task that exhibits stochastic behavior conditioned on the agent’s actions, making the environment partially observable. We make two main contributions. First, we introduce a new reinforcement learning environment, RetaiL, based on real grocery store data and expert knowledge. This environment is highly stochastic, and presents a unique challenge for reinforcement learning practitioners. We show that uncertainty about the future behavior of the environment is not handled well by classical supply chain algorithms, and that distributional approaches are a good way to account for the uncertainty. Second, we introduce GTDQN, a distributional reinforcement learning algorithm that learns a generalized lambda distribution over the reward space. GTDQN provides a strong baseline for our environment. It outperforms other distributional reinforcement learning approaches in this partially observable setting, in both overall reward and generated waste.

Published in Transactions on Machine Learning Research (04/2023)
A Simulation Environment and Reinforcement Learning Method for Waste Reduction
Sami Jullien
s.jullien@wvo.rl
ATRLab
University of Amsterdam
Amsterdam, The Netherlands
Mozhdch Ariannozhad
m.ariannechhaa@@wv...
ATRLab
University of Amstendam Amsterdam, The Netherlands
Paul Groth
p-groth&huva.rl
University of Amsteriam Amsterdam, The Netherlands
Maarten de Rijke
m. m.derijke@wvo.nl
University of Amsteriam Amsterdam, The Netherlands
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- KSvr8A62MD
Abstract
In retail (e.g., grocery stores, apparel shops, online retailers), inventory managers have to balance short-term risk (no items to sell) with long-term-risk (over ordering leading to product waste). This balancing task is made especially hard due to the lack of informa- tion about future customer purchases. In this paper, we study the problem of restocking a grocery store's inventory with perishable items over time, from a distributional point of view. The objective to maximize sales while minimizing waste, with uncertainty about the actual consumption by costumers. This problem is of a high relevance today, given the growing demand for food and the impact of food waste on the enviromment, the economy, and purchasing power. We frame inventory restocking that exhibits stochastic bchavior conditioned on the agent's 's actions, making the environ- ment partially observable. We make two main contributions. First, we introduce a new reinforcement. learning environment, RetaiL, based on real grocery store data and expert knowledge. This environment is highly stochastic, and presents a unique challenge for re- inforcement learning practitioners. We show that uncertainty about the future behavior of the environment. is not handled well by classical supply chain algorithms, and that distri- good way to account for the uncertainty. Sccond, we introduce GTDQN, distributional reinforcement learning algorithm that learns generalized Tukey Lambda distribution over the reward space. GTDQN provides strong baseline for our environment. outperforms other distributional reinforcement learning approaches in this partially observable setting, in both overall reward and reduction of generated waste.
reinforcement learning task
butional approaches

---

Published in Transactions on Machine Learning Research (04/2023)
1 Introduction
Retail is an industry that people deal with almost every day. Whether it is to sell clothes, groceries, or shop the internet, all retailers require optimized inventory management. Inventory management considers multitude of factors. One of growing concern is waste. For example, food waste costs the worldwide economy around $1 trillion per year On top of this cost, food waste is responsible for around 10% of worldwide carbon emissions [ This is one order of magnitude higher than civil aviation 3 This means that both businesses and non-profits have an interest in working together to reduce waste, given its economic and ecological impact.
While some waste is produced during production, retailers and consumers also play significant role in the generation of food waste. In this paper, we focus on the retailer-side of waste. We use grocery stores as a canonical example of retailer. Grocery stores need to manage their inventory in order to meet customer demand. To do 80, they pass orders to warehouses. When restocking an inventory, an order is made lo receive n units of a product at a later time. Often, stocks are provisioned in order to ensure customers always have access to an item (Horoš & Ruppenthal] 2021). This mcans that, in the case of perishable items, they might waste items that have stayed in stock for too long. On the other hand, if ilems are under-stocked, it might lead to customers not finding the products they want. This results in balancing problem where orders have to account for uncertainty in demand, both to minimize waste and meet customer demand. This process is of course repeated over several periods- a grocery store is usually open 6 to 7 days a week. This makes inventory replenishment a sequential decision making problem, where actions have potentially delayed outcomes.
Simulation environment. Simulation environments have proven promising for supply chain prob- lems (Cstoro et al.] 2022), as they allow for experimentation from both the concerned community and machine learning experts. Currently, there is no available framework that allows us to properly simulate a grocery store that takes waste into account for different items. Hence, to help evaluate the performance of agents on the inventory restocking (or inventory replenishment) problem, we introduce grocery store environment that takes waste and stochastic customer demand into account.
Learning method. The stochasticity of customer consumption makes the inventory replenishment prob- lem partially observable: the demand being different. from its forecast, two identical situations at first. sight can result in different outcomes. This creates problem: if an action, for a given observation, can result in various rewards, how do we ensure that we properly learn the dynamics of the euvironment? A possibility is to consider non-eeteriiiistic action-value value functions, where we ascribe the randomness in the environment to its reward distribution. Given this, as a strong baseline, we propose to make use of distributional re- inforcement learning (DRL). In DRL, the agent aims to estimate the distribution of the state-action value function Q rather than its expectation (Bellemare et al. 2017). In this paper, we adopt a new direction to estimate the distribution. Non-parametric estimations of summary statistics of the probability distribution are preferred for unconventional data distributions, but are often prone to overfitting and require more sam- ples (Pados& & Papantoni-Kazakos, 1994 Sarle 1995). To circunvent this limitation, we istainate parameters of flexible distribution, in order to facilitate learning. Actual reinforcement Jetooreemet---erning based approaches to waste reduction in the inventory problem do not look at the item-level (Kara & Dogan 2018). We aim to fill this gap in perishable item replenishment by making use of distributional reinforcement learning.
We introduce GTDQN, generalized Tukey deep Q-network; a reinforcement learning algorithm that estimates parameters of well-defined parametric distribution. Currently, distributional approaches rely mostly on non- paramerett estimation of quantiles. We find that distributional algorithms with a reliable mean estimate outperform non-distributional approaches, with GTDQN outperforming expectile-based approaches. While we focus on the task of inventory replenishment, GTDQN does not make any assumption on the task we present here.
World Food 2WWF: Driven to http:///ww..aaaaaaoooo/oo/ooooor wawaadddaaeeedeaaeeeeeeeeeeeeeee centte/drivee_too
waste_ vaste_globsl fooddooodoooooeoo loss, on farms
https://www.ie. lea http:s//www..tt.oorr/eettttaaaa

---

Published in Transactions on Machine Learning Research (04/2023)
Research questions. Overall, we aim answer the following research questions:
1. Given forecast for the consumption of perishable item, can we find an optimal strategy to restock it while maximizing overall profits?
2. Can we ensure that such a policy does not lead to increased waste?
3. Which distributional method the most efficient to solve the problem?
To answer those questions, we compare various discrete-action based DRL methods, including our newly proposed GTDQN, as well as classic inventory replenishment houristics. Previous work has tried to answer those questions partially. Meisheri et al. (2022) do not look at waste through a cost-based approach. And Moor et al. [2022] Ahmadi et al. 2002 solely look at single item, with unchanging demand distribution. Likewise, Selukar et al. (2022) look only at very limited number of items and only consider the problem in a LIFO manner. Overall, the previous work does not provide a common solution for practitionners to try new ordering policies, nor do they provide new ordering algorithms.
Contributions. In sunmary, our contributions are as follows:
We provide RetaiL, a new, complete simulation environment for reinforcement learning and other replenishment policies based on realistic data;
We showcase the performance of classic reinforcement learning algorithms on Retail;
Additionally, propose GTDQN, new distributional reinforcement learning algorithm for the evaluation of state-action values; and
We show that GTDQN outperforms the current statc-of-the-art in stochastic environments, while still reducing wastage of products, making it strong baseline for RetaiL.
Below, survey related work, introduce simulation environment, discuss baselines for sales improvement and waste reduction in this environment, including our newly proposed distributional reinforcement learning method, report on the experimental results, and conclude.
2 Related Work
The inventory restocking problem The literature on ordering policies is extensive. Most work is based on the classic S) policy introduced by Arrow et al. (1951). Yet, their inventory model does not factor in waste. Inventory policies for fresh products as field was kick-started to optimize blood bag 1975). Since then, there is increased attention in the classic van Donsclaar al. 2006 Brockmeulen & Donsclaar sarming--based policies (e.g., Kim et al. 2005 Valluri et al. 2009: Sui et al. 2010:
management (Jennings 1968 Brodheim et al.
supply chain litcrature models to limit waste
2009 Minner & Transchel 2010 Chen et al.
2014 Recently, various reinforcement
have been developed for supply chains; see
Gijsbrechts et al.) 2019:; Sun et al.) (201)). More specifically, Kara & Dogan (2018) pioneered the use of
reinforcement learning for waste reduction in the inventory restocking problem by using a DQN to solve the problem at hand. Their approach can be improved upon, as they aggregate the total shelf lives of the items at hand thus, their agents only have access to the average sholf life of the inventory. Morcover, this makes it impossible to account. for all items independently, to remove expired items from the stock, and to penalize the agent for the generated waste. Indeed, waste be considered tail event it happens suddenly once item has reached its maximum consumption date. Ttem-level waste is currently not considered in the literature. This is why we advocate for simulations where the agent considers all of the inventory.
We think it is not enough to limit the agent's 's knowledge by only looking at the mean. Indeed, a distribution has more summary statistics than its first moment, especially to characterize its tail. We should make use of those, and we believe that this is required for proper evaluation of waste. With distributional reinforcement learning, the agent can learn its summary characteristics, which will be more suited to the task at hand.
3

---

Published in Transactions on Machine Learning Research (04/2023)
Partially observable Markov decision processes. Randomness in environments is common in rein- forcement. learning [Monahan 1982: Ragi & Chong, 2013 Goindani & Neville, 2020). We can distinguish two approaches to this stochasticity, that are not necessarily disjoint. The first is to consider robust Markov decision processes. They make the assumption that a policy should be robust to changes in the data gen- erating process over time, order to have a better estimation of the transition matrix (Xu et al. 2021: Derman et al., 2020). The other approach is to consider the reward as a non-deterministic random variable whose distribution is conditioned on the environment's observation and on the agent's 's action. This usually means that the agent acts under partial information about the environment's state. While one can make the argument that this is only due to the lack of information about the environment (Doshi- Velez, 2009 this is not a setting that generalizes well to unseen situations.
In this paper, we consider that our agent can see the current state of the stock for a given item and its characteristics, but lacks the information over the past realizations of the temporally joint demand distribution, making the environment partially observable.
Distributional reinforcement learning. Learning the Q-value is the most straightforward way to de- velop a Q-learning algorithm, but is most likely to be inefficient, as noted by Bellemare et al. (017). Belle- mare et al.) introduce the C51 algorithm, where they divide the possible Q-value interval in 51 sub-intervals, and perform classification on those. The goal is learn the distribution of future returns instead of their expectation Q. This allows one to achieve gain in performance, compared to using only the expectation; this paper launched the idea of distributional deep reinforcement learning. Later, the authors introduced a more generalizable version of their algorithm, the quantile-rogression DQN (Dabney et al., 2018). Instcad of performing classification on sub-intervals, Dabney el al.] directly learn the quantiles of the Q-value dis- tribution through the use of a pinball loss. While this method proved efficient, its main drawback is that it does not prevent crossing quantiles mcaning that it is possible in theory to obtain Q1 > qo (where Fi the first decile and 99 is the ninth decile). To fix this, different approaches have been tried to approximate the quantiles of the distribution (Yang et al. 2019; Zhou et al. 2020), through the use of distribution distances rather than quantile loss. The work listed above takes a non con-paraettric approach, from classic statistical viewpoint, as they do not assume any particular shape for the distribution. While panaaaaettric methods are known for their flexibility, they sometimes exhibit a high variance, depending on their smoothing pa- rameters. Moreover, the use of non-parametric estimations of quantiles prevents aggregation of agents and their results, as one cannot simply sum quantiles. More recently, research has been conducted on robust Bayesian reinforcement learning (Derman et al.] 2020) to adapt to environment changes. In this paper, the authors develop a model geared towards handling distributional shifts, but not towards handling the overall distributional outcomes of the Q-value.
In our paper, we consider a very flexible distribution that is parameterized by its quantiles, and from which we can both sample and extract summary characteristios (Chalabi et al. 2012).
3
RetaiL, An Inventory Replenishment Simulator
In this section, we detail the inner workings of the simulation environment we introduce
3.1 Inventory replenishment
We can frame part of the process of inventory replenishment as a manager passing item orders to a warehouse to restock a store. At every step, items in the store are consumed by customers. Let us consider a single item and its observation o(i) with a shelf life 8;. We study restocking and consumption of this item over total of T' time periods, each composed of T E N sub-periods that we call time steps. During each time period t e (1,...,,y) T), the manager can perform T orders of up to n instances of the item i. Each of those orders is then added I. time steps later to the inventory termed the lead-time. In the meantime, 7 consumptions of up to n items are realized by customers. Each of those purchases then results in a profit. Assuming that
AA preliminary version of the Retail. environment was presented in Jullien al. 2020 ; the preliminary version lacked the time component in the forecast, dependency management, and detailed examples.
4

---

Published in Transactions on Machine Learning Research (04/2023)
not enough instances of i are present in the stock meet customer demand, it then results in missed opportunity for the manager, resulting in loss. At the end of the period 1, all instances of i currently present in the store have their shelf life decreased by one, down to a minimum of zero. Once an instance of reaches a shelf life of zero, it is then discarded from the inventory, and creates a loss of i's costs for the manager. Furthermore, the restocking and consumpption of i are made in a LIFO way; as customers tend to prefer items that expire furthest from their purchase date (Li al., 2017: Cohen & Pekelman 1978 F
To fulfill its task, the manager has access to forecast of the customer demand for i in the next w time steps, contained in o(i). While we could argue that the agent should be able to act without forecast, this does not hold in real-world applications. In most retail organizations, forecasts are owned by a team and used downstream by multiple teams, including the planning ones that take decisions from it. This means that the forecast "free-to-use" information for our agent. Moreover, this means that adapting to the forecast will prove more reliable in the case of macrocconomic tail-events (lockdowns, pandemics, canal blockades, etc.) as those can be taken into account by the forecast. Obviously, this forecast is only an estimation of the actual realization of i's consumption, and is less accurate the further it is from the current. time step I.
Items are considered independent, meaning that we do not take exchangeability into account. Using this information about all individual items in the store, our goal is to learn an ordering policy to the warchouse that generalizes to all items. An ordering policy simply refers to how many units we need to order at every time step, given the context information we have about the state. The goal of our policy is to maximize overall profit, instead of simply sales. This means that waste, and missed sales are also taken into account. Moreover, while our policies have access to information about the consumption forecast of the items, this forecast is not deterministic. Indeed, some of the mechanics of the environment the number of customers per time-step, despite being correlated to the previous time-step, is hidden, the agent is delayed in its observation. This means that reinforcement learning agents evolve in a partially observable Markov decision process (POMDP), where an observation and an action correspond to reward and state distribution, and not a scalar.
hidden to the agent:
Formally, we can write the problem as finding a policy x :0 - N such that:
= aig Max NET TET
ΣΣΣ (o)),.)))
(1)
where or (i) is the observation of item i at the time-step 7 and Rg the reward function parameterized by the policy æ. the following sections, we detail how we model the items, the consumption process as well as the Markov decision process we study.
3.1.1 Item representation
Using weal-world data of items being currently sold is impossible, as il would contain confidential information (e.g., the cost obtained from the supplier). This is why we fit copula on the data we sourced from the retailer to be able to generate what we call pscudo-items: tuples that follow the same distribution actual item set. Having pseudo-items also allows to generate new, unseen item sets for any experiment. This proves useful for many reinforcement learning endeavors (Tobin et al. 2017).
When an instance of our experimental environment is created, it generates an associated set of pseudo-items with their characteristics: cost, price, popularity and shelf life. These characteristics are enough to describe an item in our setting: we do not recommend products, we want to compute waste and profit. We provide the item generation model and its parameters along with our experiments.
3.1.2 Consumption modelling
We model the consumption as the realization of so-called n. P process, as this way of soparating the number of customers and purchasing probability is common in retail forecasting (Juster 1966). We consider that a
We make the assumption that the price does not depend on the remaining shelf life of the item.
5

---

Published in Transactions on Machine Learning Research (04/2023)
day is composed of several time steps, each representing the arrival of given number of customers in the store.
3.1.3 POMDP formalization
Customer consumption depends on alcatoric uncertainty, and forecast inaccuracy derives mostly from epis- temic uncertainly: forecast. capable of knowing customer intent would leave little room for aleatoric un- certainty. Yet, there is difference for our agent, as both of those uncertainties affect the reward and transitions in the environment. This means that the euvironment is partially observable to our agent, the agent is unable to see past realizations of demand over linked time periods: a customer that comes in the morning will not come in the afternoon in most cases. A fully observable state would contain past real- izations of the demand over the provious time steps. Formally, we can write a partially obscrvable Markov decision process as tuple (O, A, R P); where O the observation we have of our environment, A the action space, R the reward we receive for taking that action, and P the transition probability matrix. Here, they correspond to:
O The full inventory position of the given item (all its instances and their remaining shelf lives), its shelf life at order, its consumption forecast, its cost and its price;
A How many instances of the item we need to order; and
R The profit, to which we subtract profit of missed sales and cost of waste.
3.2 Environment modeling
We aim to model realistic grocery store that evolves daily basis through customer purchases and inventory replenishment. To do so, we rely on expert knowledge from major grocery retailer in Europe. Our environment relies on four core components: item generation, demand generation, forecast generation, and stock update for reward computation.
Item gencration. We define an item as a tuple containing characteristics common to all items in an item set: shelf life, popularity, retail price, and cost: i= b,,b, v,,,, e) [ As data sourced from the retailer contains sensitive information, we want be able to generate items on-the-fly. As purchases in retail are highly repetitive, we will base ourselves on the popularity of the items generate the demand forecast in Section 3.2 On top of helping with anonymity, being able to learn in different but similar environment has proved to help with the generalization of policies (Tobin et al.) 2017). To do so, we fit Clayton copula. (Yan, 2007) on the marginal laws (gamma and log-normal) of our tuple. The parameterized model is available with the code. Given the parameterized copula, we can generate an unlimited number of tuples that follow the same multivariate distribution as the items available in the data sourced from the retailer.
Demand generation. To represent variely of demand scenarios, we based the demand on the popularity of the items given by the past purchases in the real data. We then modeled a double seasonality for items: weekly and yearly[] Overall, given a customer visiting the store, we write the purchase probability at time period t, pt(t) for pseudo-item i as:
(t) = b; cos(wwtt + @1.1) cos(wyt + d2.1),
(2)
where b, is the popularity (or base demand) for item i, Φ. its phases, and Ww, wy are the woekly and yearly pulsations of the demand signal, respectively. The base demand by comes from the fitted copula, and the phases Φ,, are randomly sampled to represent a variety of items. Together with the purchase probability, we also determine the number of customers who will visit the store on a given time-step. To do so, we model multivariate Gaussian over the day sub-periods, with negatively correlated marginal laws (if customer
6uur repository also includes dimensions, allow for transportation cost computation. For example, beers are often sold the end of the week, and ice cream the summer.

---

Stock update. To step in the environment, the agent needs to make order of units u, of the item i. We consider that a time period t is a succession of several time-steps At the beginming of a time period, items that were ordered L time-steps before are added to the stock, where L is the lead time. If the total numbers of items would exceed a maximum stock size M, the order is capped at M - n. The generated demand is then matched to the stock. Items are removed from the stock a LIFO manner, as the case in most of the literature (Li et al. 2017 Cohen & Pekelman 1978). Items that are removed see their profit their profits Published in Transactions on Machine Learning Research (04/2023)
comes in the morning, they will not come in the evening). Having the purchase probability and the number of customers n(!), we can then simply sample from binomial law B(n(U),Pi()) P:()) to obtain the number of units u, of item sold at the time-step t.
Forecast generation. The parameters (n(t), (t)) of the aforementioned binomial law (Section 3.2 are not known to the manager that orders items. Instead, the manager has access to forecast an estimator of the parameters. We simply use a mean estimator for nt),, as seasonality is mostly taken into account via our construction of (t))
As for the purchase probability estimator, we assume that the manager has access to a week-ahead forecast. We write the estimator as such:
At + 6e) = pp + 5t) + 8.63.
(3)
where 6, E {L....,7} 7 and 6, ~ o0) The noise e, represents the forecast inaceuracy for the item i, and the uncortainty about the customer bchavior the manager and the store will face in the future. We assume a single a for all items and a mean of 0, as most single point forecasts are trained to have a symmetric, gqually-weightee error. ó, is used to show the growing uncertainty we have the further we look in the future.
added to the reward. If the demand is higher than the current stock, the lacking items removed from the reward (missed sales). Finally, if the step is at the end of the day, all items in store receive penalty of one day on their remaining shelf lives. Items that reach a shelf life of 0 are then removed from the inventory, and their cost is then removed from the reward: these items are the waste.
4 Inventory Replenishment Methods for Perishable Items
In the previous section, we introduced the environment we built, along with its dynamics. In this section, we introduce the baselines we consider, together with our own algorithm, GTDQN.
4.1 Baselines
In this subsection, we introduce the various algorithms that serve as baselines. We draw one example from the supply chain literature, well as several from the field of Reinforeement Learning. We focus DQN (Mnih et al. 2015) and its derivatives, as they are simple to apprehend.
(s, Q) Ordering policy. The (s, Q) ordering policy (Nhhmias & Demmy units of stock when the inventory position goes below certain threshold in use (along with some of its derived cousins (Kelle & Milne 1999 Cachon 1999) for decades in supply
1981 consists of ordering Q While very simple, it has been
chain settings.
Bhep---nnetorrss (DQN). The first. reinforcement learning baseline we use is Deep-Q-Nee Networss (Mnih) et al. 2015). While this model is not SOTA anymore, it is often reliable approach to sequential decision- making problem, mainly in games like Atari, for instance. The idca behind DQN is to predict the Q-value of all possible actions that can be taken by the agent for specific input. By using those values, we are able to use the corresponding policy to evolve in the environment.
UUsually, time period would he day, meaning that store can be replenished several times during the course of day.
7

---

Published in Transactions on Machine Learning Research (04/2023)
C51. Categorical DQN [Bellemare et al. 2017 can be seen multinomial DQN with 51 categories and is a distributional version of DQN. Instead of predicting the Q value, the model divides the possible sum of future rewards interval in 51 (can be more or less) intervals. Then, the network assigns a probability to each interval, and is trained like a multinomial classifier.
Quantile regression DQN (QR-DQN). DQN using quantile regression (Danney et al. 2018 is not necessarily more performant than C51. Instead multinomnil classifier, this algorithun performs a regres- sion on the quantiles of the distribution function. This approach has the benefit of being non non-paametrri, but does not guarantee that the quantiles will not cross each other: we obtain Q10 < Q90, which would be impossible in theory. While some authors sort the obtained quantiles to remove the contradiction, we think this results in bias in the statistics that are learnt that way.
Expectile regression DQN. Expectile regression DQN (ER-DQN) (Rowland et al. 2019) takes the idea behind QR-DQN and replaces quantiles with expectiles. It is possible to interpret an expectile as the "value that would be the mean if values above were more likely to occur than they actually are" (Philipps 2022
4.1.1 Underlying neural architecture
All the considered DQN-based algorithms, including the following GTDQN, are based on the same feed- forward architecture. The individual shelf lives of the already stocked items are first processed together in a convolution layer. They are then concatenated with the item characteristies and processed throngh a simple Feedl-Forward Deep Neural Network with Layer Norm and SELU activation (Klambauer et al. 2017
4.2 Generalized Tukey deep Q-network
In this section, introduce new baseline, generalized Tukey deep Q-network (GTDQN), for decision- making in stochastic environments. As the problem we study requires planning under uncertainty, we need a baseline that can consider randomness in the signals it receives from the environment. While we can use classic off-policy architectures like Deep Q Networks, the partial observability of our environment is more likely to be encompassed by an algorithm that assumes value distributions over actions rather than simple scalar values.
Thus, we assume that the Q-value follows a generalized lambda distribution, also known as generalized Tukey distribution. This is weak assumption that does not constrain the shape too much. Indeed, the use of four parameters allows for a very high degree of flexibility of shapes for this distribution family (Chalabi et al., 2012): unimodal, s-shaped, monotone, and even u-shaped. The generalized lambda distribution can be expressed with its quantile function a as follows:
au(u)= A1 + 1 jurv - (1 (1-u)^4),
(4)
A2
where A (A1, A2; As, dg) is the tuple four parameters that define our distribution. Those parameters can then be used to compute the distribution's four first moments (mean, variance, kurtosis and skewness). if they are defined.
Thus, we build our Q-network not to predict the expected Q-value nor its quantiles, but to predict the parameters Å1, d2, As, A of a generalized lambda distribution. This allows us to obtain both guarantees on the behavior of the distribution's tail, and non- con-crossing quantiles. Still, we perform our Bellman updates by estimating the quantiles derived from the valuos of the distribution's parameters. To obtain a quantile's value, we simply need to query by using Equation f
Working with quantiles guarantees that the Bellman operator we use is contraction, when using a smoothed pinball loss (Yang et al. 2019). While written differently in most of the literature, the classic pinball loss can be written as follows:
PLuly, ((1)) (u- (n)) max(0,,(w) v),
(5)
8

---

Published in Transactions on Machine Learning Research (04/2023)
Algorithm 1: Generalized Lambda Distribution Q-Learning
Require: quantiles (9......9N}, parameter
Input 0,a, E
[0,1]
1 A(d', Va' E
# Compute distribution parameters
A' argmax, j(A((d, d'))
# Compute optimal action (Equation
Tq + + (4:), Vi
# Update projection via Equation
4 Optimize via loss function (Equation 6
Output: Mny [, (Tq, (A(o,))
where u is quantile, y the realized value, and (u) the predicted value of quantile Its 6-smoothed version is obtained by plugging this loss estimator instead of the square error in Huber loss (Huber 1992 This gives us the following loss function:
{ {
2 ly DA (u) A), 8 (lv (1))) 6/2) A),
for y - (x S otherwise,
C (v.
(6)
with A PLu DA (u)), where is a smoothing parameter and the considered quantile for the loss. Algorithm shows the way update the parameters of our network through temporal difference learning adapted to quantile setting.
Unlike C51 [Bellemare et al. 2017) and QR-DQN (Dbnney et al., 2018). we do not select the optimal action (line 2 of Algorithm 1 via an average of the quantile statistics, but via a mean estimator obtained via our GLD distribution's parameters FFurnier et al. 2007
(AA) \ + 1+ 1+
(7)
A2
This approach closer the implementation of ER-DQN (Rowland al. 2019), where only the expectile 0.5 is used, rather than QR-DQN, where the quantiles averaged to obtain an estimation of the mean (Dabney
et al. 2018
5 Experiments
In this section, we compare the performance in inventory replenishment simulation of our new baseline against a number of basclines (RQ3), for a varicty of scenarios. We want to see whether we can improve overall profit (RQ1), and, if so, if it comes at the cost of generating more waste (RQ2).
5.1 Experimental setup
We train our DQN-family policies (baselines and GTDQN) on total 6:000 pseudo-items, for transitions of 5 000 steps. We do so in order to expose our agents a variety of possible scenarios and items. Morover, do not train accounting in supply chain planning (Beamon & Fernandes) 2004). We evaluate the performance of our agents on a total of 30 generations of 100 unseen pseudo-items, for 2 000 steps. We repeat this for 3 different scenarios of randomness, indicating how observable the environment is. We name them H = 0, H = = 2:
agents in average reward framework, discounting also presents an interest for
H = 0 In this scenario, the enviromment's mechanics are not random. Here, Equation reduces to (.(1+)))) p:(t + 6.). This means that the agent knows exactly the purchase probability of items. In this sccnario, there little neod for adaptability as the inter-day variations customer behavior are close to non-existent.
= In this scenario the environment's mochanics are slightly random and overall exhibit little variation. In this scenario, the agent needs to learn how to interpret the week-ahead forecast and leverage it to increase profit.
9

---

Published in Transactions on Machine Learning Research (04/2023)
Conv Layer
Stock
Fully Connected Layer
SELU Activation
Input
Layer Norm
Output
Item characteristics
H times
Figure Neural architecture for all considered DQN- models
2 In this scenario, the environment is highly noisy and becomes much harder to predict.
These scenarios allow us to verify whether an agent has learned decent policy and is able to generalize to unseen data. Real grocery stores with a "good" forecast are more likely to be represented by the H = 1 and H 2 scenarios (Ramanathan 2012). The agents are trained on a reward function R defined as R Sales Waste. In Section [52.3] the reward function used is Sales 10 Waste in order assess how well agents reduce waste when given new target.
We perform two experiments, where we look at overall profit performance and waste reduction relative to baseline, respectively.
Experiment 1: Impact of forecast inaccuracy. In this experiment, we measure the overall performance of the various agents, for the different levels environment randomness (RQ1). This experiment allows us mcasure the impact of randomness and unprodictability of consumption behavior on our agents, and see whether they are an improvement over deterministic heuristic.
Experiment 2: Impact of unstable order behavior waste. In this experiment, show how the orders translate into generated waste. This way, see whether the improvement in the previous section comes at the cost of more waste or not (RQ2).
Implementation and computational details. Our code was implemented in PyTorch PPaskee et al. 2019) and available on GitHub @ We ran our experiments on a RTX A6000 GPU, 16 CPU cores and 128GB RAM. All models use the same underlying neural network architecture as shown in Figure [ We notice that GTDQN was approximately times faster than QR-DQN and ER-DQN for more than 4 quantiles, as needs estimating constant number of parameters. We performed grid search on DQN for all hyperparameters, and kept those for all models. However, set the exploration rate at 0.01 for distributional methods, following Dabney et al. 2018). Training curves are available in Appendix A.2
5.2 Results
In this section, detail the performance of the various baselines as well GTDQN, introduced in Section 4.2 for both resistance to uncertainty and waste reduction. We averaged the results of the different algorithms over total of 6,000 pseudo-items.
5.2.1 Overall performance
We report the performance in Table [ as the improvement relative to a simple (s, Q) policy, this kind
of policy is still prominent in supply chain practices (Jalali & Van Nieuwenhuyse 2015 In this table, we
see that all models perform better than the baseline when there is no uncertainty (H = 0). Yet, there is no significant difference between them.
10

---

Published in Transactions on Machine Learning Research (04/2023)
Table 1: Human- mamaan-normalized profit (higher is better). Results on trajectories of length 2 2000, averaged over 000 items, for 3 different consumption volatility scenarios. Bold indicates best.
Quantiles
H =
H =
H 2
DQN
146.1% =0.7
178.8%
+0.5
146.5% +0.3
C51
142.7%
#0.9
173.0%
#0.4
156.1% #0.2
146.7%
=0.7
190.4%
10.9
176.6%
#0.6
147.2% =0.8
204.2% #0.11
189.5% +0.6
QR-DQN
15
146.7%
#0.1
193.3%
#0.8
161.6% #0.4
19
146.1%
+0.1
198.7% +0.8
170.1% +0.5
5
147.4%
+0.9
203.5%
+0.9
172.5% +0.5
9
145.1% +0.7
177.7%
10.6
174.2%
+0.5
ER-DQN
15
148.7% #0.9
202.2%
10.8
168.0%
#0.5
19
146.1%
#0.1
209.3%
10.9
170.3% 10.6
5
147.8%
+0.8
213.3%
+0.9
186.2% +0.5
GTDQN
9
147.9% #1.1
208.3%
#1.0
194.1%+0.77
(ours)
15
143.1%
+0.6
209.5%
+0.9
192.8% +0.6
19
147.3%
#0.9
212.6%
+0.9
191.8% #0.6
In the second scenario with medium volatility (H = 1), the performance improvement of distributional methods over deterministic ones shows clearly, highlighting the performance of QR-DQN, GTDQN and ER-DQN. C51 exhibits performance closer to DQN than to the other distributional approaches. It is additionally much slower to train than all others. C51 being unable to update its bucket values might be a reason why its performance is slightly disappointing - yet, it still is a clear improvement over the (s, Q) baseline.
In the third scenario (H = 2), it is made even more obvious that the non-bounded distributional approaches can capture the uncertainty, they widen the gap with the more simple DQN. Our method, GTDQN, is overall better than ER-DQN, that bases itself expectiles. Our method is relatively more stable with respect to how many quantiles or expectiles it estimates with. Morcover, its computation time is much lower than ER-DQN and QR-DQN for > 4, as it does not estimate new parameters.
Looking closer at the results in Figure 2 we can see that improvements in profit by GTDQN relative the bascline are strictly onc-sided in high-entropy scenarios. Morcover, they show that the resulting distribution of improvements is not gaussian this is why the MAD was preferred as a metric for uncertainty quantification. This mcans that using GTDQN results in consistent improvement in profit performance.
5.2.2 Waste reduction
In Table 2 we visualize the waste generated relative to our simple (s, Q) policy baseline. In all scenarios. we see that all methods reduce waste relative to the baseline. This means that they managed to improve the overall score (Table D without increasing waste: they ordered more than the baseline, and wasted less products. This is not surprising: the baseline only considers the number of items in the stock, not when they expire. Learning this both contributes increased score and reduced waste. In all scenarios, we see that all methods reduce waste relative to the baseline.
In the scenario with full information, C51 performs very well, followed closely by GTDQN and some verisons of ER-DQN. In the the H = secnario, where the environment is partially observable, both GTDQN and QRDQN perform similarly. Surprisingly, C51 does not perform well and is the worst of all models considered here. Given the significant improvement over the baseline in a partially observable environment brought by those methods, we conclude that they were able to adapt to the environment's 's dynamics and its randomness, while still taking the potential waste into account. Finally, in the perform comparably well.
scenario, all models
11

---

Published in Transactions on Machine Learning Research (04/2023)
Table 2: Human-normalized waste (lower is better). Results on trajectories of length 000, averaged over 300 items, for 3 different consumption volatility scenarios. Bold indicates best.
Quantiles
H =
H
H =
DQN
16.8%
23.0%
13.2%
C51
2.6%
66.9%
17.3%
5
32.1%
16.5%
9.6%
9
46.7%
13.9%
12.6%
QR-DQN
15
20.1%
17.7%
16.3%
13.8% 19
19.4%
11.1%
5
46.5%
26.7%
13.3%
9
6.1%
23.4%
12.8%
ER-DQN
15
8.4%
14.5%
11.5%
19
30.1%
13.1%
11.3%
5
15.6%
13.8%
14.6%
GTDQN
9
8.1%
19.3%
16.2%
(ours)
15
4.9%
14.5%
16.3%
19
6.1%
13.9%
15.9%
7
6
Count
4
3
1
0
90
130
Improvement relative to (s,Q) baseline, in %
Figure 2: Improvement of GTDQN over (s, Q)-polcc,, for the H 2 sccnario, for 30 generations of 100 items.
Note that GTDQN is constantly in the same neighborhood as the bost solution, no matter the number of computed quantiles or the randomness of the environment.
5.2.3 Waste reduction for waste-averse agents.
In this section, we look the performance of all evaluated algorithms, trained under more waste-averse reward. We evaluated the algorithms with the same setup, and Reward function R Sales 10 Waste.
12

---

Published in Transactions on Machine Learning Research (04/2023)
Comparison of normalized performance in profit and generated waste, for a higher weight on waste during learning. (bold indicates best). Results on transitions of length 2 000, averaged over 6000 items, for 3 different consumption volatility scenarios.
Table 3:
Profit
Waste
=
==1
=
H
H
H 2
DQN
140.9% 0.8
179.4% + 0.4
158.3% + 0.3
8.6%
28.9%
13.9%
C51
139.3% 0.8
175.1% = 0.7
152.3%
24.5%
89%
27.7%
QR-DQN09
138.6% + 1.0
185.1% 0.7
162.5%
26.1%
14.3%
13.6%
ER-DQN@9
138.5% 1.1
167.3% 0.5
163.9%
23.9%
11.6%
8.9%
GTDQN@9 (ours)
140.3% 1.1
186.3% + 0.7
164.4% + 0.3
4.1%
17.3%
8.8%
Table 4: Comparison of normalized performance in profit and generated waste of GTDQN with and without Equation 7 (bold indicates best). Results on transitions of length 2 2000, averaged over 6 600 items, for 3 different consumption volatility scenarios.
Profit
Waste
H = 0
H =
H =
H = 0
H =
H =
GTDQN without Equation
141.5%
150.5%
132.1%
93.6%
39%
40.1%
GTDQN
147.8%
213.3%
186.2%
15.6%
13.8%
14.6%
Table shows that all agents do reduce their sales, and that it mostly comes lower waste, compared to Table We can see from the table that GTDQN maintains its lead in profit, and also reduces its generated waste on every scenario compared to the previous reward signal.
5.2.4 Impact of the mean estimator
It is of interest to know why GTDQN performs well on the RetaiL environment, despite it not being tuned for it. We thus perform an ablation study, where we estimated our parameter vector A = (A1, A2, A3, Aa). However, instead of using Equation 7 to select the optimal action, we compute 5 quantiles and average them to estimate the mean, as the case in QR-DQN. As shown in Table our mean estimator in Equation has strong impact, both on waste and profit.
In conclusion, we have shown that it is possible to improve the restocking strategy for perishable items by using distributional algorithm (RQI). Moreover, this improvement in overall profit does translate to lower waste (RQ2). Finally, we have shown that the algorithm we propose, GTDQN, does present strong alternative to other distributional algorithms, as it is constant in its good performance (RQ3).
6 Conclusion
In this paper, we have introduced a new reinforcement learning environment, RetaiL, for both supply chain and reinforcement learning practitioners and researchers. This environment is based on expert knowledge and uses real-world data to generate realistic scenarios. By taking waste at the item-level into account, and by being able to lune the forecast accuracy as well the customer's behavior, we can act on the environment's noisiness; this results in partially observable MDP, with tunable stochasticity, which is lacking for most RL tasks. Inventory management in Retail needs the agent to pick up scasonal patterns, unpredictability of customer demand, as well as delayed action effects, and credit assignment as it works in a LIFO manner.
Additionally, we have proposed Generalized Tukey Deep Q Networks (GTDQN), a new algorithm aimed at estimating wide range of distributions, based on DQN. GTDQN offers the consistency of parameterized distributions, but can be trained by quantile loss instead of bisiihood--assde approaches. Moreover, GTDQN represent wide array of distributions, and does not suffer from the quantile crossing phenomenon. We
13

---

Published in Transactions on Machine Learning Research (04/2023)
have found that GTDQN outperforms other methods from the same family in most cases for the task replenishment of perishable items under uncertainly. GTDQN does so by using quantile loss to optimize a well-defined distribution's parameters and selecting optimal actions using a mean estimator. GTDQN does not require any assumptions specific to the simulation environment we provide. We have also found that GTDQN can offer significant and constant improvement over our classic supply chain baseline, as well as over other distributional approaches, outperforming ER-DQN in highly unpredictable environments. Morcover, GTDQN does this without generating more waste through its replenishment policies, hinting that it learnt the environment's dynamics better than the baselines. Our results point towards distributional reinforcement learning a way to solve POMDPs.
As to the broader impact of our work, the simulation environment we provide with the paper rewards weighting the risks of wasting an instance an item and the profit from selling it. This might favor resupply of stores more wealthy geographical areas where the average profit per item is higher. Thus, any deployment of such an automated policy should be evaluated on different sub-clusters of items, ensure it does not discriminate on the purchasing power of customers. This kind of simulation can be replicated for all domains that face uncertainty and inventory that lowers in perceived quality with time- for instance, the fashion industry.
A limitation of our work is that we only considered discrete action spaces, whereas our environment would more be adapted to infinite-countalee ones. Moreover, we consider marginal demand between items to be independent, which is unlikely to be the case in real life. Finally, our environment, Retail, assumnss no cost to restock. This most likely inflates slightly the performance of the algorithms we consider for stores that do not have scheduled restocking as the ones consider. Furthermore, proper implementation in production would require continuous control mechanism (such Model Predictive Control) to match the desired stock level.
For future work, we intend to model price elasticity of customers in order to model item consumption in case of out-of-stock items. We also want to add a restocking cost based on volume and weight of items. We plan to extend GTDQN for multi-agent reinforcement learning, as our estimation of parameters gives us access to cumulants that can be used sum rewards of various agents and policies. This would be especially interesting given the low number of parameters in GTDQN. Furthermore, action spaces, to leverage the structure of the data more elficiently. Finally, we plan to study whether those automated replenishment policies based on balancing profits and waste do not disadvantage some categories of customers more than others.
plan to extend it to continuous
Acknowledgements
We want thank our anonymous reviewers for helping us improve this paper. This research was supported by Ahold Delhaize and the Hybrid Intelligence Center, a 10-year program funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research. https:
//
All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
References
Ehsan Ahmadi, Hadi Mosadegh, Reza. Maihami, Iman Ghalehkhondabi, Minghe Sun, and Gürsel A. Süer. In- telligent inventory management approaches for perishable pharmaceutical products in healthcare supply chain. Computers & Operations Research, 147:105968, 2022.
Kenneth J. Arrow, Theodore Harris, and Jacob Marshak. Optimal inventory policy. Economic Information, Decision, and Prediction, 19(3):5-28, 1951.
Benita M. Beamon and Clara Fernandes. Supply-chain network configuration for product recovery. Produc- lion Planning & Control, 15(3):270-28 281, 2004.
14

---

Published in Transactions on Machine Learning Research (04/2023)
Marc G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. In International Conference on Machine Learning, pp. 449-458, 2017.
Eric Brodheim, Cyrus Derman, and Gregory Prastacos. On the evaluation a class of inventory policies for perishable products such as blood. Management Science, 21(11):330-13325, 1975.
Rob A.C.M.
Brockmeulen and Karel H. van Donselaar. A heuristic to manage perishable inventory with batch ordering, positive lead-times, and time-varying demand. Computers and Operations Research, 36 (11):3013-301)) 300-30888 2009.
Gérard P. Cachon. Managing supply chain demand variability with scheduled ordering policies. Management Science, 45(6))843-855 1999.
Julen Cestero, Marco Quartulli, Alberto Maria Metelli, and Marcello Restelli. Storehouse: reinforcement lcarning environment for optimizing warchouse management. In International Joint Conference on Neural Networks, pp. 1-9. IEEE, 2022.
Yohan Chalabi, David Scout, and Diethelm Wuertz. Flexible distribution modeling with the general- ized lambda distribution. Munich Personal RePEc Archive, 2012. https://mpra. ub. mmi-muenchen.de/ 43333/
Xin Chen, Zhan Pang, and Limeng Pan. Coordinating inventory control and pricing strategics for perishable products. Operalions Research, 62(2):284 300, 2014.
Morris A. Cohen and Dov Pekelman. LIFO inventory systems. Management Science, 24(11):110-11122 1978.
Will Dabney, Mark Rowland, Mare G. Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In AAAI Conference on Artificial Intelligence, 2018.
Bram J. De Moor, Joren Gijsbrechts, and Robert N. Boute. Reward shaping to improve the performance of deep reinforcement learning in perishable inventory management. European Journal of Operational Research, 301 (2):535-55))) 545, sep 2022.
Esther Derman, Daniel Mankowitz, Timothy Mann, and Shie Mannor. A Bayesian approach to robust reinforcement learning. In Uncertainty in Artificial Intelligence, pp. PMI.R, 2020.
Finale Velei-Velez. The infinite partially observable Markov decision process. Advances in Neural Informa- tion Processing Systems, 22, 2009.
Benjamin Fournier, Nicolas Rupin, Maxence Bigerelle, Denis Najjar, Alain Tost, and Rand R. Wilcox. Esti- mating the parameters of a generalized lambda distribution. Computational Statistics & Data Analysis, 51(6)22813-285 20-35,35222 2007.
Joren Gijsbrechts, Robert N. Boute, Jan A. Van Mieghem, and Dennis Zhang. Can deep reinforcement learn- ing improve inventory management? Performance and implementation of dual sourcing-mode problems. SSRN 3302881, 2019.
Mahak Goindani and Jennifer Neville. Social reinforcement learning to combat fake news spread. In Uncer- tainty in Artificial Intelligence, pp. 1016. PMLR, 2020.
Izabela Karolina Horoš and Tonia Ruppenthal. Avoidance of food waste from a grocery retail store owner's perspective. Sustainability, 13(2):550, 2021.
Peter J. Huber. Robust estimation of location parameter. In Breakthroughs in Statistics, pp. 492 518. Springer, 1992.
Hamed Jalali and Inneke Van Nieuwenhuyse. Simulation optimization in inventory replenishment: A classi- fication. ITF Transactions, 11111331 2015.
15

---

Published in Transactions on Machine Learning Research (04/2023)
J. B. Jennings. An analysis of hospital blood bank whole blood inventory control policies. Transfusion, 8 (6):335 (1333-342, 1968.
Sami Jullien, Sebastian Schelter, and Maarten de Rijke. Retail: Open your own grocery store to reduce waste. In NeurIPS, 2020.
F. Thomas Juster. Consumer buying intentions and purchase probability: An experiment in survey design. Journal of the American Statistical Association, 61(315):658 696, 1966.
Ahmet Kara and Tbrahim Dogan. Reinforcement learning approaches for specifying ordering policies of perishable inventory systems. Expert Systems with Applications, 91:150-158, Jamuary 2018.
Peter Kelle and Alistair Milne. The effect of (s, s) ordering policy on the supply chain. International Journal of Production Economics, 59(1-3):113-1 1999.
Chang Ouk Kim, J. Jun, J. K. Back, Robert L. Smith, and Y. D. Kim. Adaptive inventory control models for supply chain ma management. International Journal of Advanced Manufacturing Technology, 26(9-10): 1184 1192, October 2005.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Qing Li, Peiwen Yu, and Xiaoli Wu. Shelf life extending packaging, inventory control and grocery retailing. Production and Operations Management, 26(7):1369-1385 2017.
Hardik Meisheri, Nazneen N. Sultana, Mayank Baranwal, Vinita Baniwal, Somjit Nath, Satyam Verma, Balaraman Ravindran, and Harshad Khadilkar. Scalable multi-producl inventory control with lead time constraints using reinforcement learning. Newral Computing and Applications, 34(3):1735-1755) feb 2022.
Stefan Minner and Sandra Transchel. Periodic review inventory-control for perishable products under service- level constraints. OR Spectrum, 92(4):979-996 2010.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglon, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
George E. Monahan. State of the art-A survey of partially observable Markov decision processes: Theory, models, and algorithms. Management Science, 28(1):1-16, 1982.
Steven Nahmias and W. Steven Demmy. Operating characteristics of inventory system with rationing. Management Science, 27(11):1236 1245, 1981.
Dimitris A. Pados and P. Papantoni-Kazze Kazakos. A note on the estimation of the generalization error and the prevention of overfitting [machine learning]. In IEEE International Conference on Neural Networks, volume 1, pp. 321 326 vol.1, 1994.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Grogory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorcc:: An imperative style, high-performance deep learning library. In Advances in Newral Information Processing Systems, pp. 8024-8035. Curran Associates, Inc., 2019.
Collin Philipps. Interpreting expectiles. SSRN 3881402, 2022.
S. Ragi and E. K. P. Chong. Uav path planning in dynamic environment via. partially observable markov decision process. IEEE Transactions on Aerospace and Electronic Systems, 4944):237722412, 2013.
Usha Ramanathan. Supply chain collaboration for improved forecast accuracy of promotional sales. Inter- national Journal of Operations & Production Management, 32(6):676 06666695555 2012.
16

---

Published in Transactions on Machine Learning Research (04/2023)
Mark Rowland, Robert Dadashi, Saurabh Kumar, Remi Munos, Mare Bellemare, and Will Dabney. Statistics and samples in distributional reinforcement, learning. In International Conference on Machine Learning, pp. 5528 5536. PMLR, 2019.
Warren S. Sarle. Stopped training and other remedies for overfitting. In Symposium on the Interface of Computing Science and Statistics, pp. 352-360, 1995.
Mayur Selukar, Pooja Jain, and Tapan Kumar. Inventory control of multiple perishable goods using deep reinforeement learning for sustainable environment. Sustainable Energy Technologies and Assessments, 52:102038, 2022.
Zheng Sui, Abhijit Gosavi, and Li Lin. A reinforcement learning approach for inventory replenishment in vendor-managed inventory systems with consignment inventory. Engineering Management Journal, 22(4): 44-53, 2010.
Ruoying Sun, Peng Sun, Jiahui Li, and Gang Zhao. Inventory cost control model for fresh product retailers based on DQN. In IEEE International Conference on Big Data, pp. 5321 5325, 2019.
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schncider, Wojeiech Zaremba, and Picter Abbecl. Domain randomization for transferring deep neural networks from simulation to the real world. In IEEB/RSJ International Conference on Intelligent Robots and Systems, pp. 23-30, 2017.
Annapurna Valluri, Michael J. North, and Charles M. Macal. Reinforcement learning in supply chains. International Journal of Neural Systems, 19(05):331-344 2009.
Karel H. van Donselaar, Tom van Woensel, Rob A.C.M. Broekmeulen, and Jan C. Fransoo. Inventory control of perishables in supermarkets. International Journal of Production Economics, 104(2):462 472, 2006.
Lily Xu, Andrew Perrault, Fei Fang, Haipeng Chen, and Milind Tambe. Robust reinforcement learning under minimax regret for green security. In Uncertainty in Artifical Intelligence, pp. 257 267. AUAI Press, 2021.
Jun Yan. Enjoy the joy of copulas: With package copula. Journal of Statistical Software, 21(4):1 2007.
Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, and Tic-Yan Liu. Fully parameterized quantile function for distributional reinforcement learning. In Advances in Neural Information Processing Systems, volume 32, pp.
Curran Associates, Inc., 2019.
Fan Zhou, Jianing Wang, and Xingdong Feng. Non-crossing quantile regression for distributional reinforce- reinffrre ment learning. In Advances in Neural Information Processing Systems, volume 33, pp. Curran Associates, Inc., 2020.
17

---

Published in Transactions on Machine Learning Research (04/2023)
A Appendix
A.1 Environment logic
We present a few code examples of the environment we provide in Scction Listing | shows the global stepping logic of the enviromment. Listing 2 shows the addition of items in the environment's stock: as individual items are represented by their shelf life, items are added via the sorting of the current stock to prevent item "mix-up". Finally, Listing shows part of the reward computation, when items are sold and removed from the stock matrix.
Listing 1: Stepping logic of the environment
def step (self, action): #Put bounds on action
new new_action =
torch as tensor (action, dtype-torch.inte int32)
clamp(0, self max stock)
.to device)
#If this is new day, we lake waste into accounl if self day tay_position % self. _iti___uusteppp__ count = 0: order cost = self. make fast order new action)
(sales, availability) = self generateeeatt sellf real clammp((.0)) 1.0)) waste self waste () Update waste and store result
self. reduceShelfLives( self. step counter +=
self. __ppateEur() clse:
self. day position +
order_ cost = self self._mllaa __aeee__ee_____ _eceeerrrrrrrreeeeeeee nne_acction)
(sales availability) = self generateDeemand (self. f. real clamp (0.0, 1.0)) waste torch. zeros(
self. assortment. size
# By default, no waste before the end of day self. (ppaatcbss))
sales sales.sub_ (order_
self. sales sales
self. total sales += sales self. waste waste
self.total waste += waste
self. self.availability availability
Listing 2: Addition of items to the stock
def addStock (sockk(slll) (self, units): #Create padding padding self self__max stock units
replenishment torch. strch.stacke (oack((nnits padding)) ). () reshape(-1) #Create vectors of new items
restock matrix self repeater seltf_repeater. repeateeetttteeee sespeat_interlavee
ropcats-replenishnne
long(). dim=0
). view (self. assortment size, self. max stock)
# Add new items to stock torch. add
self stock sort [1)
restock matrix sort (1, descending-True) (0),
18

---

Published in Transactions on Machine Learning Research (04/2023)
out=self. stock,
return
Listing 3: Selling units and updating stock
def
sellUnits (self, units): #Get the number of sales
sold torch min( self stock, ge (1).sum sum(1). double () units) #Compute availability
summ #Items with no demand are available
availability = self. stock ge (1).sum (1). double () div (units) clamp (0, 1)
availability [torch, isnan isnan (availability)) 1.0
#Compute sales sales
sold mul (2) subb (0_(uni) (units) mullseeff.. assortment, .asormee...elllgg_ppi........... self assortment, cost
(p, n) self stock. shape
#Update stock stock vector
stock. sort (1, descending-True) [0] view (-1)
to keep
units
interleaver = torch.stase (uuits to keep) )). t)) reshape (2, p). view (-1).
binary_ vec = torch. tensor ([0.0, 1]). repeat (p). rewperriieeeeeeveeeeeeeeeeee (interleaver)
= binary_ vec.
view (p, n)
return (sales, availability)
A.2 Training Curves
Figures
3
and show the raw training scores of the models presented in Section 5.2
Average Reward During Training, H=1 QRDQN
C51
DQN
ERDQN GTDQN
Reward
1e+5
80000
60000
Step
20k
40k
100k
Figure 3: Raw scores during training for H 1.
19

---

Published in Transactions on Machine Learning Research (04/2023)
Average Reward During Training, H=2
QRDQN
C51
ERDQN
DQN GTDQN
1.2e+5
Reward
1e+5
80000
60000
40000
20000
Step
100k
Figure 4: Raw scores during training for H 2.
20


Reviews:
[{"review_id": "lWwFRrwAvm", "reviewer": "Reviewer_sE4U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.The RetaiL environments benefit the RL for real life community.\n2.A new rl algorithm is proposed and outperforms previous baselines.\n\nWeaknesses:\n1.The POMDP formulation lacks more formal definition, such as the definition of the observation space.\n2.What is the length of the episode?\n3.How to evaluate the accuracy of the environment modeling? \n4.How to choose the value for lambda_1,...,lambda_4 in Eq.(4)?\n5.How many random seeds are used in experiments? In common, RL algorithms are evaluated in 5 random seeds.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper study the problem of applying reinforcement learning for waste reduction. A RL environment RetailL is proposed, which includes the details of item generation, demand generation, forecast generation and stock update. Then a distributional rl algorithm , GLDQN is proposed , and outperforms the previous RL baselines in both reward and waste reduction cases.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See Weaknesses. Please address these points and clarify my questions.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dPoxEGSvY4", "reviewer": "Reviewer_w2jW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) A new RL test environment has been defined and released in code.\n2) Some baseline performances have been established.\n\nWeaknesses:\n1) Some clarity issues in the writeup as I document below.\n2) I'm not sure how impactful this work is, and I don't think the algorithmic contribution is major. Perhaps the environment will turn out to be useful though I am not sure.\n3) How would a simple optimal-control baseline perform (eg, MPC or similar)? This is presumably how this type of problem is dealt with in practice.\n4) I don't really understand the results. I would like to see some learning curves for the various methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new simulation environment for inventory management and proposes a new type of Q-learning algorithm to solve it. The new environment code has also been released.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1) Can you clarify why the environment is a POMDP rather than simply an MDP? What information about the state is missing from the observation?\n2) Why are you modeling the Q-values as a lambda distribution? I don't see a justification for that choice.\n3) On the name 'Generalized lambda deep Q-network' - I wonder if there is any way to avoid overloading the 'lambda' here. I understand that you are using the lambda distribution, but for most RL people lambda binds very strongly to'lambda-returns', eg., Q-lambda, and I think this may cause confusion.\n4) Figure 1 'all DQN-based models', presumably you mean in the paper, not all published.\n5) Why are you using SELU actuvations?\n6) Figure 2 is also not giving us much.\n7) \"Instead of predicting the Q value, the model divides the possible reward interval in 51 (can be more or less) intervals.\" Is it the reward or the Q-values?\n8) Why are the summations in Eq. 1 so small?\n9) Why does ' forecast inaccuracy derive from epistemic uncertainty'? Surely there is aleatoric noise in the process being forecasted too.\n10) Please add learning curves to the results section.\n11) Please add an optimal control baseline.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3eKnL8dcaa", "reviewer": "Reviewer_pVwr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nS1: A domain that is both practically relevant and could also be used as a benchmark for testing RL algorithms\n\nS2: A simple variant of the distributional RL method using the (fully parameterized) generalized lambda distribution that seems to work well on the presented domains.\n\nS3: There are some missing details, but otherwise, the document is well-written.\n\n*Weaknesses*\n\nW1: The main weakness of the domain is related to the point of marginal demand of items being independent. Often even if one item is not there, customers might buy a related item. I suspect if each item had a feature/covariate then one could model similarity between the items as well.\n\nW2: See the requested changes section below.\n\n*Questions*\n\nQ1.  Why was the episodic setting chosen instead of the average reward setting? For these problems, it seems like the average reward is more useful, because discounting is not meaningful here, and having episodes as long as 5000 timesteps can blow up the return values (unless additional care is taken to adjust the reward magnitude or the number of timesteps). Further, how is a stock update and lead time handled toward the end of an episode in this episodic setting?\n\nQ2: How were the parameters of eqn 2 chosen?\n\nQ3: How do the modeling equations change when H=0?\n\nQ4: The time in purchase probability corresponds to 'time-step' or to 'time period'?\n\nQ5: How to compute the value of a given quantile when using the generalized lambda distribution?\n\nQ6: Experimental setup says that methods are evaluated on 100 unseen pseudo-items but Tables 1 and 2 show results on the 6000 items that were used during training.\n\nQ7: \"(H = 0). Yet, there is no significant difference between them\": Can the stderr be provided in Table 1 to deduce these?\n\nQ8: GLDQN: alation study for hyper-parameter sensitivity? How were hyper-parameters for the baseline and proposed method chosen?\n\nQ9: I am a little confused by how the term POMDP is used in this paper. Just because the transition and reward are stochastic, one need not need to model it as a POMDP (of course, one could consider the seed of the random variable as the hidden variable. If it was observed then everything in the domain would be deterministic). Till the distribution of the next state and reward is independent of the history, conditioned on the current observation and action, then one could still model it as an MDP. If this is not the case, then I would suggest authors to avoid mentioning the contraction of the distributional Bellman operator as it need not be a contraction in the POMDP case (even if state/action sets are finite).\n\nMinor:\n\n- Turn on line numbers for ease of review feedback\n- In conclusion: \" as it works in a FIFO manner\" -> \" as it works in a LIFO manner\"?\n- Should \"experiments setup\" and Table 1 have \"transitions of length …\" replaced with \"trajectories of length ..\"?\n- \"Learning the expectation of the Q-value is the most straightforward way…\": Q-value *is* the expected return. \n-  Is there a maximum stock capacity? \n- \"Yet, there is no difference for our agent, as both of them affect the reward and transitions in the environment\": Why dos forecast effect the transition probability?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The proposed work presents a new domain for benchmarking (RL) algorithms to manage perishable inventory in a shop/warehouse. The goal in this domain is to maximize profits and minimize waste. Observations correspond to features of the product (cost price, selling price, shelf life, popularity) and also an approximate forecast of its demand. These features are derived from (anonymized) realistic data. Actions correspond to the number of new units to order. Items get replenished based on order and get consumed based on a consumption model. \n\nApart from introducing the domain, the paper also introduces a variant of the distributional RL algorithm, that uses the generalized lambda distribution to parameterize the return distribution. Empirical results suggest that this provides a strong baseline for the proposed domain.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "*R1: Code documentation* The code could benefit a lot from proper documentation. While this wouldn't be a major point of concern for most papers, the proposed work is explicitly about developing an environment that others can use to test their algorithms. Currently, the submitted code lacks clear documentation. \n\n*R2: Visualizing/interpreting the results + reward hacking* Personally, having implemented a supply-demand domain based on queueing theory, I have found that implementing a new domain is often prone to subtle bugs. These bugs get exploited by RL agents to obtain scores better than non-adaptive/fixed/classical policies taken from the literature. Unfortunately, as a reviewer, it is hard to assess the correctness of domains. Therefore, I was wondering if the authors could provide some more interpretable results/visualization of what the agent is doing when it achieves significantly better than the baseline. This could not only help identify potential bugs but could also provide insights into how to manage inventory.\n\n*R3: Formal modeling equations* It would be very helpful to the readers if the modeling (or the core code) for the domain can be presented in the appendix of the paper.\n\n*R4: Describe the reward function:* I suspect that the reward function is just a weighted combination of profits, cost of waste, and other variables. Depending on the combination of the weights (i.e., how much importance does reducing waste have in the reward), I suspect that the results in table 2 can change significantly. Since this relates to a primary research question RQ2 posed by the paper, is it possible to provide an ablation study for that? ", "broader_impact_concerns": "I believe that the authors have done a good job of discussing these. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: A RL environment RetailL is proposed, which includes the details of item generation, demand generation, forecast generation and stock update. Then a distributional rl algorithm , GLDQN is proposed , and outperforms the previous RL baselines in both reward and waste reduction cases. The authors carefully responded to all questions and all reviewers unanimously propose accept.
Decision: Accept (poster)