SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Massil Hihat $^{1,2}$ , Stéphane Gaiffas $^{1,2}$ , Guillaume Garrigos $^{1,2}$ , Simon Bussy $^{1}$
<sup>1</sup>LOPF, Califrais' Machine Learning Lab, Paris, France
<sup>2</sup>Université Paris Cité and Sorbonne Université, CNRS,
Laboratoire de Probabilités, Statistique et Modélisation, Paris, France
{hihat, stephane.gaiffas, garrigos}@lpsm.paris, simon.bussy@califrais.fr
We study multi-product inventory control problems where a manager makes sequential replenishment decisions based on partial historical information in order to minimize its cumulative losses. Our motivation is to consider general demands, losses and dynamics to go beyond standard models which usually rely on newsvendor-type losses, fixed dynamics, and unrealistic i.i.d. demand assumptions. We propose MaxCOSD, an online algorithm that has provable guarantees even for problems with non-i.i.d. demands and stateful dynamics, including for instance perishability. We consider what we call non-degeneracy assumptions on the demand process, and argue that they are necessary to allow learning.
An inventory control problem is a problem faced by an inventory manager that must decide how much goods to order at each time period to meet demand for its products. The manager's decision is driven by the will to minimize a certain regret, which often penalizes missed sales and storage costs. It is a standard problem in operations research and operations management, and the reader unfamiliar with the topic can find a precise description of this problem in Section 2.
The classical literature of inventory management focuses on optimizing an inventory system with complete knowledge of its parameters: we know in advance the demands, or the distribution they will be drawn from. Many efforts have been put into characterizing the optimal ordering policies, and providing efficient algorithms to find them. See e.g. the Economic Order Quantity model [7], the Dynamic Lot-Size model [30] or the newsvendor model [1]. Nevertheless, in many applications, the parameters of the inventory system are unknown. In this case, the manager faces a joint learning and optimization problem that is typically framed as a sequential decision making problem: the manager bases its replenishment decisions on data that are collected over time, such as past demands (observable demand case) or past sales (censored demand case). The early attempts to solve these online inventory problems employed various techniques and provided only weak guarantees or no guarantees at all [23, 4, 6, 12].
With the recent advances in online learning frameworks such as online convex optimization (OCO), bandits, or learning with expert advice, the literature of learning algorithms for inventory problems took a great leap forward. There is currently a growing body of research aiming at solving various online inventory problems while providing strong theoretical guarantees in the form of regret bounds [9, 13, 24, 3, 34, 25, 35, 15, 32, 33, 31].
However, these works rely on mathematically convenient but unrealistic assumptions. The most common one being that the demands are assumed to be independent and identically distributed (i.i.d.) across time, which rules out correlations and nonstationarities that are common in real-world scenarios.
Furthermore, these works focus on specific cost structures (typically the newsvendor cost) and inventory dynamics (like lost sales models with nonperishable products), which we detail in Section 2.2. The main goal of this paper is to go beyond these restrictions and consider general demand processes, losses and dynamics, in order to provide numerical methods backed with theoretical guarantees compatible with real-world problems.
To do so, we recast the online inventory problem into a new framework called Online Inventory Optimization (OIO), which extends OCO. Our main contribution is a new algorithm called MaxCOSD, which can be seen as a generalization of the Online Subgradient Descent method. It solves OIO problems with provable theoretical guarantees under minimal assumptions (see Section 4). Here is an informal version of our main statement:
Theorem 1 (Informal version of Theorem 12). Consider an OIO problem that satisfies convexity and boundedness assumptions. Assume further that demands are not degenerate (see Assumption 10). Then, running MaxCOSD (see Algorithm 2) with adequate adaptive learning rates gives an optimal  $O(\sqrt{T})$  regret, both in expectation and in high probability.
Our main assumption is a non-degeneracy hypothesis on the demand process, which we present and discuss in Section 5. This assumption generalizes typical hypotheses made in the inventory literature, while not requiring the demand to be i.i.d.. We also show that this assumption is sharp, in the sense that the OIO cannot be solved without such assumption. Finally, in Section 6 we present numerical experiments on both synthetic and real-world data that validate empirically the versatility and performances of MaxCOSD. The appendices gather the proofs of all our statements.
This paper helps to bridge the gap between OCO and inventory optimization problems, and we hope that it will raise awareness of OCO researchers to this family of under-studied problems, while being of importance to the industry.
2 A general model for online inventory problems
In this section, we present a new but simple model allowing to study a large class of online inventory problems. Then, in a series of remarks we discuss particular instances of these problems and the limitations of our model.
2.1 Description of the model and main assumptions
In the following,  $n \in \mathbb{N} = \{1, 2, \ldots\}$  refers to the number of products and  $\mathcal{Y} \subset \mathbb{R}_+^n$  denotes the feasible set. An online inventory optimization (OIO) problem is a sequential decision problem where an inventory manager interacts with an environment according to the following protocol.
First, the environment sets the initial inventory state to zero ( $x_{1} = \mathbf{0} \in \mathbb{R}^{n}$ ), and chooses (possibly random) demands  $d_{t} \in \mathbb{R}_{+}^{n}$  and losses  $\ell_t: \mathbb{R}^n \to \mathbb{R}$  for every time period  $t \in \mathbb{N}$ . Then, the interactions begin and unfold as follows, for every time period  $t \in \mathbb{N}$ :
i) The manager observes the inventory state  $x_{t} \in \mathbb{R}^{n}$ , where  $x_{t,i}$  encodes the quantity of the  $i^{\text{th}}$  product available in the inventory.
ii) The manager raises this level by choosing an order-up-to level  $y_{t} \in \mathcal{V}$  which satisfies the feasibility constraint:
$$
y _ {t} \succeq x _ {t}. \tag {1}
$$
Then, the manager receives instantaneously  $y_{t,i} - x_{t,i} \geq 0$  units of the  $i^{\text{th}}$  product.
iii) The manager suffers a loss  $\ell_t(y_t)$  and observes a subgradient  $g_t \in \partial \ell_t(y_t)$ .
iv) The environment updates the inventory state by choosing  $x_{t + 1} \in \mathbb{R}^n$  satisfying the following inventory dynamical constraint:
$$
x _ {t + 1} \preceq \left[ y _ {t} - d _ {t} \right] ^ {+}. \tag {2}
$$
The goal of the manager is to design an online algorithm that produces feasible order-up-to levels  $y_{t}$  which minimizes the cumulative loss suffered using past observations (past inventory states and subgradients). Let us emphasize here the fact that demands and losses are not directly observable. Throughout the paper, we make the following assumptions on the feasible set and the losses.
Assumption 2 (Convex and bounded problem).
i) (Convex and bounded constraint) The feasible set  $\mathcal{V}$  is closed, convex, nonnegative  $(\mathcal{V} \subset \mathbb{R}_+^n)$  and bounded:  $\mathrm{diam} \mathcal{V} \leq D$  for some  $D \geq 0$ .
ii) (Convex losses) For every  $t \in \mathbb{N}$ , the loss function  $\ell_t$  is convex.
iii) (Uniformly bounded subgradients) There exists  $G > 0$  such that, for all  $t \in \mathbb{N}$ ,  $y \in \mathcal{V}$  and  $g \in \partial \ell_t(y)$  we have  $\| g \|_2 \leq G$ .
Apart from the non-negativity assumption  $\mathcal{Y} \subset \mathbb{R}_+^n$  which is specific to inventory problems, Assumption 2 is very common in online convex optimization [36].
Definition 3 (Regret). Given an horizon  $T \in \mathbb{N}$ , we measure the performance of an algorithm with the usual notion of regret which is defined as:
$$
R _ {T} = \sum_ {t = 1} ^ {T} \ell_ {t} \left(y _ {t}\right) - \inf  _ {y \in \mathcal {Y}} \sum_ {t = 1} ^ {T} \ell_ {t} (y). \tag {3}
$$
Observe that  $R_{T}$  is possibly random, so we call its expectation the expected regret  $\mathbb{E}[R_T]$ .
Notice that in our model any constant strategy is feasible (see Lemma 26 in the appendix). Thus, the regret (3) is well-defined and can be interpreted as the difference between the cumulative loss incurred by the algorithm and that incurred by the best feasible constant strategy<sup>1</sup>. It is an easy exercise to see that under Assumption 2 we always have  $R_{T} \leq DGT$  (see Lemma 27 in the appendix). Thus, our goal is to design algorithms with sublinear regret with respect to  $T$ , achieving  $\mathbb{E}[R_T] \leq o(T)$ . Note that some authors consider the equivalent notion of averaged regret  $(1 / T)R_{T}$ . In this context, we would talk about no-regret algorithms.
2.2 Description of standard inventory models and limitations of our model
Remark 4 (On the demands). Demands are modeled by a stochastic process  $(d_{t})_{t\in \mathbb{N}}\subset \mathbb{R}_{+}^{n}$  with fixed in advance distribution. In other words, we consider a general oblivious model for demands where we make no assumptions of regularity, stationarity or independence. Thus, our model accommodates both i.i.d. [9, 3, 25, 32, 33, 31] and deterministic demands [13, 24, 34, 35, 15], while also allowing for correlations and nonstationarities that appear for instance in autoregressive models. However, we rule out strategic behaviors: this is not a game-theoretic model.
Remark 5 (On the dynamic). Inventory states  $x_{t}$  are constrained by the inventory dynamical constraint (2) which links states, demands, and order-up-to levels. This constraint resembles the notion of partial perishability introduced in [9, Section 3.3] which imposes further that inventory states are non-negative. In the following, we present some standard dynamics that conform to our model. We warn the reader that we try here to simplify the vocabulary used in the scattered inventory literature, and that some authors [9, 3] may refer to these dynamics using different terms<sup>2</sup>.
- Stateless dynamic. In stateless inventory problems, we assume that no product is carried over from one period to the other, i.e.  $x_{t} = 0$ . Observe that due to the non-negativity assumption  $\mathcal{V} \subset \mathbb{R}_{+}^{n}$ , the feasibility constraint (1) is satisfied by any choice of  $y_{t} \in \mathcal{V}$  in stateless problems. This means that stateless inventory problems coincide with the usual online convex optimization (OCO) framework [36]. See Appendix D.1 for a discussion on the relationship between OCO and OIO. Any dynamic which is not stateless is called stateful, see below.
- Backlogging dynamic. In backlogging inventory problems, excess demand stays on the books until it is satisfied and inventory leftovers are carried over to the next period. It corresponds to set  $x_{t+1} = y_t - d_t$ . Notice that in this case the inventory state may be negative to represent backorders. This kind of dynamic has been widely studied in the context of classical inventory theory due to its linear nature, see e.g. [26, Chapter 4].
<sup>1</sup>In the context of inventory problems, these constant strategies are known under the name of (stationary) base-stock policies,  $S$ -policies, or order-up-to policies. See e.g. [26, Chapter 4].
For instance the stateless dynamic is usually referred to as the "perishable" setting, and lost sales and backlogging dynamics may both be referred to as "nonperishable" settings.
- Lost sales dynamic. Assume that products are nonperishable, excess demand is lost and inventory leftovers are carried over to the next period. We refer to this case as the lost sales dynamic which corresponds to set  $x_{t+1} = [y_t - d_t]^+$ . See e.g. [9, 25].
- Perishable dynamic. In perishable inventory systems [19], newly ordered products are fresh units that have a fixed usable lifetime. To model such a dynamic, it is necessary to track the entire age distribution of the on hand inventory and to specify the stockout type (lost sales or backlogging) and the issuing policy (i.e. how items are issued to meet demand). For instance, [32] describes a perishable setting modeling a single product with first-in-first-out issuing policy, which satisfies our inventory dynamical constraint (2).
All those dynamics are what we call deterministic dynamics, in the sense that they take the form:
$$
(\forall t \in \mathbb {N}) \quad x _ {t + 1} = X _ {t} \left(y _ {1}, d _ {1}, \dots , y _ {t}, d _ {t}\right), \tag {4}
$$
where  $X_{t}:(\mathcal{V}\times \mathbb{R}_{+}^{n})^{t}\to \mathbb{R}^{n}$  is a fixed in advance function satisfying  $X_{t}(y_{1}^{\prime},d_{1}^{\prime},\ldots ,y_{t}^{\prime},d_{t}^{\prime})\preceq [y_{t}^{\prime} - d_{t}^{\prime}]^{+}$  for all realizations  $(\ell_t^\prime)_{t\in \mathbb{N}}$ $(d_t^\prime)_{t\in \mathbb{N}}$  and  $(y_{t}^{\prime})_{t\in \mathbb{N}}$
Remark 6 (On the feasible set). The feasible set  $\mathcal{V}$  models fixed constraints on the order-up-to levels. Since it should satisfy Assumption 2.i), our framework does not allow for discrete sets as they appear in [15] for instance. This is one of the main limitations of our work. Typical choices include box constraints:  $\mathcal{V} = \prod_{i=1}^{n}[\underline{y}_i, \overline{y}_i]$ , or capacity constraints [25]:  $\mathcal{V} = \{y \in \mathbb{R}_+^n \mid \sum_{i=1}^{n} y_i \leq M\}$ . Note that in some instances, the parameters of the box constraint is dictated by some assumptions on the problem. For instance, it is assumed in [9, 32] that  $\mathcal{V} = [0, D]$  where  $D$  is a known upper bound for an optimal constant policy.
Remark 7 (On the losses). Losses  $(\ell_t)_{t\in \mathbb{N}}$  are random functions drawn before the interactions start. As for demands, we consider an oblivious model for the losses which allows to handle both the i.i.d. case and the deterministic case. Most interestingly, losses can depend on the demands. This allows to consider the newsvendor loss, which writes  $\ell_t(y) = c(y,d_t)$  with:
$$
c (y, d) = \sum_ {i = 1} ^ {n} \left(h _ {i} \left[ y _ {i} - d _ {i} \right] ^ {+} + p _ {i} \left[ d _ {i} - y _ {i} \right] ^ {+}\right). \tag {5}
$$
Here  $h_i \in \mathbb{R}_+$  and  $p_i \in \mathbb{R}_+$  are respectively the unit holding cost (a.k.a. average cost) and unit lost sales penalty cost (a.k.a. underage cost) of product  $i \in [n]$ . The newsvendor loss satisfy assumptions 2.ii) and 2.iii) with  $G = \sqrt{n} \max_{i \in [n]} \max \{h_i, p_i\}$ . Our model also accommodates the newsvendor loss with time-varying unit cost parameters, as long as these remain bounded.
Because the losses are drawn before-hand, our model usually does not allow to incorporate costs that depend explicitly on the inventory states such as purchase costs, outdating costs, or fixed costs. However, there are exceptions: when the dynamic is lost sales, purchase costs can be included into our model, by considering the newsvendor loss onto which a cost transformation is applied (a.k.a. explicit formulation [26, Paragraph 4.3.2.4]). See also [9, 25] and the references therein.
Remark 8 (On the observability structure and subgradients). To handle arbitrary losses, we required in our model that in addition to inventory states  $x_{t}$ , at least one subgradient  $g_{t} \in \partial \ell_{t}(y_{t})$  is revealed at each period. In the case of the newsvendor loss, this is less demanding than both the observable demand setting and the censored demand setting [3]. In the former, the demand  $d_{t}$  is revealed instead of a subgradient  $g_{t}$ , meaning that the manager has complete information on the newsvendor loss  $\ell_{t} = c(\cdot ,d_{t})$ . In the latter, the sale  $s_t \coloneqq \min \{y_t,d_t\}$  is revealed, allowing the manager to compute a subgradient through the following formula (see Lemma 28 in Appendix B):
$$
\left(h _ {i} \mathbb {1} _ {\{y _ {t, i} > s _ {t, i} \}} - p _ {i} \mathbb {1} _ {\{y _ {t, i} = s _ {t, i} \}}\right) _ {i \in [ n ]} \in \partial \ell_ {t} (y _ {t}).
$$
In this case, we see that no randomization is involved in the choice of the subgradient. This means that the subgradient selection is deterministic, in the sense that:
$$
(\forall t \in \mathbb {N}) \quad g _ {t} = \Gamma_ {t} \left(\ell_ {t}, y _ {t}\right), \tag {6}
$$
where  $\Gamma_t: \mathbb{R}^{n} \times \mathcal{Y} \to \mathbb{R}^n$  is a fixed in advance function such that  $\Gamma_t(\ell_t', y_t') \in \partial \ell_t'(y_t')$  for all realizations  $(\ell_t')_{t \in \mathbb{N}}$  and  $(y_t')_{t \in \mathbb{N}}$ .
Remark 9 (OIO vs OCO). In this final remark, we would like to point out that OIO is a novel and strict extension of OCO, that is, OIO cannot be casted into OCO or one of its known extensions. More details on this are provided in Appendix D.1.
3 Partial results for simple inventory problems
Previous works on online inventory problems mainly focused on two settings: stateless inventory problems and stateful inventory problems with i.i.d. demands. Both settings are discussed here.
3.1 Stateless inventory problems
In the literature of stateless inventory problems, arbitrary deterministic demands have already been considered. This has been done for instance in [13, 34, 35], which assume demand is observable and rely on the "learning with expert advice" framework. On the other hand [24] is the first work that considered the stateless setting with censored demand. See also [15] which tackled these problems in the discrete case, by reducing them to partial monitoring (an online learning framework that generalizes bandits). All these works achieve a  $O(\sqrt{T})$  regret (up to logarithmic terms), but are restricted to the newsvendor cost structure.
In our work, we aim at solving inventory problems with arbitrary demands and losses. Recall that under Assumption 2, stateless inventory problems coincide with the standard OCO framework [36] since feasibility (1) is trivially verified (see Remark 5). Thus, a natural choice to solve stateless inventory problems is the Online Subgradient Descent (OSD) method, which we recall in Algorithm 1.
Algorithm 1: OSD
1 Parameters: learning rates  $(\eta_{t})_{t\in \mathbb{N}}$  , initial order-up-to level  $y_{1}\in \mathcal{V}$    
2 for  $t = 1,2,\ldots$  do   
3 Output  $y_{t}$  .   
4 Observe  $g_{t}\in \partial \ell_{t}(y_{t})$  .   
5 Set  $y_{t + 1} = \mathrm{Proj}_{\mathcal{Y}}(y_t - \eta_t g_t)$
Classical regret analysis of OSD (this is essentially proven in [8, Theorem 3.1], see also Corollary 20 in the appendix) shows that taking decreasing learning rates of the form  $\eta_t = \gamma D / (G\sqrt{t})$  where  $\gamma > 0$ , leads to a regret bound  $R_T = O(GD\sqrt{T})$ . It must be noted that the  $O(\sqrt{T})$  scaling is optimal under Assumption 2 (see e.g. [22, Theorem 5] or [21, Theorem 5.1]).
3.2 Stateful i.i.d. inventory problems
When non-trivial dynamics are involved, inventory problems are much more complex and have mainly been studied in the i.i.d. demands framework. We review here the literature of joint learning and inventory control with censored demand and refer to the recent review of [5, Chapter 11] for further references. We stress that all those papers obtain rates for the pseudo-regret, a lower bound of the expected regret which we consider in this paper (see Appendix D.2 for more details).
The seminal work of Huh and Rushevichientong [9] is the first that derives regret bounds for the single-product i.i.d. newsvendor case under censored demand, it is also the sole work that considers general dynamics through their notion of partial perishability [9, Section 3.3]. They were able to design an algorithm called Adaptive Inventory Management (AIM) based on a subgradient descent and dynamic projections onto the feasibility constraint (1) which achieves a  $O(\sqrt{T})$  pseudo-regret. Their main assumption is that the demands should not be degenerate in the sense that  $\mathbb{E}[d_1] > 0$  and that the manager should know a lower bound  $0 < \rho < \mathbb{E}[d_1]$ . This lower bound is then used in AIM to tune adequately the learning rate of the subgradient descent. Their analysis is based on results from queuing theory which rely heavily on the i.i.d. assumption.
Shi et al. [25] designed the Data-Driven Multi-product (DDM) algorithm which extend the AIM method of [9] to the multi-product case under capacity constraints. They also derived a  $O(\sqrt{T})$  pseudo-regret bound by assuming further that demands are pairwise independent across products and that  $\mathbb{E}[d_{1,i}] > 0$  for all  $i\in [n]$  amongst other regularity conditions.
Zhang et al. [32] tackled the case of single-product i.i.d. perishable inventory systems with outdating costs. They designed the Cycle-Update Policy (CUP) algorithm which updates the order-up-to level according to a subgradient descent, but only when the system experiences a stockout, i.e. when  $x_{t} = 0$ , the order-up-to level remains unchanged otherwise. Feasibility is guaranteed using such
a policy. However, in order to derive  $O(\sqrt{T})$  pseudo-regret they need to ensure that the system experiences frequently stockouts. To do so, they consider a stronger form of non-degeneracy, namely,  $\mathbb{P}[d_1 \geq D] > 0$  where  $\mathcal{Y} = [0, D]$ .
Variants of the subgradient descent have also been developed in order to achieve  $O(\sqrt{T})$  pseudoregret in inventory systems that includes lead times [33] or fixed costs [31] which are both beyond the scope of our model.
To summarize, an optimal  $O(\sqrt{T})$  rate for the pseudo-regret is achievable in many stateful inventory problems, under the i.i.d. assumption. To prove so, most of the cited works developed specific variants of the subgradient descent that accommodates the specific dynamic at play. We will show in Section 4 that this optimal  $O(\sqrt{T})$  rate can be achieved by our algorithm MaxCOSD when applied to general inventory problems, with no i.i.d. assumption on the demand.
4 MaxCOSD: an algorithm for general inventory problems
In this section, we introduce and study our main algorithm: the Maximum Cyclic Online Subgradient Descent (MaxCOSD) algorithm. It is a variant of the subgradient descent where instead of changing the order-up-to levels at every period, updates are done only at certain update periods denoted  $(t_k)_{k\in \mathbb{N}}$ . These define update cycles  $\mathcal{T}_k = \{t_k,\dots ,t_{k + 1} - 1\}$  during which the order-up-to level remains unchanged:  $y_{t} = y_{t_{k}}$  for all  $t\in \mathcal{T}_k$ . Update periods are dynamically triggered by verifying, at the beginning of each time period  $t\in \mathbb{N}$ , whether a candidate order-up-to level  $\hat{y}_t$  is feasible or not. This candidate is computed by making a subgradient step in the direction of the subgradients accumulated during the cycle and using the following adaptive learning rates,
$$
\eta_ {t} = \frac {\gamma D}{\sqrt {\left\| \sum_ {s = t _ {k}} ^ {t} g _ {s} \right\| _ {2} ^ {2} + \sum_ {m = 1} ^ {k - 1} \left\| \sum_ {s \in \mathcal {T} _ {m}} g _ {s} \right\| _ {2} ^ {2}}} \quad \text {f o r a l l} t \in \mathcal {T} _ {k}. \tag {7}
$$
The pseudo-code for MaxCOSD is given in Algorithm 2.
Algorithm 2: MaxCOSD
1 Parameters: learning rate parameter  $\gamma >0$  , initial order-up-to level  $y_{1}\in \mathcal{V}$    
2 Initialization:   
3 Set  $\hat{y}_1 = y_1,t_1 = 1,k = 1$  .   
4 for  $t = 1,2,\ldots$  do   
5 Output  $y_{t}$    
6 Observe  $g_{t}\in \partial \ell_{t}(y_{t})$  and  $x_{t + 1}$  .   
7 Compute  $\hat{y}_{t + 1} = \mathrm{Proj}_{\mathcal{Y}}\left(\hat{y}_{t_k} - \eta_t\sum_{s = t_k}^t g_s\right)$  where  $\eta_t$  is defined in Eq. (7);   
8 if  $x_{t + 1}\preceq \hat{y}_{t + 1}$  then   
9 Set  $y_{t + 1} = \hat{y}_{t + 1},t_{k + 1} = t + 1,k = k + 1$    
10 else   
11 Set  $y_{t + 1} = y_{t}$