Title: Online Inventory Problems: Beyond the i.i.d. Setting with Online Convex Optimization
Venue: NeurIPS 2023
Accepted: yes
Keywords: online convex optimization

inventory control

newsvendor

online learning

regret analysis
Abstract: We study multi-product inventory control problems where a manager makes sequential replenishment decisions based on partial historical information in order to minimize its cumulative losses. Our motivation is to consider general demands, losses and dynamics to go beyond standard models which usually rely on newsvendor-type losses, fixed dynamics, and unrealistic i.i.d. demand assumptions. We propose MaxCOSD, an online algorithm that has provable guarantees even for problems with non-i.i.d. demands and stateful dynamics, including for instance perishability. We consider what we call non-degeneracy assumptions on the demand process, and argue that they are necessary to allow learning.

Online Inventory Problems: Beyond the i.i.d. Setting with Online Convex Optimization

Abstract

We study multi-product inventory control problems where a manager makes sequential replenishment decisions based on partial historical information in order to minimize its cumulative losses. Our motivation is to consider general demands, losses and dynamics to go beyond standard models which usually rely on newsvendor-type losses, fixed dynamics, and unrealistic i.i.d. demand assumptions. We propose MaxCOSD, an online algorithm that has provable guarantees even for problems with non-i.i.d. demands and stateful dynamics, including for instance perishability. We consider what we call non-degeneracy assumptions on the demand process, and argue that they are necessary to allow learning.

1 Introduction

An inventory control problem is a problem faced by an inventory manager that must decide how much goods to order at each time period to meet demand for its products. The manager's decision is driven by the will to minimize a certain regret , which often penalizes missed sales and storage costs. It is a standard problem in operations research and operations management, and the reader unfamiliar with the topic can find a precise description of this problem in Section 2.

The classical literature of inventory management focuses on optimizing an inventory system with complete knowledge of its parameters: we know in advance the demands, or the distribution they will be drawn from. Many efforts have been put into characterizing the optimal ordering policies, and providing efficient algorithms to find them. See e.g. the Economic Order Quantity model [7], the Dynamic Lot-Size model [30] or the newsvendor model [1]. Nevertheless, in many applications, the parameters of the inventory system are unknown. In this case, the manager faces a joint learning and optimization problem that is typically framed as a sequential decision making problem: the manager bases its replenishment decisions on data that are collected over time, such as past demands (observable demand case) or past sales (censored demand case). The early attempts to solve these online inventory problems employed various techniques and provided only weak guarantees or no guarantees at all [23, 4, 6, 12].

With the recent advances in online learning frameworks such as online convex optimization (OCO), bandits, or learning with expert advice, the literature of learning algorithms for inventory problems took a great leap forward. There is currently a growing body of research aiming at solving various online inventory problems while providing strong theoretical guarantees in the form of regret bounds [9, 13, 24, 3, 34, 25, 35, 15, 32, 33, 31].

However, these works rely on mathematically convenient but unrealistic assumptions. The most common one being that the demands are assumed to be independent and identically distributed ( i.i.d. ) across time, which rules out correlations and nonstationarities that are common in real-world scenarios.

Furthermore, these works focus on specific cost structures (typically the newsvendor cost) and inventory dynamics (like lost sales models with nonperishable products), which we detail in Section 2.2. The main goal of this paper is to go beyond these restrictions and consider general demand processes, losses and dynamics, in order to provide numerical methods backed with theoretical guarantees compatible with real-world problems.

To do so, we recast the online inventory problem into a new framework called Online Inventory Optimization (OIO), which extends OCO. Our main contribution is a new algorithm called MaxCOSD, which can be seen as a generalization of the Online Subgradient Descent method. It solves OIO problems with provable theoretical guarantees under minimial assumptions (see Section 4). Here is an informal version of our main statement:

Theorem 1 (Informal version of Theorem 12) . Consider an OIO problem that satisfies convexity and boundedness assumptions. Assume further that demands are not degenerate (see Assumption 10). Then, running MaxCOSD (see Algorithm 2) with adequate adaptive learning rates gives an optimal O ( √ T ) regret, both in expectation and in high probability.

Our main assumption is a non-degeneracy hypothesis on the demand process, which we present and discuss in Section 5. This assumption generalizes typical hypotheses made in the inventory literature, while not requiring the demand to be i.i.d. . We also show that this assumption is sharp, in the sense that the OIO cannot be solved without such assumption. Finally, in Section 6 we present numerical experiments on both synthetic and real-world data that validate empirically the versatility and performances of MaxCOSD. The appendices gather the proofs of all our statements.

This paper helps to bridge the gap between OCO and inventory optimization problems, and we hope that it will raise awareness of OCO researchers to this family of under-studied problems, while being of importance to the industry.

2 A general model for online inventory problems

In this section, we present a new but simple model allowing to study a large class of online inventory problems. Then, in a series of remarks we discuss particular instances of these problems and the limitations of our model.

2.1 Description of the model and main assumptions

In the following, n ∈ N = { 1 , 2 , . . . } refers to the number of products and Y ⊂ R n + denotes the feasible set . An online inventory optimization (OIO) problem is a sequential decision problem where an inventory manager interacts with an environment according to the following protocol.

First, the environment sets the initial inventory state to zero ( x 1 = 0 ∈ R n ), and chooses (possibly random) demands d t ∈ R n + and losses ℓ t : R n → R for every time period t ∈ N . Then, the interactions begin and unfold as follows, for every time period t ∈ N :

The manager observes the inventory state x t ∈ R n , where x t,i encodes the quantity of the i th product available in the inventory.

ii) The manager raises this level by choosing an order-up-to level y t ∈ Y which satisfies the feasibility constraint:

Then, the manager receives instantaneously y t,i -x t,i ≥ 0 units of the i th product.

iii) The manager suffers a loss ℓ t ( y t ) and observes a subgradient g t ∈ ∂ℓ t ( y t ) .

iv) The environment updates the inventory state by choosing x t +1 ∈ R n satisfying the following inventory dynamical constraint :

The goal of the manager is to design an online algorithm that produces feasible order-up-to levels y t which minimizes the cumulative loss suffered using past observations (past inventory states and subgradients). Let us emphasize here the fact that demands and losses are not directly observable. Throughout the paper, we make the following assumptions on the feasible set and the losses.

Assumption 2 (Convex and bounded problem) .

(Convex and bounded constraint) The feasible set Y is closed, convex, nonnegative ( Y ⊂ R n + ) and bounded: diam Y ≤ D for some D ≥ 0 .

ii) (Convex losses) For every t ∈ N , the loss function ℓ t is convex.

iii) (Uniformly bounded subgradients) There exists G > 0 such that, for all t ∈ N , y ∈ Y and g ∈ ∂ℓ t ( y ) we have ∥ g ∥ 2 ≤ G .

Apart from the non-negativity assumption Y ⊂ R n + which is specific to inventory problems, Assumption 2 is very common in online convex optimization [36].

Definition 3 (Regret) . Given an horizon T ∈ N , we measure the performance of an algorithm with the usual notion of regret which is defined as:

Observe that R T is possibly random, so we call its expectation the expected regret E [ R T ] .

Notice that in our model any constant strategy is feasible (see Lemma 26 in the appendix). Thus, the regret (3) is well-defined and can be interpreted as the difference between the cumulative loss incurred by the algorithm and that incurred by the best feasible constant strategy 1 . It is an easy exercise to see that under Assumption 2 we always have R T ≤ DGT (see Lemma 27 in the appendix). Thus, our goal is to design algorithms with sublinear regret with respect to T , achieveing E [ R T ] ≤ o ( T ) . Note that some authors consider the equivalent notion of averaged regret (1 /T ) R T . In this context, we would talk about no-regret algorithms.

2.2 Description of standard inventory models and limitations of our model

Remark 4 (On the demands) . Demands are modeled by a stochastic process ( d t ) t ∈ N ⊂ R n + with fixed in advance distribution. In other words, we consider a general oblivious model for demands where we make no assumptions of regularity, stationarity or independence. Thus, our model accommodates both i.i.d. [9, 3, 25, 32, 33, 31] and deterministic demands [13, 24, 34, 35, 15], while also allowing for correlations and nonstationarities that appear for instance in autoregressive models. However, we rule out strategic behaviors: this is not a game-theoretic model.

Remark 5 (On the dynamic) . Inventory states x t are constrained by the inventory dynamical constraint (2) which links states, demands, and order-up-to levels. This constraint resembles the notion of partial perishability introduced in [9, Section 3.3] which imposes further that inventory states are non-negative. In the following, we present some standard dynamics that conform to our model. We warn the reader that we try here to simplify the vocabulary used in the scattered inventory literature, and that some authors [9, 3] may refer to these dynamics using different terms 2 .

Stateless dynamic. In stateless inventory problems, we assume that no product is carried over from one period to the other, i.e. x t = 0 . Observe that due to the non-negativity assumption Y ⊂ R n + , the feasibility constraint (1) is satisfied by any choice of y t ∈ Y in stateless problems. This means that stateless inventory problems coincide with the usual online convex optimization (OCO) framework [36]. See Appendix D.1 for a discussion on the relationship between OCO and OIO. Any dynamic which is not stateless is called stateful , see below.

Backlogging dynamic. In backlogging inventory problems, excess demand stays on the books until it is satisfied and inventory leftovers are carried over to the next period. It corresponds to set x t +1 = y t -d t . Notice that in this case the inventory state may be negative to represent backorders. This kind of dynamic has been widely studied in the context of classical inventory theory due to its linear nature, see e.g. [26, Chapter 4].

Lost sales dynamic. Assume that products are nonperishable, excess demand is lost and inventory leftovers are carried over to the next period. We refer to this case as the lost sales dynamic which corresponds to set x t +1 = [ y t -d t ] + . See e.g. [9, 25].

Perishable dynamic. In perishable inventory systems [19], newly ordered products are fresh units that have a fixed usable lifetime. To model such a dynamic, it is necessary to track the entire age distribution of the on hand inventory and to specify the stockout type (lost sales or backlogging) and the issuing policy (i.e. how items are issued to meet demand). For instance, [32] describes a perishable setting modeling a single product with first-in-first-out issuing policy, which satisfies our inventory dynamical constraint (2).

All those dynamics are what we call deterministic dynamics, in the sense that they take the form:

where X t : ( Y × R n + ) t → R n is a fixed in advance function satisfying X t ( y ′ 1 , d ′ 1 , . . . , y ′ t , d ′ t ) ⪯ [ y ′ t -d ′ t ] + for all realizations ( ℓ ′ t ) t ∈ N , ( d ′ t ) t ∈ N and ( y ′ t ) t ∈ N .

Remark 6 (On the feasible set) . The feasible set Y models fixed constraints on the order-up-to levels. Since it should satisfy Assumption 2.i), our framework does not allow for discrete sets as they appear in [15] for instance. This is one of the main limitations of our work. Typical choices include box constraints: Y = ∏ n i =1 [ y i , y i ] , or capacity constraints [25]: Y = { y ∈ R n + | ∑ n i =1 y i ≤ M } . Note that in some instances, the parameters of the box constraint is dictated by some assumptions on the problem. For instance, it is assumed in [9, 32] that Y = [0 , D ] where D is a known upper bound for an optimal constant policy.

Remark 7 (On the losses) . Losses ( ℓ t ) t ∈ N are random functions drawn before the interactions start. As for demands, we consider an oblivious model for the losses which allows to handle both the i.i.d. case and the deterministic case. Most interestingly, losses can depend on the demands. This allows to consider the newsvendor loss, which writes ℓ t ( y ) = c ( y, d t ) with:

Here h i ∈ R + and p i ∈ R + are respectively the unit holding cost (a.k.a. overage cost) and unit lost sales penality cost (a.k.a. underage cost) of product i ∈ [ n ] . The newsvendor loss satisfy assumptions 2.ii) and 2.iii) with G = √ n max i ∈ [ n ] max { h i , p i } . Our model also accommodates the newsvendor loss with time-varying unit cost parameters, as long as these remain bounded.

Because the losses are drawn before-hand, our model usually does not allow to incorporate costs that depend explicitly on the inventory states such as purchase costs, outdating costs, or fixed costs. However, there are exceptions: when the dynamic is lost sales, purchase costs can be included into our model, by considering the newsvendor loss onto which a cost transformation is applied (a.k.a. explicit formulation [26, Paragraph 4.3.2.4]). See also [9, 25] and the references therein.

Remark 8 (On the observability structure and subgradients) . To handle arbitrary losses, we required in our model that in addition to inventory states x t , at least one subgradient g t ∈ ∂ℓ t ( y t ) is revealed at each period. In the case of the newsvendor loss, this is less demanding than both the observable demand setting and the censored demand setting [3]. In the former, the demand d t is revealed instead of a subgradient g t , meaning that the manager has complete information on the newsvendor loss ℓ t = c ( · , d t ) . In the latter, the sale s t := min { y t , d t } is revealed, allowing the manager to compute a subgradient through the following formula (see Lemma 28 in Appendix B):

In this case, we see that no randomization is involved in the choice of the subgradient. This means that the subgradient selection is deterministic , in the sense that:

where Γ t : R R n × Y → R n is a fixed in advance function such that Γ t ( ℓ ′ t , y ′ t ) ∈ ∂ℓ ′ t ( y ′ t ) for all realizations ( ℓ ′ t ) t ∈ N and ( y ′ t ) t ∈ N .

Remark 9 (OIO vs OCO) . In this final remark, we would like to point out that OIO is a novel and strict extension of OCO, that is, OIO cannot be casted into OCO or one of its known extensions. More details on this are provided in Appendix D.1.

3 Partial results for simple inventory problems

Previous works on online inventory problems mainly focused on two settings: stateless inventory problems and stateful inventory problems with i.i.d. demands. Both settings are discussed here.

3.1 Stateless inventory problems

In the literature of stateless inventory problems, arbitrary deterministic demands have already been considered. This has been done for instance in [13, 34, 35], which assume demand is observable and rely on the "learning with expert advice" framework. On the other hand [24] is the first work that considered the stateless setting with censored demand. See also [15] which tackled these problems in the discrete case, by reducing them to partial monitoring (an online learning framework that generalizes bandits). All these works achieve a O ( √ T ) regret (up to logarithmic terms), but are restricted to the newsvendor cost structure.

In our work, we aim at solving inventory problems with arbitrary demands and losses. Recall that under Assumption 2, stateless inventory problems coincide with the standard OCO framework [36] since feasibility (1) is trivially verified (see Remark 5). Thus, a natural choice to solve stateless inventory problems is the Online Subgradient Descent (OSD) method, which we recall in Algorithm 1.

Algorithm 1: OSD

Classical regret analysis of OSD (this is essentially proven in [8, Theorem 3.1], see also Corollary 20 in the appendix) shows that taking decreasing learning rates of the form η t = γD/ ( G √ t ) where γ > 0 , leads to a regret bound R T = O ( GD √ T ) . It must be noted that the O ( √ T ) scaling is optimal under Assumption 2 (see e.g. [22, Theorem 5] or [21, Theorem 5.1]).

3.2 Stateful i.i.d. inventory problems

When non-trivial dynamics are involved, inventory problems are much more complex and have mainly been studied in the i.i.d. demands framework. We review here the literature of joint learning and inventory control with censored demand and refer to the recent review of [5, Chapter 11] for further references. We stress that all those papers obtain rates for the pseudo-regret , a lower bound of the expected regret which we consider in this paper (see Appendix D.2 for more details).

The seminal work of Huh and Rusmevichientong [9] is the first that derives regret bounds for the single-product i.i.d. newsvendor case under censored demand, it is also the sole work that considers general dynamics through their notion of partial perishability [9, Section 3.3]. They were able to design an algorithm called Adaptive Inventory Management (AIM) based on a subgradient descent and dynamic projections onto the feasibility constraint (1) which achieves a O ( √ T ) pseudo-regret. Their main assumption is that the demands should not be degenerate in the sense that E [ d 1 ] > 0 and that the manager should know a lower bound 0 < ρ < E [ d 1 ] . This lower bound is then used in AIM to tune adequately the learning rate of the subgradient descent. Their analysis is based on results from queuing theory which rely heavily on the i.i.d. assumption.

Shi et al. [25] designed the Data-Driven Multi-product (DDM) algorithm which extend the AIM method of [9] to the multi-product case under capacity constraints. They also derived a O ( √ T ) pseudo-regret bound by assuming further that demands are pairwise independent across products and that E [ d 1 ,i ] > 0 for all i ∈ [ n ] amongst other regularity conditions.

Zhang et al. [32] tackled the case of single-product i.i.d. perishable inventory systems with outdating costs. They designed the Cycle-Update Policy (CUP) algorithm which updates the order-up-to level according to a subgradient descent, but only when the system experiences a stockout, i.e. when x t = 0 , the order-up-to level remains unchanged otherwise. Feasibility is guaranteed using such

a policy. However, in order to derive O ( √ T ) pseudo-regret they need to ensure that the system experiences frequently stockouts. To do so, they consider a stronger form of non-degeneracy, namely, P [ d 1 ≥ D ] > 0 where Y = [0 , D ] .

Variants of the subgradient descent have also been developed in order to achieve O ( √ T ) pseudoregret in inventory systems that includes lead times [33] or fixed costs [31] which are both beyond the scope of our model.

To summarize, an optimal O ( √ T ) rate for the pseudo-regret is achievable in many stateful inventory problems, under the i.i.d. assumption. To prove so, most of the cited works developed specific variants of the subgradient descent that accommodates the specific dynamic at play. We will show in Section 4 that this optimal O ( √ T ) rate can be achieved by our algorithm MaxCOSD when applied to general inventory problems, with no i.i.d. assumption on the demand.

4 MaxCOSD: an algorithm for general inventory problems

In this section, we introduce and study our main algorithm: the Maximum Cyclic Online Subgradient Descent (MaxCOSD) algorithm. It is a variant of the subgradient descent where instead of changing the order-up-to levels at every period, updates are done only at certain update periods denoted ( t k ) k ∈ N . These define update cycles T k = { t k , . . . , t k +1 -1 } during which the order-up-to level remains unchanged: y t = y t k for all t ∈ T k . Update periods are dynamically triggered by verifying, at the beginning of each time period t ∈ N , whether a candidate order-up-to level ˆ y t is feasible or not. This candidate is computed by making a subgradient step in the direction of the subgradients accumulated during the cycle and using the following adaptive learning rates, 3

The pseudo-code for MaxCOSD is given in Algorithm 2.

MaxCOSD is inspired by CUP [32], which handles the feasibility constraint through cyclical updates. This approach differs for instance from AIM [9] or DDM [25], where the feasibility is enforced through projections onto the constraint. Among the differences between MaxCOSD and CUP is the fact that their cycle definition differ: in CUP stockouts trigger updates, whereas MaxCOSD relies directly on the feasibility condition, making its updates more frequent. Also, we use adaptive learning rates inspired by AdaGrad-Norm learning rates [27, Theorem 2] which allows us to be adaptive to the constant G and obtain high probability regret bounds which are not available for CUP. Finally, and most importantly, the assumptions required by CUP are restrictive: i.i.d. demands, single-product, perishable dynamic and a strong form of a demand non-degeneracy. On the other hand, MaxCOSD performs well under much milder assumptions, which we introduce next.

Assumption 10 (Uniformly probably positive demand) . There exists µ ∈ (0 , 1] and ρ > 0 such that, for all t ∈ N , almost surely,

In simple settings we recover through Assumption 10 conditions that already appeared in the literature:

In single-product i.i.d. newsvendor inventory problems, our assumption is equivalent to the existence of ρ > 0 such that P [ d 1 ≥ ρ ] > 0 , that is, P [ d 1 > 0] > 0 , or equivalently E [ d 1 ] > 0 . This is exactly the non-degeneracy assumption required by [9] in AIM.

In its multi-product extension, [25] assumes also pairwise independence across products. If we rather require mutual independence across products, then, Assumption 10 rewrites P [ d 1 , 1 > 0] · · · P [ d 1 ,n > 0] > 0 , thus, our assumption reduces to E [ d 1 ,i ] > 0 for all i ∈ [ n ] which is also required by [25] in their algorithm DDM.

We recover an assumption made for CUP [32] by requiring ρ = D where Y = [0 , D ] , and where D is also assumed to be large enough (see Remark 6).

If the demand is deterministic, then µ = 1 and Eq. (8) becomes d t,i ≥ ρ for all i ∈ [ n ] .

If the demand is discrete, i.e. d t,i ∈ { 0 , 1 , . . . } for all t ∈ N , i ∈ [ n ] , then, we can take ρ = 1 and rewrite Eq. (8) as follows: P [ ∃ i ∈ [ n ] , d t,i = 0 | ℓ 1 , d 1 , . . . , ℓ t -1 , d t -1 ] ≤ 1 -µ .

In addition to Assumption 10 we also introduce a mild technical condition.

Assumption 11 (Deterministic dynamic and subgradient selection) . Dynamics and subgradient selections are deterministic, see Eq. (4) and Eq. (6).

We can now state our main result: under this new set of assumptions MaxCOSD achieves an optimal O ( √ T ) regret bound both in expectation and in high probability.

Theorem 12. Consider an inventory problem, and let assumptions 2, 10 and 11 hold. Then, MaxCOSD (see Algorithm 2) run with y 1 ∈ Y and γ > 0 is feasible. Furthermore, when γ ∈ (0 , ρ/D ] , it enjoys the following regret bounds for all T ∈ N ,

and for any confidence level δ ∈ (0 , 1) we have with probability at least 1 -δ ,

The obtained regret scales in O ( √ T ) , which is similar to other methods in the literature (see Section 3.2). As for the scaling in the constants µ and ρ , our rate scales like O ( 1 ρµ ) and we do not know whether this can be improved or not. As a matter of comparison, CUP enjoys the same scaling (see [32, Theorem 2 & Remark 3 & Assumption 1]).

5 Non-degenerate demands are needed for stateful inventory problems

Throughout this paper, we have seen instances of OIO which can be solved with sublinear regret rates: stateless OIO (equivalent to OCO), and some stateful OIO. It must be noted that, contrary to OCO, solving those stateful OIO problems required a non-degeneracy assumption on the demand (see Assumption 10 and Section 3.2). We argue here that such an assumption is necessary for solving stateful OIO. Note that this idea is not new, and was already observed in the conclusion of [3]: "To control for the impact of overordering, demands must be bounded away from zero, at least in expectation." . Our contribution is to make this observation formal.

Proposition 13. Given any feasible deterministic 4 algorithm for the single-product lost sales newsvendor inventory problem with observable demand over Y = [0 , D ] , there exists a sequence of demands such that the regret is linear, i.e. R T = Θ( T ) .

Proposition 13 shows that Assumption 2 is not sufficient to reach sublinear regret in general inventory problems. This is totally unusual from an OCO perspective, and is a specificity of stateful OIO. Furthermore, the above result shows that what prevents us from reaching sublinear rates is not the limited feedback (demands and losses are observable in this example) but rather zero demands . This is why it is necessary to make an assumption preventing demands to be too small, in some sense. Note that one may think of imposing positive demands to circumvent this difficulty, but this is not sufficient. Indeed, a sequence of demands converging too fast to zero can also be problematic.

Proposition 14. Given any feasible algorithm for the single-product lost sales problem with observable demand over Y = [0 , D ] such that y 1 ∈ (0 , D ] , there exists a constant sequence of losses and a sequence of positive demands such that the regret is linear, i.e. R T = Θ( T ) .

Let us now investigate why degenerated demand becomes a problem when going from stateless to stateful OIO. The main difference between the two is that the feasibility constraint is always trivially satisfied for stateless OIO (see Remark 5). Instead, for stateful OIO, we can show that the higher is the demand, the easier it is for the feasibility constraint to be satisfied.

Lemma 15. Let y, y ′ , d ∈ R n + . If ∥ y ′ -y ∥ 2 ≤ min i ∈ [ n ] d i , then, y ′ ⪰ [ y -d ] + . In particular, given an inventory problem and a time period t ∈ N , taking y t +1 ∈ Y ⊂ R n + such that ∥ y t +1 -y t ∥ 2 ≤ min i ∈ [ n ] d i,t ensures that y t +1 is feasible, in the sense that x t +1 ⪯ y t +1 .

The above lemma shows that if y t +1 is taken close enough from the previous y t , then the algorithm is feasible. The key point here is that "close enough" is controlled by the demand, meaning that when the demand is closer to zero there are less feasible choices for the manager. In such a case, we understand that it may be impossible to achieve sublinear regret, because the set of feasible choices could be too reduced.

The distance between two consecutive decisions can easily be controlled in methods based on subgradient descents, through their learning rates. This is why Lemma 15 is very helpful in the design of efficient feasible algorithms. It has been employed in the proof of our main result regarding MaxCOSD (Theorem 12). In the following, we further illustrate its usefulness by showing that OSD (see Algorithm 1) with adequate learning rates is feasible when the demand is uniformly positive .

Assumption 16 (Uniformly positive demand) . There exists ρ > 0 such that for all t ∈ N , i ∈ [ n ] ,

Theorem 17. Consider an inventory problem, and let assumptions 2 and 16 hold. Then, OSD (see Algorithm 1) run with y 1 ∈ Y and η t = γD/ ( G √ t ) where γ ∈ (0 , ρ/D ] , is feasible and satisfies for all T ∈ N that R T ≤ (1 + 2 γ )(2 γ ) -1 GD √ T .

6 Numerical results

The goal of the following numerical experiments 5 is to show the versatility and performances of MaxCOSD in various settings. Let us consider the following problems.

Setting 1. Single-product lost sales inventory problem with i.i.d. demands drawn according to Poisson (1) .

Setting 2. Single-product perishable inventory problem with a lifetime of 2 periods and i.i.d. demands drawn according to Poisson (1) .

Setting 3. Multi-product lost sales inventory problem with n = 100 and capacity constraints. Demands are i.i.d. and drawn independently across products according to Poisson ( λ i ) where the intensities λ i have been drawn independently according to Uniform [1 , 2] .

Setting 4. Multi-product lost sales inventory problem with n = 3049 and capacity constraints. Demands are taken from the real-world dataset of the M5 competition [17].

Setting 5. Multi-product lost sales inventory problem with n = 3049 and box constraints. As in Setting 4 we considered demands from the M5 competition dataset [17].

105

105

107

104

104

106

103

AIM

CUP

DDM

MaxCOSD

103

MaxCoSD

105

MaxCOSD

10-5

10-4

10-3

10-2

10-1

100

101

105

10-4

10-3

10-2

10-1

100

101

10-5

10-4

103

102

10-1

100

101

108

108

107

106

107

DDM

105

AlM(Parallel)

MaxCOSD

MaxCoSD(Parallel)

10-5

10-4

10-3

10-2

10-1

100

101

10-5

10-4

10-3

102

10-1

100

101

人

We use the newsvendor loss in all the settings. In all the settings the cost parameters satisfy p i /h i = 200 since this ratio is known to exceed 200 in many applications [10]. In settings 1, 2 and 3, h i = 1 and in settings 4 and 5, h i and p i are proportional to the real average selling costs.

In settings 1, 2, 3 and 4 we compare MaxCOSD against the following baselines: AIM [9] for Setting 1, CUP [32] for Setting 2, and DDM [25] for settings 3 and 4. In Setting 5, we ran a parallelized version of MaxCOSD, that is, one instance of MaxCOSD per product, and a parallelized version of AIM. Notice that in Settings 4 and 5 demands are not i.i.d. and thus, do not fit the assumptions of the baselines considered. Notice also that Theorem 12 requires MaxCOSD's learning rate to be small enough ( γ ≤ ρ/D ), which we do not try to enforce here. All the algorithms have been initialized with y 1 = 0 . Settings 1, 2 and 3 have been run 10 times, with different demand realizations generated through independent samples.

Figure 1 shows, for every setting, the regret obtained after T periods as a function of the learning rate parameter γ ∈ [10 -5 , 10 1 ] . We picked T = 1969 for all the settings because it corresponds to the number of periods available in our real-world dataset [17]. We see that MaxCOSD performs well compared to baselines whenever the number of handled products remains low (Settings 1,2,3,5). Instead, we see that MaxCOSD is less efficient when the number of products becomes large, in particular in the Setting 4. The performance of MaxCOSD in the large n regime can be explained by the fact that the cycles become longer, as it becomes less likely that the feasibility condition is satisfied. Indeed, we have seen in Lemma 15 that the larger the overall demand is, the easier feasibility holds. But when n grows, min i d i,t becomes smaller, making the problem harder.

7 Conclusion

In this paper, we address Online Inventory Optimization problems by introducing MaxCOSD, the first algorithm which can be applied to a wide variety of real-world problems. More precisely, MaxCOSD enjoys an optimal O ( √ T ) regret without assuming the demand to be i.i.d. , and can handle a large class of dynamics, including perishability models. We achieved this result by applying ideas and methods from online learning to OIO problems.

Still, there is a lot of space for improvements and future developments. First, we observed that for problems with a large number of products and capacity constraints, the empirical performance of MaxCOSD could be improved. To do so, one would need to better handle the feasibility constraint, by using for instance projections onto the feasibilitly constraint, an idea already used by DDM, but with no theoretical guarantees so far in real-world scenarios. Second, we obtained regret bounds under minimal structural assumptions on the problem (convex lipschitz losses), but we could expect to obtain better rates by making stronger assumptions on the problem. One such assumption, which is classical in the Online Convex Optimization literature, is to assume further that the losses are strongly convex or exp-concave, typically leading to a logarithmic O (log( T )) regret. Another assumption,

which is more specific to the Online Inventory literature, is to make some regularity assumption directly on the demand, also leading to a logarithmic pseudo-regret in the newsvendor case [3, Subsection 2.3]. Finally, even if our model is quite versatile, it has its limits, and more work is needed to improve it. For instance, we have seen in Section 2.2 that we do not accommodate for outdating costs as they appear in [32]. Also, we do not handle discrete feasible sets for which two promising techniques have been applied in the literature: expert algorithms [15] and probabilistic rounding [9, Subsection 3.4]. We could also hope to further weaken our non-degeneracy Assumption 10, by making an hypothesis which applies independently to each product, without having to assume pairwise independence across products. We believe that an adequate adaptation of online convex optimization techniques to the online inventory framework will prove to be a successful strategy for overcoming those challenges.

References

Kenneth J Arrow, Theodore Harris, and Jacob Marschak. Optimal inventory policy. Econometrica: Journal of the Econometric Society , pages 250-272, 1951.

Masoud Badiei, Na Li, and Adam Wierman. Online convex optimization with ramp constraints. In 2015 54th IEEE Conference on Decision and Control (CDC) , pages 6730-6736. IEEE, 2015.

Omar Besbes and Alp Muharremoglu. On implications of demand censoring in the newsvendor problem. Management Science , 59(6):1407-1424, 2013.

Apostolos N Burnetas and Craig E Smith. Adaptive ordering and pricing for perishable products. Operations Research , 48(3):436-443, 2000.

Xi Chen, Stefanus Jasin, and Cong Shi. The Elements of Joint Learning and Optimization in Operations Management , volume 18. Springer Nature, 2022.

Gregory A Godfrey and Warren B Powell. An adaptive, distribution-free algorithm for the newsvendor problem with censored demands, with applications to inventory and distribution. Management Science , 47 (8):1101-1112, 2001.

Ford W Harris. How many parts to make at once. 1913.

Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends® in Optimization , 2 (3-4):157-325, 2016.

Woonghee Tim Huh and Paat Rusmevichientong. A nonparametric asymptotic analysis of inventory planning with censored demand. Mathematics of Operations Research , 34(1):103-123, 2009.

Woonghee Tim Huh, Ganesh Janakiraman, John A Muckstadt, and Paat Rusmevichientong. An adaptive algorithm for finding the optimal base-stock policy in lost sales inventory systems with censored demand. Mathematics of Operations Research , 34(2):397-416, 2009.

Rodolphe Jenatton, Jim Huang, and Cédric Archambeau. Adaptive algorithms for online convex optimization with long-term constraints. In International Conference on Machine Learning , pages 402-411. PMLR, 2016.

Retsef Levi, Robin O Roundy, and David B Shmoys. Provably near-optimal sampling-based policies for stochastic inventory control models. Mathematics of Operations Research , 32(4):821-839, 2007.

Tatsiana Levina, Yuri Levin, Jeff McGill, Mikhail Nediak, and Vladimir Vovk. Weak aggregating algorithm for the distribution-free perishable inventory problem. Operations Research Letters , 38(6):516-521, 2010.

Nikolaos Liakopoulos, Apostolos Destounis, Georgios Paschos, Thrasyvoulos Spyropoulos, and Panayotis Mertikopoulos. Cautious regret minimization: Online optimization with long-term budget constraints. In International Conference on Machine Learning , pages 3944-3952. PMLR, 2019.

Gábor Lugosi, Mihalis G Markakis, and Gergely Neu. On the hardness of inventory management with censored demand data. arXiv preprint arXiv:1710.05739 , 2017.

Mehrdad Mahdavi, Rong Jin, and Tianbao Yang. Trading regret for efficiency: online convex optimization with long term constraints. The Journal of Machine Learning Research , 13(1):2503-2528, 2012.

Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos. The m5 competition: Background, organization, and implementation. International Journal of Forecasting , 38(4):1325-1336, 2022.

Shie Mannor, John N Tsitsiklis, and Jia Yuan Yu. Online learning with sample path constraints. Journal of Machine Learning Research , 10(3), 2009.

Steven Nahmias. Perishable inventory systems , volume 160. Springer Science & Business Media, 2011.

Michael J Neely and Hao Yu. Online convex optimization with time-varying constraints. arXiv preprint arXiv:1702.04783 , 2017.

Francesco Orabona. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Francesco Orabona and Dávid Pál. Scale-free online learning. Theoretical Computer Science , 716:50-69, 2018.

Herbert Scarf. Bayes solutions of the statistical inventory problem. The annals of mathematical statistics , 30(2):490-508, 1959.

Peter Sempolinski and Amitabh Chaudhary. Online algorithms for the newsvendor problem with and without censored demands. In Frontiers in Algorithmics: 4th International Workshop, FAW 2010, Wuhan, China, August 11-13, 2010. Proceedings 4 , pages 234-249. Springer, 2010.

Cong Shi, Weidong Chen, and Izak Duenyas. Nonparametric data-driven algorithms for multiproduct inventory systems with censored demand. Operations Research , 64(2):362-370, 2016.

Lawrence V Snyder and Zuo-Jun Max Shen. Fundamentals of supply chain theory . John Wiley & Sons, 2019.

Matthew Streeter and H Brendan McMahan. Less regret via online conditioning. arXiv preprint arXiv:1002.4862 , 2010.

Wen Sun, Debadeepta Dey, and Ashish Kapoor. Safety-aware algorithms for adversarial contextual bandit. In International Conference on Machine Learning , pages 3280-3288. PMLR, 2017.

Roman Vershynin. High-dimensional probability: An introduction with applications in data science , volume 47. Cambridge university press, 2018.

Harvey M Wagner and Thomson M Whitin. Dynamic version of the economic lot size model. Management science , 5(1):89-96, 1958.

Hao Yuan, Qi Luo, and Cong Shi. Marrying stochastic gradient descent with bandits: Learning algorithms for inventory systems with fixed costs. Management Science , 67(10):6089-6115, 2021.

Huanan Zhang, Xiuli Chao, and Cong Shi. Perishable inventory systems: Convexity results for base-stock policies and learning algorithms under censored demand. Operations Research , 66(5):1276-1286, 2018.

Huanan Zhang, Xiuli Chao, and Cong Shi. Closing the gap: A learning algorithm for lost-sales inventory systems with lead times. Management Science , 66(5):1962-1980, 2020.

Yong Zhang, Vladimir Vovk, and Weiguo Zhang. Probability-free solutions to the non-stationary newsvendor problem. Annals of Operations Research , 223(1):433-449, 2014.

Yong Zhang, Xingyu Yang, and Baixun Li. Distribution-free solutions to the extended multi-period newsboy problem. Journal of Industrial & Management Optimization , 13(2):633, 2017.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th international conference on machine learning (icml-03) , pages 928-936, 2003.

A Analysis of MaxCOSD

The goal of this appendix is to study the MaxCOSD algorithm (see Algorithm 2) and prove Theorem 12. To do so, we start by introducing a generalized version of MaxCOSD named Cyclic Online Subgradient Descent (COSD) which has general update periods and learning rates (see Algorithm 3). In Proposition 19 we provide a general analysis of COSD which shows that it is sufficient to control the cycles' length to derive O ( √ T ) regret bounds. As a byproduct of this proposition we also derive the classical analysis of OSD (see Corollary 20). A way of ensuring that the cycles' lengths are efficiently controlled is through a probabilistic property (see Assumption 21) which is similar to sub-exponential concentration [29, Section 2.7]. Finally, using Lemma 15 and Assumptions 2, 10 and 11 we show that when γ ∈ (0 , ρ/D ] the cycles' length of MaxCOSD satisfy Assumption 21 which allows us to conclude with Theorem 12.

A.1 Design of COSD

The Cyclic Online Subgradient Descent (COSD) generalizes MaxCOSD by allowing arbitrary learning rates and update periods. Recall that update periods are allowed to be dynamically defined, for this reason we will refer to a sequence of update periods ( t k ) k ∈ N as an update strategy which is formally defined below.

Definition 18 (Update strategy) . An update strategy ( t k ) k ∈ N ⊂ N is a sequence of random variables such that, t 1 = 1 , t k < t k +1 for every k ∈ N , and for every t ∈ N , k ∈ N , the event { t k = t } is observable by the manager at the beginning of period t , i.e. it belongs to σ ( g 1 , x 2 , . . . , g t -1 , x t ) .

The pseudo-code of COSD is given in Algorithm 3.

Algorithm 3: COSD

By choosing appropriately the update strategy we recover the following algorithms:

OSD. If updates are made at every period, i.e. t k = k , we recover OSD. Its feasibility is not guaranteed.

Minibatch OSD. Given a fixed minibatch size τ ∈ N , the updates period are defined offline via t k = 1 + ( k -1) τ .

CUP. For every k ∈ N , t k is defined as the periods where the inventory is empty, that is,

This update strategy corresponds to that used by CUP [32]. One easily sees that for such update periods t k , feasibility always holds.

MaxCOSD. To recover MaxCOSD we define dynamically the update strategy as the periods where the candidate order-up-to levels (ˆ y t ) t ∈ N are feasible. Formally, it writes:

where (ˆ y t ) t ∈ N is defined by:

with k = max { j ≥ 1 : t j ≤ t } or equivalently t ∈ T k = { t k , . . . , t k +1 -1 } . Notice that at update periods the implemented order-up-to levels coincide with the candidate order-up-to level, that is, we have y t k = ˆ y t k for all k ∈ N .

A.2 Analysis of COSD and OSD

The following proposition summarizes the main properties of COSD. For convenience, we will use the notation ¯ t k := t k +1 -1 for the last period of the k th cycle T k .

Proposition 19. Let assumptions 2.i) and 2.ii) be satisfied. Given any update strategy and any sequence of learning rates ( η t ) t ∈ N such that: 0 < η ¯ t k +1 ≤ η ¯ t k for all k ∈ N , COSD (see Algorithm 3) has the following properties:

it is feasible if and only if for all k ∈ { 2 , 3 , . . . } , y t k ⪰ x t k ,

ii) for all K ∈ N , the regret at the end of the K th cycle satisfies,

Proof. Let us prove claim i). By definition, the algorithm produces feasible sequence of order-up-to levels if y t ⪰ x t for all t ∈ N . But for all t ∈ { t k +1 , . . . , ¯ t k } for some k ∈ N , we have necessarily y t = y t -1 ⪰ [ y t -1 -d t -1 ] + ⪰ x t where the last inequality comes from (2). Thus we only need to check feasibility at the update periods, i.e. check that y t k ⪰ x t k for all k ∈ N . Furthermore, it is clear that the latter is verified for k = 1 since x t 1 = x 1 = 0 ⪯ y t 1 .

Now to prove the regret bound (claim ii)) we follow the lines of the classical analysis of OSD (see e.g. the proof of [8, Theorem 3.1] or that of [21, Theorem 2.13]) when run against the sequence of losses ( ∑ t ∈T k ℓ t ) k ∈ N instead of ( ℓ t ) t ∈ N . For convenience, we will write ˜ g k = ∑ t ∈T k g t for all k ∈ N .

Let y ∈ Y , K ∈ N and k ∈ [ K ] , we start by bounding ∑ t ∈T k ℓ t ( y t ) -ℓ t ( y ) . By definition of the subgradient g t ∈ ∂ℓ t ( y t ) we have:

We rewrite this bound as follows:

By using the property of non-expansiveness of the Euclidean projection we have:

Combining the last two results leads to:

Finally, we combine the last inequality with inequality (14) and sum these over k = 1 , . . . , K .

Claim ii) is thereby proved by taking the supremum over y ∈ Y .

Proposition 19 gives us guidelines to design optimal update cycles and learning rates. First, this proposition states that it suffices to verify the feasibility constraint at the update periods to ensure that the whole sequence of order-up-to levels produced by COSD is feasible. On the other hand, to guarantee that this algorithm has a sublinear regret we will need to ensure that the cycles T k are not too long, i.e. that update periods are frequent enough, and then consider adequate learning rates.

Since OSD is an instance of COSD, we can recover classical regret bounds of OSD (see e.g. [8, Theorem 3.1] or [21, Theorem 2.13]) and in particular O ( √ T ) regret bounds, as a corollary of Proposition 19.

Corollary 20. Let assumptions 2.i) and 2.ii) be satisfied. Given positive non-increasing learning rates ( η t ) t ∈ N , i.e. 0 < η t +1 ≤ η t for all t ∈ N , the regret of OSD (see Algorithm 1) satisfies for all T ∈ N ,

In particular, if η t = γD/ ( G √ t ) with γ > 0 , we have

Proof. The regret bound follows from claim ii) of Proposition 19. Indeed, by taking t k = k for all k ∈ N , COSD coincide with OSD, thus, for K = ¯ t K = T we obtain the desired regret bound.

A.3 Controlling the cycles' length

In this subsection we analyze COSD with eventually unbounded cycles as it is the case for the CUP or the MaxCOSD update strategy. We start by introducing an assumption on the cycles called geometric cycles which yields expected regret bounds and high probability regret bounds.

Assumption 21 (Geometric cycles) . Let µ ∈ (0 , 1] . An update strategy ( t k ) k ∈ N has µ -geometric cycles if there exists C µ ≥ 1 such that for any m ∈ N and k ∈ N we have:

The name geometric cycles is motivated by the fact that if ξ is a geometric random variable with parameter µ , i.e. ξ ∈ N and P [ ξ > m ] = (1 -µ ) m for all m ∈ N , then Assumption 21 rewrites:

P [ t k +1 -t k > m ] ≤ C µ P [ ξ > m ] for all k, m ∈ N . This property resembles sub-exponential concentration [29, Section 2.7]. This notion was inspired from a proof in [32] where the authors observe, under their assumptions, that CUP's cycles length are geometrically distributed with a parameter lower bounded by µ . Assumption 21 generalizes this property.

The following proposition summarizes some important properties of geometric cycles.

Proposition 22. Consider an update strategy ( t k ) k ∈ N with µ -geometric cycles (see Assumption 21), then,

For all K ∈ N and any confidence level δ ∈ (0 , 1) we have with probability at least 1 -δ ,

Proof. First, let us observe that if µ = 1 in Assumption 21 then, t k = k for all k ∈ N and one can easily check that all the claims are indeed verified. In the following we assume µ ∈ (0 , 1) .

Let ξ be a geometric random variable of parameter µ , that is, ξ ∈ N and P [ ξ > m ] = (1 -µ ) m for all m ∈ N . Then, it is well-known that E [ ξ ] = 1 /µ and E [ ξ 2 ] = (2 -µ ) /µ 2 .

We now prove the first claim by means of direct computations. For all k ∈ N , we have:

Also, using the fact that for any integer a and real number b , we have, a > b if and only if a > ⌊ b ⌋ , we can upper bound E [ ( t k +1 -t k ) 2 ] as follows:

Finally, let us prove the second claim. Let K ∈ N and ε > 0 , it is classical to see that:

We use Assumption 21, which yields:

Now for δ ∈ (0 , 1) we plug ε = ( 1 + log( δ/ ( KC µ )) log(1 -µ ) ) √ K > 0 in the last result, to obtain:

with probability at least 1 -δ . We conclude by simply observing that -1 / log(1 -µ ) ≤ 1 /µ .

Using these tools, we are now ready to provide strong regret bounds for COSD under the assumption of geometric cycles. We are going from now on to focus on adaptive learning rates (see Eq. 7) which will allow us to unlock high probability regret bounds.

Corollary 23. Consider an inventory problem satisfying Assumption 2 and COSD (see Algorithm 3) with adaptive learning rates (see Eq. (7) ) and an update strategy with µ -geometric cycles (see Assumption 21), then, the following regret bounds hold for all T ∈ N ,

and for any confidence level δ ∈ (0 , 1) we have with probability at least 1 -δ ,

Finally, COSD is feasible if and only if for all k ∈ { 2 , 3 , . . . } , y t k ⪰ x t k .

Proof. Let T ∈ N and K = min { k ≥ 1 , t k ≤ T } . We start by bounding the regret R T in terms of the regret at the end of K -1 th cycle R ¯ t K -1 = R t K -1 and a remainder term as follows:

where we used g t ∈ ∂ℓ t ( y t ) . Applying Cauchy-Schwartz inequality, Assumption 2 and T ≤ t K +1 -1 leads us to:

Let us now bound R ¯ t K -1 . As a consequence of claim ii) of Proposition 19 and after substituting η t by its value and applying Lemma 25 provided in Appendix B to f ( x ) = 1 / (2 √ x ) , a 0 = 0 and a k = ∥ ∥ ∑ t ∈T k g t ∥ ∥ 2 2 for k ∈ [ K -1] we obtain:

Combining this with inequality (16) leads us to:

To obtain the expected regret bound, we start by noticing that K ≤ T then taking the expectation in this last inequality, applying Jensen's inequality, then, Proposition 22 that bounds E [ ( t k +1 -t k ) 2 ] ≤ 2 C µ /µ 2 we end up with the desired bound. Finally, to obtain the high probability regret bound, we notice again that K ≤ T and then apply the high probability bound of Proposition 22.

A.4 Proof of Theorem 12

In the following lemma we claim that under the assumptions of Theorem 12, MaxCOSD with the appropriate learning rates has geometric cycles (see Assumption 21).

Lemma 24. Consider an inventory problem and let assumptions 2, 10 and 11 hold. Then, MaxCOSD with learning rates defined in Eq. (7) and γ ∈ (0 , ρ/D ] has µ -geometric cycles with C µ = 1 .

Proof. First, notice that if min i ∈ [ n ] d t,i ≥ ρ then x t +1 ⪯ ˆ y t +1 , i.e. t + 1 is an update period for MaxCOSD. This is a consequence of Lemma 15, which applies since we have:

Now let k ∈ N and m ∈ N . Using this initial observation we have:

The last step of this proof is showing that in fact,

Weprove this by a simple induction over m ≥ 1 . Noticing that { t k = s } ∈ σ ( g 1 , x 2 , . . . , g s -1 , x s ) ⊂ σ ( ℓ 1 , d 1 , . . . , ℓ s -1 , d s -1 ) , where the last inclusion comes from Assumption 11, and using the basic properties of conditional expectations we derive the inequality for m = 1 :

where the last inequality comes from Assumption 10. Assume now the relation (17) holds for m , let us prove in a similar way that it holds for m +1 .

Summing the relations (17) over s ≥ 1 leads to the final bound: P [ t k +1 -t k > m ] ≤ (1 -µ ) m , which is our claim.

We are now ready to prove Theorem 12.

Proof of Theorem 12. By definition, MaxCOSD is always feasible (independently of the learning rates chosen). Lemma 24 ensures that when γ ∈ (0 , ρ/D ] , MaxCOSD with adaptive learning rates as defined in Eq. (7) has µ -geometric cycles with C µ = 1 . Thus, Corollary 23 applies and leads to the regret bounds we claimed.

B Other lemmas

Lemma 25 (Lemma 4.13 of [21]) . Let a 0 , a 1 , . . . , a K be non-negative numbers and f : R + → R + a measurable non-increasing function, we have:

Proof. Define s k = ∑ k m =0 a m . The following holds for all k ∈ [ K ] ,

Summing over k = 1 , . . . , K leads to the desired bound.

Lemma 26. Consider an inventory problem with non-negative feasible set Y ⊂ R n + . Then, any non-decreasing sequence of order-up-to levels ( y t ) t ∈ N ⊂ Y (i.e. such that y t ⪯ y t +1 for all t ∈ N ) is feasible. In particular, constant strategies are feasible too.

Proof. Consider a non-decreasing sequence of order-up-to levels ( y t ) t ∈ N ⊂ Y ⊂ R n + . First of all, y 1 is feasible since we impose x 1 = 0 and y 1 ∈ R n + . Then, for every t ∈ N , we can write:

The first inequality comes from the assumption that ( y t ) t ∈ N is non-decreasing, the second one from the monotonicity of the positive part and the fact that y t ∈ R n + , finally, the last inequality is the inventory dynamical constraint. Thus, the whole strategy ( y t ) t ∈ N is feasible.

Lemma 27. Consider an inventory problem that satisfies Assumption 2, then, the regret of any algorithm is bounded as follows:

Proof. Let y ∈ Y . For all t ∈ N , we have:

where we used the definition of the subgradient g t ∈ ∂ℓ t ( y t ) , then, Cauchy-Schwartz inequality and finally Assumption 2. Summing these inequalities over t = 1 , . . . , T and taking the supremum over y ∈ Y leads to the desired bound.

Lemma 28. Consider the newsvendor cost function c defined in (5) . Let d ∈ R n . A vector g ∈ R n is a subgradient of the function of c ( · , d ) at y ∈ R n if and only if for all i ∈ [ n ] we have:

In particular, denoting s = min { y, d } , the vector ( h i 1 { y i >s i } -p i 1 { y i = s i } ) i ∈ [ n ] is a subgradient of c ( · , d ) at y .

C Postponed proofs

C.1 Proof of Proposition 13

Proof of Proposition 13. Formally, in the lost sales single-product newsvendor setting with observable demand over Y = [0 , D ] , a feasible deterministic algorithm is defined by a sequence of functions ( Y t ) t ∈ N of the form Y t : R t -1 + → [0 , D ] satisfying Y t +1 ( d 1 , . . . , d t ) ≥ [ Y t ( d 1 , . . . , d t -1 ) -d t ] + for all d 1 , . . . , d t ∈ R + .

Let ˜ d ∈ (0 , D ] , and consider a constant demand sequence defined by ˜ d t := ˜ d at every period t ∈ N . Consider now (˜ y t ) t ∈ N the sequence of order-up-to levels generated by this algorithm when facing this constant demand, that is, ˜ y t := Y t ( ˜ d, . . . , ˜ d ) . We will now distinguish two cases.

Consider the case where ˜ y t = 0 for all t ∈ N . Taking y = ˜ d in the regret definition (3) yields:

ii) Now, consider the case where there exists T 0 ≥ 1 such that ˜ y 1 = · · · = ˜ y T 0 -1 = 0 and ˜ y T 0 > 0 . Consider a new demand sequence ( d t ) t ∈ N defined as follows: d 1 = · · · = d T 0 -1 = ˜ d and d t = 0 for t ≥ T 0 . Denote by ( y t ) t ∈ N the sequence of order-up-to levels generated by the algorithm against the demand sequence ( d t ) t ∈ N , that is, y t = Y t ( d 1 , . . . , d t -1 ) .

Since the algorithm is deterministic, we also have: y t = 0 for t ≤ T 0 -1 . Indeed, we have, y t = Y t ( d 1 , . . . , d t -1 ) = Y t ( ˜ d, . . . , ˜ d ) = ˜ y t = 0 . On the other hand, we have for all t ≥ T 0 +1 , y t ≥ [ y t -1 -d t -1 ] + = y t -1 , thus, y t ≥ y T 0 = ˜ y T 0 > 0 for all t ≥ T 0 .

Taking y = 0 in the regret definition (3) we obtain for T ≥ T 0 ,

C.2 Proof of Proposition 14

Proof of Proposition 14. Take ( d t ) t ∈ N ⊂ R ++ such that ∑ ∞ t =1 d t < y 1 and define C = y 1 -∑ ∞ t =1 d t > 0 . Notice that due to the feasibility constraint (1) and the lost sales dynamic we have for all t ∈ N , y t +1 ≥ x t +1 = [ y t -d t ] + ≥ y t -d t . Thus, we have for every t ∈ N , y t = y 1 + ∑ t -1 s =1 ( y s +1 -y s ) ≥ y 1 -∑ t -1 s =1 d s ≥ C. Now take the losses ℓ t ( y ) = y , then, R T = ∑ T t =1 y t ≥ CT for all T ∈ N .

C.3 Proof of Lemma 15

Proof of Lemma 15. Since [ y -d ] + = max { y -d, 0 } and y ′ ⪰ 0 , it is enough to show that y ′ ⪰ y -d . The latter holds since, for any i ∈ [ n ] , we have y i -y ′ i ≤ ∥ y ′ -y ∥ 2 ≤ min i ∈ [ n ] d i ≤ d i .

C.4 Proof of Theorem 17

Proof of Theorem 17. The regret bound follows from the classical analysis of OSD, see our Corollary 20 or the proof of [8, Theorem 3.1]. Thus, we only need to show the feasibility. For all t ∈ N , we have,

the first inequality is provided by the property of non-expansiveness of the Euclidean projection, the second inequality comes from γ ≤ ρ/D and ∥ g t ∥ 2 ≤ G , and the last inequality from √ t ≥ 1 . Combining this result with Assumption 16, we obtain ∥ y t +1 -y t ∥ 2 ≤ min i ∈ [ n ] d t,i . According to Lemma 15 and the inventory dynamical constraint (2) this guarantees feasibility.

D Discussion

D.1 On the relation between Online Inventory Optimization and Online Convex Optimization

In the usual Online Convex Optimization (OCO) framework introduced by [36] a decision-maker and an environment interact as follows: at every time period t ∈ N , first, the decision-maker chooses a decision y t ∈ Y and the environment chooses a loss function ℓ t , then, the decision-maker receive some feedback which usually consist of either the loss function itself ℓ t (full-information setting), the loss incurred ℓ t ( y t ) (bandit setting) or a subgradient g t ∈ ∂ℓ t ( y t ) (first-order feedback setting). The goal of the decision-maker is to minimize its cumulative loss incurred ∑ T t =1 ℓ t ( y t ) .

OIO extend the OCO framework by adding the feasibility constraint (1). A naive solution to accommodate such constraints into the OCO framework is by adding to the losses the convex indicator of the feasibility constraint, that is, by considering the losses ˜ ℓ t ( y ) = ℓ t ( y ) + χ t ( y ) , where χ t ( y ) takes the value 0 if y ⪰ x t holds and + ∞ otherwise. However, this is not satisfactory since by

doing so we also alter the regret (3) by imposing on the competitor y ∈ Y the feasibility constraints associated to the algorithm.

Many extensions of the OCO framework have been developed over the years. Of particular interest, those which include constraints of different forms like:

OCO with long-term constraints [16, 11], where m convex constraints of the form f i ( · ) ≤ 0 for i ∈ [ m ] should be satisfied in the long run, that is, the goal is to minimize the cumulative loss while keeping low constraint violation ∑ T t =1 f i ( y t ) for each i ∈ [ m ] .

OCO with long-term and time-varying constraints [20, 14] which compared to the previous extension, considers time-varying convex constraints of the form f t,i ( · ) ≤ 0 where f t,i is revealed at the end of time period t . Even as long-term constraints, this learning task is known to be unsolvable in general (see e.g. [28, Proposition 2.1] or [18, Proposition 4] for more precise statement), thus, restricted notions of regret have been considered in this context.

OCO with ramp constraints [2], where at each time period t ∈ { 2 , 3 , . . . } the decision-maker should choose y t ∈ Y such that | y t,i -y t -1 ,i | ≤ κ i for all i ∈ [ n ] .

We argue that OIO problems are different from these extensions. Indeed, our feasibility constraints (1) are neither long-term constraint since we do not allow for violations, nor ramp constraints since the bounds are time-varying. Also, our task is further challenging since we aim at bounding the regret (3) based on a competitor y ∈ Y which does not suffer from the feasibility constraint (1).

D.2 On the notion of pseudo-regret

There exists an alternative notion of regret we call the pseudo-regret ¯ R T which is in fact more common in the literature of online inventory problems [9, 3, 25, 32]. It is defined as the difference between the expected cumulative loss of the algorithm and that of the best fixed constant strategy, that is, formally

The difference between the expected regret E [ R T ] and the pseudo-regret ¯ R T is in the competitor y . In the former, the competitor is random and depends on the realization of the losses, whereas, in the latter the competitor is fixed and depends only on the distribution of the losses. Notice that we always have ¯ R T ≤ E [ R T ] , thus, an upper bound obtained on the expected regret applies directly to the pseudo-regret.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors consider the classic online inventory control problem. While previous works mainly consider the case where the demand is stochastically drawn from a stationary distribution, this paper considers the case where the demand can be adversarial. In addition, the authors consider a more general framework Online Inventory Optimization (OIO), which includes many different setups in the inventory literature, i.e. censored demand / backlogged demand / perishable / non-perishable. Then, the authors proposed the algorithm MaxCOSD achieving $O(\\sqrt{T})$ regret under the assumption on the lower boundedness of the demand (with constant probability). The algorithm shares similar ideas to the cyclical updates proposed in CUP [Zhang et al., 2018] but with a different definition on the cycle and an adaptive choice of learning rate. The algorithm keeps using the same base-stock level to collect enough gradient information until the time where the outcome from the (cumulative) gradient descent step gives a feasible strategy. The authors also discuss their demand assumptions and show that without lower bounded demand assumption, no deterministic algorithm can achieve sublinear regret. Experiments are also done to show the effectiveness of the algorithm MaxCOSD.\n\nSTRENGTHS:\n- The problem considered in this paper is interesting and has wide applications in real life.\n- The proposed algorithm is clean and easy to follow. The proposed framework OIO is general and includes many different setups that are considered in the inventory literature.\n- Experiments on both synthetic data and real-world data are also done for their proposed algorithm.\n\nWEAKNESSES:\n- One major concern is the novelty of the proposed algorithm. From the technical perspective, I think the analysis generally follows the classic analysis of online gradient descent with a self-confidence tuning of the learning rate in order to handle the issue raised by the scale of the gradient that may cumulate over rounds. Based on this, I think the $\\sqrt{T}$ regret is not very surprising.\n- Several claims made in this paper may need more explanations or clarifications.\n    - On Assumption 10, it is unclear how to transform the assumption that $\\mathbb{E}[d]\\geq \\rho$ assumed in AIM to this assumption with the same theoretical guarantees. Specifically, with the current Assumption 10, when $\\mu$ and $\\rho$ is $\\Theta(T^{-\\alpha})$ and $\\Theta(T^{-\\beta})$, the regret bound becomes $T^{\\frac{1}{2}+\\alpha+\\beta}$. It would be better if the authors can compare the guarantees of both algorithms under the same assumptions.\n    - The lower bound analysis is problematic or not rigorous. For proposition 13, in line 601, the authors argue that \n$(T-T_0+1)hy_{T_0}=\\Omega(T)$ , which does not hold when ${y}_{T_0}=o(T)$. Similarly, in Proposition 14, $y_1$ can also be $o(T)$ making the regret sublinear. A refined analysis on both propositions should be included.\n- The experiment results do not show a better performance of MaxCOSD compared to other algorithms in Settings 1-4, even though some algorithms are not designed to handle the adversarial/non-stationary demand. This weakens the motivation of providing an algorithm that performs better in the adversarial environment. In addition, for Settings 1-3, showing the error bar or the std of the algorithm would be better since the algorithm actually guarantees high-probability regret bound.\n\nQUESTIONS:\n-  In line 244, I do not understand why Assumption 10 with $\\rho=D$ matches the assumption 1 made by CUP in [Zhang et al., 2018]. In CUP, they are assuming the knowledge of the upper bound on the optimal inventory level.\n- On the optimality on the parameters $\\rho$ and $\\mu$ in Theorem 12: I wonder whether there is lower bound on $\\mu$ and $\\rho$ for this problem? This may show further tightness of the obtained bound.\n\nLIMITATIONS:\nSee Weakness and Questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a general online decision making problem for inventory management. The problem setting extends the generic online convex optimization by introducing structures such as states and demands. Compared to settings studied by related works in inventory management, the setting of this paper relaxes several restrictive assumptions on the loss functions, dynamics and demands. Then, the problem is solved by a new batch-update online gradient descent algorithm with an optimal regret upper bound, whose analysis relies on a geometric cycle property induced by the problem structure. The strengths and limitations of this algorithm are further demonstrated through experiments.\n\nSTRENGTHS:\nOverall, this is a strong submission. \n\n- It is very well-written, with a smooth and clear coverage of the motivation, how the problem connects to classical online learning models, and the novelty of the proposed algorithm. \n\n- Related works on the inventory problem are discussed in detail. This is especially valuable since the readers are most likely from the machine learning community, who are not familiar with the necessary background. \n\n- Although I didn't check everything in the proof, the analysis is solid to my knowledge. Intuitions of the algorithm including the nondegeneracy assumption are clearly presented and natural. \n\n- Experiments are included, with both positive and negative results. The experimental methodology makes sense.\n\nWEAKNESSES:\nI don't have much to complain in general. One aspect I'm not sure is the novelty. The backbone of the proposed algorithm is a batch-update version of online gradient descent, which is standard from the perspective of online learning. Hence, the novelty of the paper is mainly on the combination of OGD with inventory structures, which requires sufficient domain knowledge to appreciate. Unfortunately due to my unfamiliarity with inventory problems, I cannot make a very informed evaluation. This is more of my limitation rather than the limitation of this paper. Has the geometric cycle appeared in existing analyses?\n\nQUESTIONS:\n- It is assumed that the adversary is oblivious, as the loss function and the demand sequence are determined at the beginning. Is there any particular reason for making this assumption? In online learning, people mainly need oblivious adversaries when the algorithm is randomized, but the algorithm in this paper is deterministic. \n\n- The paper leaves discrete feasible sets to future works (L135). I guess this might be handled by an expert algorithm, following the similar batch-update idea. Probably out of the scope of this paper, but it seems a reasonable addition. \n\n- The paper discusses the many ways that the setting of this paper generalizes existing works. Does the regret bound also recover existing bounds for more restrictive settings? \n\n- Appendix D compares the algorithm to a more direct approach based on OCO. The limitation of the latter is that the competitor is subject to the feasibility constraint. This leaves me wondering, if the competitor in this paper is not feasible, then technically we cannot implement it. Then, the meaning of the competitor term $\\sum l_t(y)$ becomes more obscure.\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-23"}

comments: SUMMARY:
The paper introduces the online inventory optimisation problem. In the problem, we have $n$ products to stock and every day choose the levels to stock up the products to. We then suffer loss according to a convex loss function, which may vary from day to day (e.g., it may depend on the demand). We also get to know a subgradient of the loss. After the demand has wiped out some of the stock, we need to choose new levels to top up the remainders too etc.

We want an online algorithm with a guarantee on the regret w.r.t. all possible constant levels.

The paper proposes an algorithm, which is an extension of the online subgradient descent (OSD). 

OSD would have been applicable to the problem directly, if not for the presence of remainders (we cannot top up to a level below the current remainder). The version of the protocol without remainders is called stateless in the paper, so the proposed algorithm is a generalisation of the OSD to the "stateful" setup.

The paper proceeds to obtain a $O(\sqrt{T})$ regret bound for the algorithm.

STRENGTHS:
I think the paper formulates an important problem and obtain a useful result.

WEAKNESSES:
The algorithm and the pound in the paper are an extension of the online subgradient descent (OSD). One can argue that the result is of the incremental nature.

QUESTIONS:
A constant strategy will only work if the constant is above the remainders all the time; otherwise it makes no sense. Am I correct that the regret is w.r.t. the constant strategies that work in this sense?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers a general family of online inventory management problems under convex losses and stateful inventory dynamics. They propose a general algorithm based on an adaptation of ideas from online convex optimization. The method is essentially AdaGrad enhanced with an additional delayed update mechanism that ensures that the inventory constraints are always satisfied. The main result is a sublinear regret guarantee against the best fixed base-stock policy chosen in hindsight, given full knowledge of the realization of the demand sequence.\n\nSTRENGTHS:\nThe paper is very well-written when it comes to both its informal and technical content. In particular, the authors provide an excellent and succinct introduction into the relevant part of the management / operations literature, and discuss their results in this context extensively. The algorithm itself is natural and effective.\n\nWEAKNESSES:\nWhile I generally like the paper, there are a couple of things that I would like to have some more clarity on before I can fully support its acceptance:\n- Regret is defined against the \"best feasible constant strategy\", as defined in Eq.(3). I worry that this comparator strategy is problematic in the stateful setting. First off, some fixed constant strategies may be infeasible when some of the carryover inventory levels x_t are large. Thus, the best feasible constant strategy may be a very poor one, ordering very high levels to make sure that the constraint is never violated. Second, I am concerned that evaluating a fixed policy on the loss sequence generated in response to the online agent's actions may be meaningless in the first place, and it might be more sensible to consider \"policy regret\" by taking into account the fact that the environment would respond differently to the comparator policy than to the agent. Treating this case, however, seems much harder than what is being handled in the present paper. Can the authors let me know if I'm missing something here?\n- The assumption that the order levels are continuous seems like a very important one. The authors themselves admit that this is a limitation of their method. However, I tend to believe that it is a rather major limitation, as just about every practically relevant problem instance in this space features discrete actions. In lack of continuity, one has to deal with the challenge of partial information due to lost sales, which makes it impossible to estimate the gradients even after convexifying the action space via randomization (as noted by Huh and Rusmevichientong [9]). It would be useful to clarify this to the reader so they don't end up believing that this assumption is minor.\n- I find it a bit disappointing that the regret bounds depend so heavily on the demand parameters mu and rho. I appreciate the negative results showing that certain degenerate demand sequences can lead to linear regret, but it's not clear to me if this justifies the poor linear dependence on 1/mu and 1/rho demonstrated by the bound. How do we know that this dependence cannot be improved to log(1/mu) and log(1/rho) or something even better? I really appreciate the experiments that plot performance as a function of the learning rate gamma, but I would also like to understand which values of gamma actually satisfy the constraint required by the theorem --- I suspect only the very very small ones, for which the algorithm works rather poorly. In this sense, it may be unfair to criticize DDM for not having performance guarantees since MaxCOSD doesn't have any either for the majority of the studied stepsizes. This is of course always a problem for theoretically motivated stepsizes, but it would be important to mention it at least once in passing. (A final note on the experiments: my impression was that AIM was also a very similar policy to what is being proposed here, but see no discussion on this.)\n\nOverall, I think this could be a good addition to the program, but I cannot confidently recommend acceptance until the authors respond to the above concerns.\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}