# Solving a Joint Pricing and Inventory Control Problem for Perishables via Deep Reinforcement Learning

Authors: Rui Wang; Xianghua Gan; Qing Li; Xiao Yan
Journal: Complexity
 (2021)

---

## Abstract

We study a joint pricing and inventory control problem for perishables with positive lead time in a finite horizon periodic-review system. Unlike most studies considering a continuous density function of demand, in our paper the customer demand depends on the price of current period and arrives according to a homogeneous Poisson process. We consider both backlogging and lost-sales cases, and our goal is to find a simultaneously ordering and pricing policy to maximize the expected discounted profit over the planning horizon. When there is no fixed ordering cost involved, we design a deep reinforcement learning algorithm to obtain a near-optimal ordering policy and show that there are some monotonicity properties in the learned policy. We also show that our deep reinforcement learning algorithm achieves a better performance than tabular-based Q-learning algorithms. When a fixed ordering cost is involved, we show that our deep reinforcement learning algorithm is effective and efficient, under which the problem of “curse of dimension” is circumvented.


---

## Body

## 1. Introduction

The inventory control of perishables has received increasing attention from the business community and academia. According to a report released by the Food Market Institute (2012) in the United States, as of 2005, the total sales of perishables accounted for more than half of sales in supermarkets and grocery stores in the US, and this proportion is still increasing. Meanwhile, losses due to the deterioration of perishables also account for a large proportion of the total retail cost. Besides, pricing is also an important and effective lever for the retail industry to manage the profitability of perishables. As shown in Karaesmen et al. [1] and Chen et al. [2], a firm’s profit increases significantly by dynamic adjustment of prices of perishables according to the availability of the inventory and the remaining lives of perishables.In this research, we study a joint pricing and inventory control problem for perishables in a finite planning horizon. Demand in each period depends on the current price and satisfies a Possion distribution. The problem of inventory control for perishables is usually more difficult than the one for nonperishables, in which the inventory state can be represented by a single variable. The state of perishables has to be recorded by a vector to account for items with different lifetimes, which makes the analytical studies much more difficult. As a fixed ordering cost can make the problem even more difficult in a dynamic setting, few studies consider it due to the tractability in analysis.One main contribution in this research is that we consider a fixed ordering cost in our model. We study both the backlogging case and the lost-sales case and allow for positive lead time. Our goal is to find a near-optimal ordering and pricing policy to maximize the expected profit in the planning horizon. This problem is hard to analyze by traditional dynamic programming approach in the inventory control literature. Therefore, we use a reinforcement learning approach to solve the problem.In the literature, there have been a few papers that study inventory control problems with reinforcement learning, such as Charharsooghi et al. [3], Dogan et al. [4], and Kara et al. [5]. Unlike these papers which use Q-learning, we take a deep reinforcement learning approach and show that it outperforms Q-learning models that do not use neural networks. The outperformance of deep reinforcement learning has also been shown by Ke et al. [6] and Shihab et al. [7] for complex problems.In this paper, we set up deep reinforcement learning models to study the joint pricing and inventory control problem of perishables. We adopt a FIFO (first-in-first-out) policy in this study. When there is no fixed ordering cost involved, we show that the fixed pricing strategy is dominated by the dynamic pricing strategy, under which the price can be adjusted according to the availability of inventory and the lives of remaining items. We set up a benchmark based on realized demand for this no fixed ordering cost case and show that our designed deep reinforcement learning methods achieve a better performance than tabular-based Q-learning. We also find some monotonicity properties in our learned policies; our learned order quantity is nonincreasing in inventory position or on-hand inventory and price decision is most sensitive to the oldest on-hand inventory. Moreover, in order to show the expansibility of the proposed algorithm, we extend the distribution of the demand and take anadditive form in Chen et al. [2] where the customer demand depends on the price of current period plus an additive random term; finally, we obtain a near-optimal performance by our proposed deep reinforcement learning models. When the fixed ordering cost is taken into account in the joint pricing and inventory control system, we set up a performance upper bound based on the realized demand in each period in order to assess the performance. Through our proposed methods, we find convergent policies and critical values under which orders should be placed.


## 2. Literature Review

We review two streams of literature which are closely related to our research: traditional inventory control management for perishables and inventory control management with reinforcement learning.


### 2.1. Traditional Inventory Control Management for Perishables

There is a considerable literature devoted to dynamic inventory control for nonperishable products; see, for example, Presman and Sethi [8], Caliskan-Demirag et al. [9], Alp et al. [10], Almaktoom et al. [11], Azghandi et al. [12], Li et al. [13], and Gan et al. [14]. The dynamic inventory control for perishable products has not been widely studied in the literature. This is not to say that the literature does not realize the importance of the study of perishable studies.Indeed, there are a number of papers devoted to the study of inventory decisions for perishable products. Nahmias and Pierskalla [15] studied a dynamic inventory control with a fixed lifetime, zero lead time, and uncertainty demand for perishable products. Nahmias [16], Fries [17], and Nahmias [18] studied the same problem, with multiple periods of lifetime and zero lead time, and their research studies are all to satisfy the same assumption that only products that exceed the life cycle will be abandoned, which is known as the first-in-first-out policy (FIFO), and this policy is widely used in the research of perishable retailing. And they proved that the optimal order quantity under different inventory ages is decreasing. Prastacos [19] reviewed some important theories and practices in blood inventory management and proposed that this kind of application can be extended to other perishable product inventory control problem. Ferguson and Koenigsberg [20] considered a two-period joint pricing and inventory control problem with a random lifetime, emphasizing and discussing the impact of competition between new inventory and surplus inventory over the previous period on inventory and pricing decisions for the first time interval. Chen et al. [21] used Pontryagin’s maximum principle method to investigate the optimal policies for the pricing and replenishment of fashion apparel with short product lifecycles. Heuristic algorithms are also increasingly being used to address the problem of dynamic pricing and inventory control for perishables. Li et al. [22] proposed a base-stock/list-price heuristic policy to solve the problem of dynamic pricing and inventory control for a perishable product, assuming that the demand is a function of price and zero lead time. Li and Lu [23] studied a joint optimization of the price and order quantity of a perishable product and proposed a Minimax Regret algorithm. Li et al. [24] discussed a new dynamic pricing and inventory control scenario for perishables. New and old products cannot be sold at the same time. The seller can decide whether to discard the remaining inventory in the previous period, even though the lifetime may not be over. And they proposed a fractional programming heuristic algorithm to obtain a stable structural policy.Chen et al. [2] is closely related to our research. They considered positive lead time and used the concept of L-convexity/concavity to analyze the problem and proposed a heuristic algorithm to solve the problem. However, the traditional approach used in their research is not able to solve the problem with a fixed ordering cost. By using neural networks with hidden layers to approximate state-action values, our deep reinforcement learning approach exploits the advantages of deep learning [25] and reinforcement learning and is shown to be effective and efficient to find the solution.


### 2.2. Inventory Control with Reinforcement Learning

In the literature, there have been few papers that study inventory control problems with reinforcement learning.Giannoccaro and Pontrandolfo [26] studied the coordination of inventory policies adopted by different supply chain factors which are a major issue in supply chain inventory management, and they used a reinforcement learning approach to manage inventory decisions at all stages of the supply chain in an integrated manner and aimed at optimizing the performance of the whole supply chain. Chaharsooghi et al. [3] proposed an inventory control system based on reinforcement learning methods, which included uncertain delivery times and uncertain customer requirements to determine the ordering policy for each order point in the supply chain. Chaharsooghi et al. [3] used Q-learning to solve supply chain ordering management and applied to the beer game. Jiang and Sheng [27] proposed a case-based reinforncement learning algorithm (CRL) for dynamic inventory control in a multiagent supply-chain system. They studied a multiagent simulation of a simplified two-echelon supply chain and showed the effectiveness of the method they proposed. Sui et al. [28] considered a Vendor-Managed Inventory (VMI) system where the supplier makes decisions of inventory decisions of inventory management for the retailer, and the retailer is not responsible for placing orders. Through a methodology based on reinforcement learning and numerical study, they show their approach can outperform the newsvendor. Zarandi et al. [29] presented a flexible fuzzy reinforcement learning algorithm where the value function is approximated by a fuzzy rule-based system and considered the problem of a fuzzy agent (supplier), that is, how to determine the amount of orders for each retailers based on their utility for supplier when its supply capacity is limited. Finally, the effectiveness of their proposed algorithm is proved by a simulation. Dogan et al. [4] used the Q-learning method to study an ordering and pricing policy in a multiretailer environment. Rana and Oliveira [30] use reinforcement learning methods to develop dynamic pricing strategies for interdependent perishable products or service. Kara and Dogan [5] used Q-learning and Sarsa reinforcement learning algorithms to study a dynamic inventory control issues for perishable products, with positive lead time and fixed lifetime. Our research further uses deep reinforcement learning to study this dynamic inventory control of perishable products.The aforementioned studies investigate the inventory problem for nonperishable and perishable products and use the nondeep reinforcement learning methods. Compared to their problems, our problem focuses on the inventory control of perishables, which makes the problem much more difficult. We use neural networks to avoid the curse of dimensionality and show that our deep reinforcement learning model outperforms the traditional reinforcement learning models without using neural networks.


## 2.1. Traditional Inventory Control Management for Perishables

There is a considerable literature devoted to dynamic inventory control for nonperishable products; see, for example, Presman and Sethi [8], Caliskan-Demirag et al. [9], Alp et al. [10], Almaktoom et al. [11], Azghandi et al. [12], Li et al. [13], and Gan et al. [14]. The dynamic inventory control for perishable products has not been widely studied in the literature. This is not to say that the literature does not realize the importance of the study of perishable studies.Indeed, there are a number of papers devoted to the study of inventory decisions for perishable products. Nahmias and Pierskalla [15] studied a dynamic inventory control with a fixed lifetime, zero lead time, and uncertainty demand for perishable products. Nahmias [16], Fries [17], and Nahmias [18] studied the same problem, with multiple periods of lifetime and zero lead time, and their research studies are all to satisfy the same assumption that only products that exceed the life cycle will be abandoned, which is known as the first-in-first-out policy (FIFO), and this policy is widely used in the research of perishable retailing. And they proved that the optimal order quantity under different inventory ages is decreasing. Prastacos [19] reviewed some important theories and practices in blood inventory management and proposed that this kind of application can be extended to other perishable product inventory control problem. Ferguson and Koenigsberg [20] considered a two-period joint pricing and inventory control problem with a random lifetime, emphasizing and discussing the impact of competition between new inventory and surplus inventory over the previous period on inventory and pricing decisions for the first time interval. Chen et al. [21] used Pontryagin’s maximum principle method to investigate the optimal policies for the pricing and replenishment of fashion apparel with short product lifecycles. Heuristic algorithms are also increasingly being used to address the problem of dynamic pricing and inventory control for perishables. Li et al. [22] proposed a base-stock/list-price heuristic policy to solve the problem of dynamic pricing and inventory control for a perishable product, assuming that the demand is a function of price and zero lead time. Li and Lu [23] studied a joint optimization of the price and order quantity of a perishable product and proposed a Minimax Regret algorithm. Li et al. [24] discussed a new dynamic pricing and inventory control scenario for perishables. New and old products cannot be sold at the same time. The seller can decide whether to discard the remaining inventory in the previous period, even though the lifetime may not be over. And they proposed a fractional programming heuristic algorithm to obtain a stable structural policy.Chen et al. [2] is closely related to our research. They considered positive lead time and used the concept of L-convexity/concavity to analyze the problem and proposed a heuristic algorithm to solve the problem. However, the traditional approach used in their research is not able to solve the problem with a fixed ordering cost. By using neural networks with hidden layers to approximate state-action values, our deep reinforcement learning approach exploits the advantages of deep learning [25] and reinforcement learning and is shown to be effective and efficient to find the solution.


## 2.2. Inventory Control with Reinforcement Learning

In the literature, there have been few papers that study inventory control problems with reinforcement learning.Giannoccaro and Pontrandolfo [26] studied the coordination of inventory policies adopted by different supply chain factors which are a major issue in supply chain inventory management, and they used a reinforcement learning approach to manage inventory decisions at all stages of the supply chain in an integrated manner and aimed at optimizing the performance of the whole supply chain. Chaharsooghi et al. [3] proposed an inventory control system based on reinforcement learning methods, which included uncertain delivery times and uncertain customer requirements to determine the ordering policy for each order point in the supply chain. Chaharsooghi et al. [3] used Q-learning to solve supply chain ordering management and applied to the beer game. Jiang and Sheng [27] proposed a case-based reinforncement learning algorithm (CRL) for dynamic inventory control in a multiagent supply-chain system. They studied a multiagent simulation of a simplified two-echelon supply chain and showed the effectiveness of the method they proposed. Sui et al. [28] considered a Vendor-Managed Inventory (VMI) system where the supplier makes decisions of inventory decisions of inventory management for the retailer, and the retailer is not responsible for placing orders. Through a methodology based on reinforcement learning and numerical study, they show their approach can outperform the newsvendor. Zarandi et al. [29] presented a flexible fuzzy reinforcement learning algorithm where the value function is approximated by a fuzzy rule-based system and considered the problem of a fuzzy agent (supplier), that is, how to determine the amount of orders for each retailers based on their utility for supplier when its supply capacity is limited. Finally, the effectiveness of their proposed algorithm is proved by a simulation. Dogan et al. [4] used the Q-learning method to study an ordering and pricing policy in a multiretailer environment. Rana and Oliveira [30] use reinforcement learning methods to develop dynamic pricing strategies for interdependent perishable products or service. Kara and Dogan [5] used Q-learning and Sarsa reinforcement learning algorithms to study a dynamic inventory control issues for perishable products, with positive lead time and fixed lifetime. Our research further uses deep reinforcement learning to study this dynamic inventory control of perishable products.The aforementioned studies investigate the inventory problem for nonperishable and perishable products and use the nondeep reinforcement learning methods. Compared to their problems, our problem focuses on the inventory control of perishables, which makes the problem much more difficult. We use neural networks to avoid the curse of dimensionality and show that our deep reinforcement learning model outperforms the traditional reinforcement learning models without using neural networks.


## 3. Model

We consider a periodic-review single-product inventory system over a finite horizon ofT periods. The whole process can be defined as a Markov Decision Process. The decision maker is called the agent, and the thing it interacts with is called the environment. At each period (step) of a sequence of discrete time periods, t=1,2,…,T, the agent and the environment interact; the agent selects the action denoted by At , and the environment responds to At and presents a new situation to the agent. At the end of the period, the agent receives a numerical reward denoted by Rt+1, Rt+1∈ℝ, in part as a consequence of its action. Throughout this paper, we let superscript t denote the period. More specifically, by superscript t, we mean the beginning of the period; we denote the end of period t, which coincides with the beginning of the next period as t+1.Customer demand, denoted byDt, at the beginning of period t, is represented by a Poisson distribution with the parameter as dt or dtAt if the agent’s action At at the beginning of period t changes the demand distribution and d⋅ is a function of selling price p, strictly decreasing the selling price p. Let the product’s finite lifetime be denoted by l, variable cost by ct, and leadtime by Lt (0≤Lt<l). Let the age of an item be 0 by the time it is shipped to the agent, and its residual lifetime be l−i when its age is i. When an item’s age is greater than l, it has to be disposed. The inventory state, also known as the state of the agent, at the beginning of period t can be represented by a l−1-dimensional vector:(1)Xt=x1t,…,xit,…,xl−1t,where xit represents the level of inventory position of the items at the age of i. In particular, xot≡∑Ll−1xit (when L=0, xot≡∑1l−1xit) is the level of on-hand inventory, and xt≡∑1l−1xit is the level of inventory position of all ages.


### 3.1. An Action

Here, action spaceAt refers to the order quantity qt and price decision pt. The selling pricing pt is restricted to an interval p¯,p¯. Based on the selling price p, the parameter of Poisson demand d∈d¯,d¯, where d¯=dp¯,d¯=dp¯:(2)At=qt,pt.


### 3.2. Update Rule

Update rule, denoted byh⋅, describes the update of the environment state. In our research, the supply state remains unchanged in each period (Mt+1=Mt). The demand state in each period depends on selling price p. Last, we need to define the update rules for the inventory state.The update rules for the inventory stateXt can be divided into two cases according to the unmet demand handing principle. We first consider the backlogging case. If L=0 and L=1, then xit+1=xi−1t−Dt−xit+⋯+xl−1t+ for i=1 and xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=2,…,l−1; although L=0 and L=1 have the same state transition rule, the reward function is different; if L>1, then xit+1=xi−1t for i=1,…,L−1, xit+1=xi−1t−Dt−xit+⋯+xl−1t+ for i=L, and xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=L+1,…,l−1.For the lost-sales case, ifL=0 and L=1, then xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=1,…,l−1; if L>1, then xit+1=xi−1t for i=1,…,L−1 and xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=L,…,l−1.


### 3.3. Reward Function

In our study, our goal is to maximize the accumulative expected profit in the planning horizon, so our reward functionRt+1 can be represented by the following form. We first consider the backlogging case. If L=0, then(3)Rt+1=pt∗Dt−c∗qt−h∗xot+x0t−Dt−ν∗xl−1t−Dt+,ifxot+x0t≥Dt,pt∗Dt−c∗qt−u∗Dt−xot−x0t,ifxot+x0t<Dt.IfL>0, then(4)Rt+1=pt∗Dt−c∗qt−h∗xot−Dt−ν∗xl−1t−Dt+,ifxot≥Dt,pt∗Dt−c∗qt−u∗Dt−xot,ifxot<Dt.For the lost-sales case, ifL=0, then(5)Rt+1=pt∗Dt−c∗qt−h∗xot+x0t−Dt−ν∗xl−1t−Dt+,ifxot+x0t≥Dt,pt∗xot+x0t−c∗qt−u∗Dt−xot−x0t,ifxot+x0t<Dt.IfL>0, then(6)Rt+1=pt∗Dt−c∗qt−h∗xot−Dt−ν∗xl−1t−Dt+,ifxot≥Dt,pt∗xot−c∗qt−u∗Dt−xot,ifxot<Dt.where inventory carried forward to the next period incurs a unit holding cost h, unmet demand incurs a unit penalty cost u, and ν is unit disposal cost. When fixed ordering cost K is considered, reward function above will subtract K if order quantity is not 0.The sequence of events in periodt is as follows:(1)
Based on the environment stateEt, Et≡Dt,Mt,Xt, the agent selects an actionAt. Note that At is a vector, including ordering and pricing decisions. The order will be delivered at the beginning of period t+L; when L=0 the order is delivered immediately.(2)
During periodt, demand Dt arrives, which is discrete and stochastic depending on the selling price pt, and is satisfied by the on-hand inventory as much as possible by the agent. Unsatisfied demand is either backlogged or lost; the remaining inventory with positive lifetime can be carried over to the next period.(3)
At the end of periodt, the agent receives a reward Rt+1, which depends on the environment state and action At.(4)
At the beginning of periodt+1, the agent receives an order (if any), and the environment state is updated to Et+1 according to the update rule.(7)h:Et,At⟶Et+1.For this joint pricing inventory problem, we introduce the notations in Table1.Table 1
Notations grouped by the elements of the RL problem.
ptPrice charged by the agent for each itemqtOrder quantityDtCustomer demand (Dtpt if depending on pt)lLifetime of the product,0<l<∞LOrder lead time,L<lKFixed order costcVariable cost per itemuPenalty cost per unmet itemhHolding cost for per item leftνCost for each item disposedxitInventory position of agei, 0≤i≤l−1xtInventory position of all agesxotOn-hand inventoryRt+1The reward function of the agentIn this paper, we assume as in Chen et al. [2] that c≤u/1−γ, which eliminates the incentive to intentionally carry the back orders. We also assume that items with different lifetimes are charged the same price and that the back orders are met at cost c at the end of each planning period.


## 3.1. An Action

Here, action spaceAt refers to the order quantity qt and price decision pt. The selling pricing pt is restricted to an interval p¯,p¯. Based on the selling price p, the parameter of Poisson demand d∈d¯,d¯, where d¯=dp¯,d¯=dp¯:(2)At=qt,pt.


## 3.2. Update Rule

Update rule, denoted byh⋅, describes the update of the environment state. In our research, the supply state remains unchanged in each period (Mt+1=Mt). The demand state in each period depends on selling price p. Last, we need to define the update rules for the inventory state.The update rules for the inventory stateXt can be divided into two cases according to the unmet demand handing principle. We first consider the backlogging case. If L=0 and L=1, then xit+1=xi−1t−Dt−xit+⋯+xl−1t+ for i=1 and xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=2,…,l−1; although L=0 and L=1 have the same state transition rule, the reward function is different; if L>1, then xit+1=xi−1t for i=1,…,L−1, xit+1=xi−1t−Dt−xit+⋯+xl−1t+ for i=L, and xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=L+1,…,l−1.For the lost-sales case, ifL=0 and L=1, then xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=1,…,l−1; if L>1, then xit+1=xi−1t for i=1,…,L−1 and xit+1=xi−1t−Dt−xit+⋯+xl−1t++ for i=L,…,l−1.


## 3.3. Reward Function

In our study, our goal is to maximize the accumulative expected profit in the planning horizon, so our reward functionRt+1 can be represented by the following form. We first consider the backlogging case. If L=0, then(3)Rt+1=pt∗Dt−c∗qt−h∗xot+x0t−Dt−ν∗xl−1t−Dt+,ifxot+x0t≥Dt,pt∗Dt−c∗qt−u∗Dt−xot−x0t,ifxot+x0t<Dt.IfL>0, then(4)Rt+1=pt∗Dt−c∗qt−h∗xot−Dt−ν∗xl−1t−Dt+,ifxot≥Dt,pt∗Dt−c∗qt−u∗Dt−xot,ifxot<Dt.For the lost-sales case, ifL=0, then(5)Rt+1=pt∗Dt−c∗qt−h∗xot+x0t−Dt−ν∗xl−1t−Dt+,ifxot+x0t≥Dt,pt∗xot+x0t−c∗qt−u∗Dt−xot−x0t,ifxot+x0t<Dt.IfL>0, then(6)Rt+1=pt∗Dt−c∗qt−h∗xot−Dt−ν∗xl−1t−Dt+,ifxot≥Dt,pt∗xot−c∗qt−u∗Dt−xot,ifxot<Dt.where inventory carried forward to the next period incurs a unit holding cost h, unmet demand incurs a unit penalty cost u, and ν is unit disposal cost. When fixed ordering cost K is considered, reward function above will subtract K if order quantity is not 0.The sequence of events in periodt is as follows:(1)
Based on the environment stateEt, Et≡Dt,Mt,Xt, the agent selects an actionAt. Note that At is a vector, including ordering and pricing decisions. The order will be delivered at the beginning of period t+L; when L=0 the order is delivered immediately.(2)
During periodt, demand Dt arrives, which is discrete and stochastic depending on the selling price pt, and is satisfied by the on-hand inventory as much as possible by the agent. Unsatisfied demand is either backlogged or lost; the remaining inventory with positive lifetime can be carried over to the next period.(3)
At the end of periodt, the agent receives a reward Rt+1, which depends on the environment state and action At.(4)
At the beginning of periodt+1, the agent receives an order (if any), and the environment state is updated to Et+1 according to the update rule.(7)h:Et,At⟶Et+1.For this joint pricing inventory problem, we introduce the notations in Table1.Table 1
Notations grouped by the elements of the RL problem.
ptPrice charged by the agent for each itemqtOrder quantityDtCustomer demand (Dtpt if depending on pt)lLifetime of the product,0<l<∞LOrder lead time,L<lKFixed order costcVariable cost per itemuPenalty cost per unmet itemhHolding cost for per item leftνCost for each item disposedxitInventory position of agei, 0≤i≤l−1xtInventory position of all agesxotOn-hand inventoryRt+1The reward function of the agentIn this paper, we assume as in Chen et al. [2] that c≤u/1−γ, which eliminates the incentive to intentionally carry the back orders. We also assume that items with different lifetimes are charged the same price and that the back orders are met at cost c at the end of each planning period.


## 4. Deep Reinforcement Learning Methods

The objective of reinforcement learning is to learn a policyπ that achieves near-optimal accumulated reward for the agent. Q-learning [31] is one widely used value iterative reinforcement learning method where the expected total discount rewards of state-action pairs can be approximated by a Q-function table based on the bellman equation, as shown in Function 7. Q-learning also has obvious limitation, that is, when there is a large state space, it is impractical and inefficient to record all the states and actions. Mnih et al. [32] extends Q-learning to Deep Q-network (DQN) which uses a neural network to approximate the Q-function table. DQN updates the parameters of the neural network by minimizing the difference between the predicted Q-values and the target Q- values, where the target Q-values are estimated by current reward and predicted Q-values from the next state. Meanwhile, to avoid training instability caused by correlation between training data, a replay memory pool is used:(8)Qs,a←Qs,a+αR+γmaxa′Qs′,a′−Qs,a.As mentioned before, in our joint pricing and inventory control problem, the state of the agent is expressed by the inventory stateXt which integrates different ages and corresponding quantities. Here, the initial inventory state is X0. In our proposed algorithm PAQ-DQN, there are two same neural networks with the same structure but different parameters θ and θ^, respectively. We adopt the fixed Q-targets’ policy in standard DQN. The neural network that predicts Q-values has the latest parameters, while the neural network that predicts target Q-values uses the old parameters. Each neural network has two hidden layers, and there are 128 neurons in each layer, we use the ReLU as the activation function. In each time period, based on Q-values from the neural network, the ε-greedy policy will be executed to select an action from the action space which contains a combination of ordering and pricing. After receiving the reward from the environment, the target Q-values are estimated by current rewards and discounted predicted Q-values from the next state, as shown in equation (9). The parameters of the network θ are updated by minimizing the difference between the predicted Q-values and the target Q- values, as shown in equation (10). After a fixed number of steps, assign the value of parameter θ to θ^. The details of the algorithm named perishables integrate age and quantity deep Q-network (PAQ-DQN) are shown in Algorithm 1:(9)yt=Rt+1,if epoch terminates at stept+1,Rt+1+γ∗maxA′Q^Xt+1,A′;θ^,otherwise,(10)Lθ=EXit,Ait,Rit+1,Xit+1yt−QXt,At;θ2.Algorithm 1:Perishables integrate age and quantity deep Q-network.
(1)
Initialize replay memory poolD to capacity N(2)
Use random weightsθ to initialize the action-value function Q(3)
Initialize target action-value functionQ^ with weights θ^=θ(4)
Forepoch  = 1 to number of epochsdo(5)
Reset the environment and initialize stateX0(6)
fort=1,Tdo(7)
With probabilityε, select a random action At, otherwise select At=argmaxAtQXt,At;θ (ϵ-greedy policy)(8)
Execute actionAt and observe reward Rt+1 and Xt+1(9)
Store transition (Xt,At,Rt+1,Xt+1) in the replay memory pool D(10)
SetXt+1=Xt(11)
Sample a minibatch of transitionsXit,Ait,Rit+1,Xit+1, ∀i=1,…,N from replay memory pool D(12)
Calculate the targetQ-value by equation (9)(13)
Update the parameters of networkθ by equation (10)(14)
Every C steps resetQ^=Q(15)
end for(16)
end forThe second reinforcement learning algorithm named perishables integrate age and quantity advantage actor-critic (PAQ-A2C). A2C is a method combining policy gradient and function approximation. Actor-critic (A2C) has two networks, one policy network, known as actor and used to output policy, and one value network, known as critic and used to evaluate the policy from actor. In our algorithm, both the policy network and value network have two hidden layers, and there are 128 neurons in each layer with the ReLU activation function. Especially, the activation function of the policy network output layer is the Softmax, which outputs the probability of each action being executed in the current state. In each time period, based on the current inventory state, an action will be executed by the policy network. After receiving the reward from the environment, the value network will evaluate this policy and output atd_error. The parameters of value network θv can be updated by Equation (11), where yt is the target value calculated by equation (12). The policy network is updated by θp←θp+α∗▽θpJθp, where α is learning rate, and gradient ▽θpJθp is shown in equation (13) where advantage function is estimated by equation (14). The details of the algorithm of proposed perishables integrate age and quantity advantage actor-critic (PAQ-A2C) are shown in Algorithm 2:(11)Lθv=yt−VπXt;θv,(12)yt=Rt+1+γ∗VπXt+1;θv,(13)▽θpJθp≈▽θplogπθpAt|XtA^Xt,At,(14)A^Xt,At=Rt+1+γ∗VπXt+1;θv−VπXt;θv.Algorithm 2: Perishables integrate age and quantity advantage actor-critic.
(1)
Use random weightsθpandθν to initialize the policy network and value network(2)
forepoch  = 1 to number of epochsdo(3)
Reset the environment and initialize stateX0(4)
Fort  = 1, Tdo(5)
Take actionAt based on action probability πθp⋅|Xt(6)
Execute actionAt and observe reward Rt+1 and Xt+1(7)
Update the parametersθν of the value network by minimizing the loss function equation (11)(8)
Estimate advantage function by equation (14)(9)
Update the policy network parametersθp←θp+αp▽θpJθp, where ▽θpJθp is calculated by equation (13)(10)
SetXt+1=Xt(11)
end for(12)
end for


## 5. Experiments

In this section, we conduct simulation studies to evaluate the performance of our proposed reinforcement learning algorithms and investigate the positive effects of the proposed algorithms on the profit of dynamic pricing and the impacts of the key parameters. Ordering and pricing policy are also discussed in situation involving fixed ordering cost. In this experiment, we only show the discussions on the backlogging case, the discussions of the lost-sales case is carried out in Appendix.The values of various parameters are set in Table2. For simplicity, the value range of the price p and order quantity q are restricted to 32,37 and 0,31, respectively.Table 2
The parameter values.
ParameterValuesDescriptionT30Periods of the planning horizonl{2,3,4}Lifetime of perishablesxl0Initial inventory position of all agesγ0.9Discount factorL{0, 1, 2}Lead timeK{25, 50}Fixed ordering costc22.5Unit variable costh0.22Unit holding costu10.78Unit penalty costν10unit disposal costdt84−2pFunction of selling priceε01Initial exploration rateα{0.01, 0.001, 0.0001}Learning rateεdecay1×103,1×104,1×105Exploration rate decay parameterIn the reinforcement learning method, the effect of hyperparameters on final performance is very important, so we need to set the variation rules for relevant parameters, exploration rateε and learning rate α. We adopt ε-greedy policy here; ε is decreasing linearly, that is, search-then-convergence form in Darken et al. [33]:(15)εepoch=ε01+y,where y=epoch2/εdecay, ε0 is the initial value of the ε, and εdecay is the decay parameter.


### 5.1. Experiments on Dynamic Pricing with Positive Lead Time

Firstly, we conduct experiments for perishables’ joint ordering and pricing with positive lead time (whereL=1). In order to examine the positive impact of dynamic pricing, we consider a fixed-price policy where the agent always takes the fixed best price which achieves the highest revenue. Let MEP and MEPFP be the expected mean epochs profits for the dynamic ordering and pricing policy and the fixed-price ordering policy, respectively, and MDC and MDCFP be the mean epochs disposal cost. After ten thousand simulations, we get the results in Table 3. Table 3 shows that PAQ-DQN achieves better performance than PAQ-A2C when lifetime is 2 and 3, but when lifetime is 4, they achieve almost the same results. Mean epochs profits and mean epochs disposal cost for two algorithms are almost all increasing and decreasing with lifetimes, which are in line with expectations, because the longer the lifetime is, the more similar it is to ordinary goods and perishables have more lifetime to sell out under the FIFO policy. From Table 3, it is easy to find out that it is better to adjust the price in a dynamic way so that the price can be adjusted according to the availability of inventory and the remaining life of the product and maximize the profits.Table 3
Results for dynamic pricing with positive lead time.
MethodLifetimeMEPMEPFPMDCMDCFPPAQ-DQN23242.1042911.139254.905291.99734734.7664455.81440.439110.14644840.4434714.71234.06628.642PAQ-A2C23081.5462305.394232.000573.74934513.7864474.915207.470130.27844919.3322278.32449.802748.916Table4 shows the comparison between the tabular Q-learning and reinforcement learning methods on mean epoch profits and mean epoch disposal cost. From the table, we can see our proposed PAQ-DQN and PAQ-A2C obviously performs better than the Q-learning method. As we have mentioned before, Q-learning is a tabular method; it stores every state-action value in a table, but in our perishables inventory system, we considered the different ages, so the state space increases exponentially with lifetime, which is inefficient and impractical. Moreover, the amount of computing power and time required increase greatly with lifetime for the Q-learning method.Table 4
Results.
MethodlifetimeLead timeMEPMDCPAQ-DQN205377.02118.74713242.104254.905PAQ-A2C205385.65321.22213081.546232.000Q-learning204800.62231.3761592.721151.462


### 5.2. Experiments on the Performance of Proposed Algorithms

In this case, we compute the mean epoch profits for the optimal policy and proposed PAQ-DQN and PAQ-A2C with zero lead time. In particular, we set up an upper bound benchmark for this computation and define it as the optimal policy. The optimal policy takes the same price action as the PAQ-DQN and PAQ-A2C in each period, and its order quantity is always equal to the real demandDt in each period, which means there is always no holding cost, penalty cost, and disposal cost for the planning horizon. Although there may be still some unreasonable place, this can be a useful metric to gauge the performance of the agent. Table 5 shows the computed results after twenty thousand simulations and MEPaverage, MEPaverage=MEPoptimal−MEP/T, where T denotes the mean difference between the mean epochs profits from deep reinforcement learning methods and the mean epoch profits from the optimal policy. From the table, we can see our proposed algorithms achieve a good performance for three different lifetimes, where the benchmark is a loose upper bound from the real demand Dt and the difference from the average optimal profit is almost always less than the highest possible profit per unit, that is, MEPaverage≤p¯−c. And the algorithm PAQ-A2C is slightly better than algorithm PAQ-DQN. Figure 1 shows the real epoch profits for the proposed PAQ-DQN and PAQ-A2C (in order to show the variation, we let the initial negative values as zero); from the figure, we can see that two methods quickly reached a relatively flat of profitability and PAQ-A2C showed more stable properties at the beginning of the learning process.Table 5
MEP for proposed algorithm and optimal policy.
MethodLifetimeMEPMEPoptimalMEPaverageMEP/MEPoptimal∗100%PAQ-DQN25377.0215691.35510.47794.47735514.7105691.2535.88496.89845409.5875664.8038.50795.495PAQ-A2C25385.6535662.3799.22495.11335590.5595674.6252.80298.51945428.0725677.2018.30495.612Figure 1
Real epoch profits for PAQ-DQN and PAQ-A2C, where the lifetime 3 and lifetime 4 are shown.
(a)(b)Figures2–4 show the scatter plots of the profits difference between the optimal policy and the proposed PAQ-DQN algorithm for three different lifetimes. To better show the convergence rate, the figure is drawn on a log-log scale. From three figures, we can see three MEP differences begin to decrease rapidly after about fifty simulations; this demonstrates our deep reinforcement learning method works, the agent gradually learns how to order, and price is near optimal. Besides, the fitting lines in the figures are used to depict the convergence rate, and the following fitting line functions are for lifetime 2, 3, and 4. Here, we also carry out sensitivity analysis to investigate the effects of learning rate α and exploration parameter εdecay for the training of the proposed deep reinforcement learning methods, respectively. Figure 5 demonstrates the MEP for three different learning rates on PAQ-DQN and the learning rate α at 0.001 is the best for three different lifetimes, and α at 0.01 is very close to the best performance. Figure 6 shows the effects of exploration parameters εdecay on PAQ-DQN, and when the exploration parameter εdecay is 1×103, the agent gets a higher reward than the other two parameters. And the difference between the three parameters is very obvious. From the above two sensitivity analysis cases, the importance of hyperparameter is verified, and this is a common problem in deep learning. To show the expansibility of the algorithm, we also extend the distribution of the demand into an additive form in Chen et al. [2], where random term has a zero mean. By setting the random term which satisfies a uniform distribution in A,B, where A and B are symmetric and the absolute value is 2, we get a near-optimal performance with optimal rate 96.344%:(16)logMEPdiff≈−0.609logepochs+11.794r2=0.963,(17)logMEPdiff≈−0.737logepochs+12.445r2=0.986,(18)logMEPdiff≈−0.575logepochs+11.278r2=0.956.Figure 2
Log-log scaleMEP difference.Figure 3
Log-log scaleMEP difference.Figure 4
Log-log scaleMEP difference.Figure 5
MEP for learning rates α.Figure 6
MEP for εdecay.


### 5.3. Experiments on Dynamic Ordering and Pricing with No Fixed Ordering Cost

In this case, when the real epoch profits gradually becomestable (stable means the real epoch profits always go up and down in a small fixed range over time) with the number of epochs, it indicates that the agent has learned a relatively stable state-to-action mapping relationship. In this section, we first extract the latest n epochs with stable mapping, denoted by dataset p=xi1,Ai1,…,xit,Ait,…,xiT,AiTi=1n, where xit is the inventory state vector for epoch i and period t. To facilitate discussion, we will use the fragment p˜t=xit,Aiti=1nt=1,…,T extracted from p.Chen et al. [2] has discussed the properties of optimal policies in the joint pricing and inventory system without fixed ordering cost. From the above settings and through our proposed reinforcement learning methods, when there is no fixed ordering cost, we get that the learned order quantity is nonincreasing in both outstanding and on-hand inventory levels. When L=0, the learned price is always equal to the price that achieves highest expected revenue, and when L>0, the learned price is most sensitive to the oldest on-hand inventory.Figure7 shows that the order quantity decreases with the inventory position and on-hand inventory. In order to show the sensitivity, we extract the fragment P˜5 from P as an example, where l=4,L=2, and n=1000. In the inventory state X5=x1,x2,x3, where x1 is the outstanding order and x3 is the oldest on-hand inventory, we find that x1 and x2 are equal to a fixed value happens more than 500 times out of 1000, and when x1=x2, the price decreases with the oldest on-hand inventory. The same results can be obtained from other fragments. In this setting, Figure 8 shows that the price decreases with the oldest on-hand inventory, which means when the oldest inventory increases, the agent tends to set a lower price.Figure 7
This statistic results are fromp˜5, where l=4 and L=2. When L=2, xit is a vector and order quantity is the mean from the same inventory value.
(a)(b)Figure 8
This statistic results are fromp˜5, where l=4 and L=2. When L=2, xit is a vector and price is the mean from the same inventory value.


### 5.4. Experiments on Dynamic Ordering and Pricing with Fixed Ordering Cost

In this part, we will consider the case when there is a fixed ordering cost in this joint pricing and inventory system. We use our propose deep reinforcement learning algorithms to solve this case, and in order to measure the final performance, we set up a loose upper bound as our benchmark. In this benchmark, there are trade-offs between different costs. The price decision is supplied by algorithms. For ease of discussion and simplicity, we assume zero penalty costs and zero disposal costs to be achieved, which mean each demand will be met and each order will be sold withinl period. We also assume that the initial inventory is zero; thus, the first order will always be placed at the beginning of the planning horizon. In particular, when L>0, there may be a penalty cost at the beginning. Dt is the real demand in period t, t=1,…,l,…,T. It is obviously unwise to order every period in this setting.WhenL=0, taking into account the width of finite lifetime l and the minimization of total cost, it is easy to see that the agent needs to place at least one order every l term and every order is just consumed by the next one. In the first l term, if D2+⋯+Dl+D3+⋯+Dl+⋯+Dl−2+Dl−1+Dl−1∗h≤K, it only needs one order at the beginning of the first period and ordering quantity q=D1+⋯+Dl. Morover, at some point to, whether to order depends on the time of last order tn,to−tn≤l, if Dtn+1+⋯+Dto−1+Dtn+2+⋯+Dto−1+⋯+Dto−1∗h≤K and Dtn+1+⋯+Dto+Dtn+2+⋯+Dto+⋯+Dto∗h>K, it should order at point to and the order quantity for tn is q=Dtn+⋯+Dto−1. When L>0, taking into account the width of finite lifetime l and the minimization of total cost, it is easy to see that the agent needs to place at least one order every l term. In the first l term, there is a penalty cost, D1+⋯+DL∗u, due to the lag of the order. At some point to (to≠1), whether to order depends on the time of last order tn and in order to make the subsequent penalty cost zero, to−tn≤l−L. If Dtn+L+1+⋯+Dto+L−1+Dtn+L+2+⋯+Dto+L−1+⋯+Dto+L−1∗h≤K and Dtn+L+1+⋯+Dto+L+Dtn+L+2+⋯+Dto+L+⋯+Dto+L∗h>K, it should order at point to, and when tn=1, the order quantity for tn is q=Dtn+⋯+Dto+L−1; when tn≠1, the order quantity for tn is q=Dtn+L+⋯+Dto+L−1.We consider two different fixed ordering costsK, K∈25,50, two different penalty costs u, u∈10.78,4.18 (corresponding, u/h+u∈98%,95%), and two different price-demand functions dt, and the first one is shown in Table 2 and another one is dt=380−10p. Lifetime l=4, lead time L=0,1, and a larger order action space is considered for dt=380−10p.Under all of the above setting, we find that whenL=0, the convergent price is always the price that maximizes the expected revenue. This is in line with expectations. Same as the no fixed ordering cost case, the order quantity is nonincreasing in both outstanding and on-hand inventory levels. Table 6 shows the MEP results from PAQ-DQN and PAQ-A2C when the fixed ordering cost is 25 and 50 and price-demand function dt=380−10p after thirty thousand simulations. From the table, we can see that, under the same conditions, the mean epoch profits MEP decreases with the lead time and the fixed ordering cost. When L=0, algorithm PAQ-A2C performs better than PAQ-DQN, and when L>0, our proposed PAQ-DQN performs better than PAQ-A2C. More interestingly, in our learned convergent policies, we find there exist one or two critical values in the inventory position in each period in each case when L=0 and L=1. We denote cvt as the critical value in each period for the one critical value cases and cv1t and cv2t in each period for the two critical value cases, cv1t<cv2t. In the one critical value case, when xt<cvt, there will be a fixed order quantity q1; when xt≥cvt, the fixed order quantity is q2. In the case of two critical values, when xt<cv1t, the fixed order quantity is q1; when cv1t≤xt<cv2t, the fixed order quantity is q2; when xt≥cv2t, the fixed order quantity is q3. For more details about obtaining the critical values, see Appendix.Table 6
MEP with fixed ordering cost.
MethodLifetimeLead timeKu/h+u (%)MEPMEPbenchmarkMEP/MEPbenchmark∗100%PAQ-DQN40509815041.5516172.3993.011509814484.1815576.9992.980259815753.9416507.7995.431259814544.0015436.5394.22PAQ-A2C40509815116.7716178.2293.441509814223.1315423.0392.220259815813.4916502.1695.831259814896.7015858.3293.94Table7 shows the MEP of learned policies from algorithm PAQ-DQN; from the table, we can see our learned policies achieve a higher optimal rate and are closer to the upper bound, compared to Table 6. Table 8 shows the MEP comparison between the learned policies and the algorithm PAQ-DQN; from the table, we can see our learned policies achieve a higher MEP and lower MDC, which means our learned policies are working well.Table 7
MEP for the learned policies.
LifetimeLead timeKMEPpoliciesMEPbenchmarkMEPpolicies/MEPbenchmark∗100%402516013.5616525.7596.9012514911.0615540.6295.95405015421.4516197.4594.1015014771.2115583.8994.79Table 8
Comparison for PAQ-DQN.
LifetimeLead timeKMEPMEPpoliciesMDCMDCpolicies402515753.9416013.562.45012514544.0014911.0636.870405015041.5515241.451.92015014484.1814771.2197.6756.97The above discussion is mainly based on the current inventory state. Next, we will try to add the historical inventory states and action information to the state to discuss its impact on the final performance. Here, we define the new state to beSt=Xt−L,At−L,Xt−L+1,…,Xt when L>0, and when L=0, we also discuss the gradual influence of the addition of information in the state on the final performance. At the same time the dimension of the state accompanying the increase in information will also increase.Table9 shows the results after ten thousand simulations and there we add the inventory state and action from the previous period for the new state when L=0. From the table, we can see that the new state contains more information almost all performs better than the single current inventory state, which means the current decisions of the agent are influenced by not only the current inventory state but also the inventory states of the previous periods. In Table 10, MEPS1 and MEPS2, respectively, represent the inventory state and action information of the previous period and the previous two periods added to the current state. From the table, we find that the final performance did not get better and better with the continuous addition of the historical information, which also confirms that the dimensions of the state mentioned above continue to increase with the addition of information, which may have a negative impact on learning.Table 9
MEP for new state forms.
LifetimeLead timeKu/h+u (%)MEPPAQ−DQNMEPPAQ−A2CMEPnew−state30509815052.74715126.55315465.9631509813685.98713544.11513842.541250988310.2387018.2349633.66530509514937.28514996.44415604.5071509513894.83213847.47414317.4532509510470.1349989.93510299.061Table 10
MEP for new state forms.
LifetimeLead timeKu/h+u (%)MEPPAQ−DQNMEPPAQ−A2CMEPS1MEPS230509514937.28514996.44415604.50715309.8989815052.74715126.55315465.96315117.522


## 5.1. Experiments on Dynamic Pricing with Positive Lead Time

Firstly, we conduct experiments for perishables’ joint ordering and pricing with positive lead time (whereL=1). In order to examine the positive impact of dynamic pricing, we consider a fixed-price policy where the agent always takes the fixed best price which achieves the highest revenue. Let MEP and MEPFP be the expected mean epochs profits for the dynamic ordering and pricing policy and the fixed-price ordering policy, respectively, and MDC and MDCFP be the mean epochs disposal cost. After ten thousand simulations, we get the results in Table 3. Table 3 shows that PAQ-DQN achieves better performance than PAQ-A2C when lifetime is 2 and 3, but when lifetime is 4, they achieve almost the same results. Mean epochs profits and mean epochs disposal cost for two algorithms are almost all increasing and decreasing with lifetimes, which are in line with expectations, because the longer the lifetime is, the more similar it is to ordinary goods and perishables have more lifetime to sell out under the FIFO policy. From Table 3, it is easy to find out that it is better to adjust the price in a dynamic way so that the price can be adjusted according to the availability of inventory and the remaining life of the product and maximize the profits.Table 3
Results for dynamic pricing with positive lead time.
MethodLifetimeMEPMEPFPMDCMDCFPPAQ-DQN23242.1042911.139254.905291.99734734.7664455.81440.439110.14644840.4434714.71234.06628.642PAQ-A2C23081.5462305.394232.000573.74934513.7864474.915207.470130.27844919.3322278.32449.802748.916Table4 shows the comparison between the tabular Q-learning and reinforcement learning methods on mean epoch profits and mean epoch disposal cost. From the table, we can see our proposed PAQ-DQN and PAQ-A2C obviously performs better than the Q-learning method. As we have mentioned before, Q-learning is a tabular method; it stores every state-action value in a table, but in our perishables inventory system, we considered the different ages, so the state space increases exponentially with lifetime, which is inefficient and impractical. Moreover, the amount of computing power and time required increase greatly with lifetime for the Q-learning method.Table 4
Results.
MethodlifetimeLead timeMEPMDCPAQ-DQN205377.02118.74713242.104254.905PAQ-A2C205385.65321.22213081.546232.000Q-learning204800.62231.3761592.721151.462


## 5.2. Experiments on the Performance of Proposed Algorithms

In this case, we compute the mean epoch profits for the optimal policy and proposed PAQ-DQN and PAQ-A2C with zero lead time. In particular, we set up an upper bound benchmark for this computation and define it as the optimal policy. The optimal policy takes the same price action as the PAQ-DQN and PAQ-A2C in each period, and its order quantity is always equal to the real demandDt in each period, which means there is always no holding cost, penalty cost, and disposal cost for the planning horizon. Although there may be still some unreasonable place, this can be a useful metric to gauge the performance of the agent. Table 5 shows the computed results after twenty thousand simulations and MEPaverage, MEPaverage=MEPoptimal−MEP/T, where T denotes the mean difference between the mean epochs profits from deep reinforcement learning methods and the mean epoch profits from the optimal policy. From the table, we can see our proposed algorithms achieve a good performance for three different lifetimes, where the benchmark is a loose upper bound from the real demand Dt and the difference from the average optimal profit is almost always less than the highest possible profit per unit, that is, MEPaverage≤p¯−c. And the algorithm PAQ-A2C is slightly better than algorithm PAQ-DQN. Figure 1 shows the real epoch profits for the proposed PAQ-DQN and PAQ-A2C (in order to show the variation, we let the initial negative values as zero); from the figure, we can see that two methods quickly reached a relatively flat of profitability and PAQ-A2C showed more stable properties at the beginning of the learning process.Table 5
MEP for proposed algorithm and optimal policy.
MethodLifetimeMEPMEPoptimalMEPaverageMEP/MEPoptimal∗100%PAQ-DQN25377.0215691.35510.47794.47735514.7105691.2535.88496.89845409.5875664.8038.50795.495PAQ-A2C25385.6535662.3799.22495.11335590.5595674.6252.80298.51945428.0725677.2018.30495.612Figure 1
Real epoch profits for PAQ-DQN and PAQ-A2C, where the lifetime 3 and lifetime 4 are shown.
(a)(b)Figures2–4 show the scatter plots of the profits difference between the optimal policy and the proposed PAQ-DQN algorithm for three different lifetimes. To better show the convergence rate, the figure is drawn on a log-log scale. From three figures, we can see three MEP differences begin to decrease rapidly after about fifty simulations; this demonstrates our deep reinforcement learning method works, the agent gradually learns how to order, and price is near optimal. Besides, the fitting lines in the figures are used to depict the convergence rate, and the following fitting line functions are for lifetime 2, 3, and 4. Here, we also carry out sensitivity analysis to investigate the effects of learning rate α and exploration parameter εdecay for the training of the proposed deep reinforcement learning methods, respectively. Figure 5 demonstrates the MEP for three different learning rates on PAQ-DQN and the learning rate α at 0.001 is the best for three different lifetimes, and α at 0.01 is very close to the best performance. Figure 6 shows the effects of exploration parameters εdecay on PAQ-DQN, and when the exploration parameter εdecay is 1×103, the agent gets a higher reward than the other two parameters. And the difference between the three parameters is very obvious. From the above two sensitivity analysis cases, the importance of hyperparameter is verified, and this is a common problem in deep learning. To show the expansibility of the algorithm, we also extend the distribution of the demand into an additive form in Chen et al. [2], where random term has a zero mean. By setting the random term which satisfies a uniform distribution in A,B, where A and B are symmetric and the absolute value is 2, we get a near-optimal performance with optimal rate 96.344%:(16)logMEPdiff≈−0.609logepochs+11.794r2=0.963,(17)logMEPdiff≈−0.737logepochs+12.445r2=0.986,(18)logMEPdiff≈−0.575logepochs+11.278r2=0.956.Figure 2
Log-log scaleMEP difference.Figure 3
Log-log scaleMEP difference.Figure 4
Log-log scaleMEP difference.Figure 5
MEP for learning rates α.Figure 6
MEP for εdecay.


## 5.3. Experiments on Dynamic Ordering and Pricing with No Fixed Ordering Cost

In this case, when the real epoch profits gradually becomestable (stable means the real epoch profits always go up and down in a small fixed range over time) with the number of epochs, it indicates that the agent has learned a relatively stable state-to-action mapping relationship. In this section, we first extract the latest n epochs with stable mapping, denoted by dataset p=xi1,Ai1,…,xit,Ait,…,xiT,AiTi=1n, where xit is the inventory state vector for epoch i and period t. To facilitate discussion, we will use the fragment p˜t=xit,Aiti=1nt=1,…,T extracted from p.Chen et al. [2] has discussed the properties of optimal policies in the joint pricing and inventory system without fixed ordering cost. From the above settings and through our proposed reinforcement learning methods, when there is no fixed ordering cost, we get that the learned order quantity is nonincreasing in both outstanding and on-hand inventory levels. When L=0, the learned price is always equal to the price that achieves highest expected revenue, and when L>0, the learned price is most sensitive to the oldest on-hand inventory.Figure7 shows that the order quantity decreases with the inventory position and on-hand inventory. In order to show the sensitivity, we extract the fragment P˜5 from P as an example, where l=4,L=2, and n=1000. In the inventory state X5=x1,x2,x3, where x1 is the outstanding order and x3 is the oldest on-hand inventory, we find that x1 and x2 are equal to a fixed value happens more than 500 times out of 1000, and when x1=x2, the price decreases with the oldest on-hand inventory. The same results can be obtained from other fragments. In this setting, Figure 8 shows that the price decreases with the oldest on-hand inventory, which means when the oldest inventory increases, the agent tends to set a lower price.Figure 7
This statistic results are fromp˜5, where l=4 and L=2. When L=2, xit is a vector and order quantity is the mean from the same inventory value.
(a)(b)Figure 8
This statistic results are fromp˜5, where l=4 and L=2. When L=2, xit is a vector and price is the mean from the same inventory value.


## 5.4. Experiments on Dynamic Ordering and Pricing with Fixed Ordering Cost

In this part, we will consider the case when there is a fixed ordering cost in this joint pricing and inventory system. We use our propose deep reinforcement learning algorithms to solve this case, and in order to measure the final performance, we set up a loose upper bound as our benchmark. In this benchmark, there are trade-offs between different costs. The price decision is supplied by algorithms. For ease of discussion and simplicity, we assume zero penalty costs and zero disposal costs to be achieved, which mean each demand will be met and each order will be sold withinl period. We also assume that the initial inventory is zero; thus, the first order will always be placed at the beginning of the planning horizon. In particular, when L>0, there may be a penalty cost at the beginning. Dt is the real demand in period t, t=1,…,l,…,T. It is obviously unwise to order every period in this setting.WhenL=0, taking into account the width of finite lifetime l and the minimization of total cost, it is easy to see that the agent needs to place at least one order every l term and every order is just consumed by the next one. In the first l term, if D2+⋯+Dl+D3+⋯+Dl+⋯+Dl−2+Dl−1+Dl−1∗h≤K, it only needs one order at the beginning of the first period and ordering quantity q=D1+⋯+Dl. Morover, at some point to, whether to order depends on the time of last order tn,to−tn≤l, if Dtn+1+⋯+Dto−1+Dtn+2+⋯+Dto−1+⋯+Dto−1∗h≤K and Dtn+1+⋯+Dto+Dtn+2+⋯+Dto+⋯+Dto∗h>K, it should order at point to and the order quantity for tn is q=Dtn+⋯+Dto−1. When L>0, taking into account the width of finite lifetime l and the minimization of total cost, it is easy to see that the agent needs to place at least one order every l term. In the first l term, there is a penalty cost, D1+⋯+DL∗u, due to the lag of the order. At some point to (to≠1), whether to order depends on the time of last order tn and in order to make the subsequent penalty cost zero, to−tn≤l−L. If Dtn+L+1+⋯+Dto+L−1+Dtn+L+2+⋯+Dto+L−1+⋯+Dto+L−1∗h≤K and Dtn+L+1+⋯+Dto+L+Dtn+L+2+⋯+Dto+L+⋯+Dto+L∗h>K, it should order at point to, and when tn=1, the order quantity for tn is q=Dtn+⋯+Dto+L−1; when tn≠1, the order quantity for tn is q=Dtn+L+⋯+Dto+L−1.We consider two different fixed ordering costsK, K∈25,50, two different penalty costs u, u∈10.78,4.18 (corresponding, u/h+u∈98%,95%), and two different price-demand functions dt, and the first one is shown in Table 2 and another one is dt=380−10p. Lifetime l=4, lead time L=0,1, and a larger order action space is considered for dt=380−10p.Under all of the above setting, we find that whenL=0, the convergent price is always the price that maximizes the expected revenue. This is in line with expectations. Same as the no fixed ordering cost case, the order quantity is nonincreasing in both outstanding and on-hand inventory levels. Table 6 shows the MEP results from PAQ-DQN and PAQ-A2C when the fixed ordering cost is 25 and 50 and price-demand function dt=380−10p after thirty thousand simulations. From the table, we can see that, under the same conditions, the mean epoch profits MEP decreases with the lead time and the fixed ordering cost. When L=0, algorithm PAQ-A2C performs better than PAQ-DQN, and when L>0, our proposed PAQ-DQN performs better than PAQ-A2C. More interestingly, in our learned convergent policies, we find there exist one or two critical values in the inventory position in each period in each case when L=0 and L=1. We denote cvt as the critical value in each period for the one critical value cases and cv1t and cv2t in each period for the two critical value cases, cv1t<cv2t. In the one critical value case, when xt<cvt, there will be a fixed order quantity q1; when xt≥cvt, the fixed order quantity is q2. In the case of two critical values, when xt<cv1t, the fixed order quantity is q1; when cv1t≤xt<cv2t, the fixed order quantity is q2; when xt≥cv2t, the fixed order quantity is q3. For more details about obtaining the critical values, see Appendix.Table 6
MEP with fixed ordering cost.
MethodLifetimeLead timeKu/h+u (%)MEPMEPbenchmarkMEP/MEPbenchmark∗100%PAQ-DQN40509815041.5516172.3993.011509814484.1815576.9992.980259815753.9416507.7995.431259814544.0015436.5394.22PAQ-A2C40509815116.7716178.2293.441509814223.1315423.0392.220259815813.4916502.1695.831259814896.7015858.3293.94Table7 shows the MEP of learned policies from algorithm PAQ-DQN; from the table, we can see our learned policies achieve a higher optimal rate and are closer to the upper bound, compared to Table 6. Table 8 shows the MEP comparison between the learned policies and the algorithm PAQ-DQN; from the table, we can see our learned policies achieve a higher MEP and lower MDC, which means our learned policies are working well.Table 7
MEP for the learned policies.
LifetimeLead timeKMEPpoliciesMEPbenchmarkMEPpolicies/MEPbenchmark∗100%402516013.5616525.7596.9012514911.0615540.6295.95405015421.4516197.4594.1015014771.2115583.8994.79Table 8
Comparison for PAQ-DQN.
LifetimeLead timeKMEPMEPpoliciesMDCMDCpolicies402515753.9416013.562.45012514544.0014911.0636.870405015041.5515241.451.92015014484.1814771.2197.6756.97The above discussion is mainly based on the current inventory state. Next, we will try to add the historical inventory states and action information to the state to discuss its impact on the final performance. Here, we define the new state to beSt=Xt−L,At−L,Xt−L+1,…,Xt when L>0, and when L=0, we also discuss the gradual influence of the addition of information in the state on the final performance. At the same time the dimension of the state accompanying the increase in information will also increase.Table9 shows the results after ten thousand simulations and there we add the inventory state and action from the previous period for the new state when L=0. From the table, we can see that the new state contains more information almost all performs better than the single current inventory state, which means the current decisions of the agent are influenced by not only the current inventory state but also the inventory states of the previous periods. In Table 10, MEPS1 and MEPS2, respectively, represent the inventory state and action information of the previous period and the previous two periods added to the current state. From the table, we find that the final performance did not get better and better with the continuous addition of the historical information, which also confirms that the dimensions of the state mentioned above continue to increase with the addition of information, which may have a negative impact on learning.Table 9
MEP for new state forms.
LifetimeLead timeKu/h+u (%)MEPPAQ−DQNMEPPAQ−A2CMEPnew−state30509815052.74715126.55315465.9631509813685.98713544.11513842.541250988310.2387018.2349633.66530509514937.28514996.44415604.5071509513894.83213847.47414317.4532509510470.1349989.93510299.061Table 10
MEP for new state forms.
LifetimeLead timeKu/h+u (%)MEPPAQ−DQNMEPPAQ−A2CMEPS1MEPS230509514937.28514996.44415604.50715309.8989815052.74715126.55315465.96315117.522


## 6. Conclusions

In this paper, we investigate a joint pricing and inventory control problem and obtain near-optimal pricing and replenishment policies for stochastic perishable inventory systems with positive lead time by deep reinforcement learning algorithms. Through our designed algorithms, we show that, in a perishable inventory control problem, the expected profit is maximized by adjusting the price according to the availability of inventory and the remaining lives of the items. We consider the case of no fixed ordering cost and the one involving a fixed ordering cost and find near-optimal policies for both cases. Our findings when a fixed ordering cost is involved contribute to the literature of inventory control for perishables, which has not been studied before. In this paper, we only focus on a single agent’s joint pricing and inventory control problem. However, multiple agents are usually involved in supply chains, and their interactions may have a big impact on each agent’s pricing and inventory decisions. Therefore, the study of the competition and cooperation of participants under complete and incomplete information is an interesting topic for future research.



---

*Source: 6643131-2021-01-30.xml*