Dynamic Pricing on E-commerce Platform with Deep Reinforcement Learning

1 INTRODUCTION . Dynamic pricing , often referred to revenue management as well , is to adjust prices according inventories left and demand response observed from time to time to maximize revenue . It has drawn great attentions during the past decades since the deregulation of the airline industry in the 1970s . Weatherford & Bodily ( 1992 ) and Talluri & Van Ryzin ( 2006 ) gave overviews of the research that has been done in the field of perishable-asset revenue management , which is a field that combines the areas of yield management , overbooking , and pricing . During the recent development of business , many industries have become more active in revenue management . Riding-share platforms like Uber has implemented dynamic pricing strategy , known as ‘ surge ’ pricing and Chen & Sheldon ( 2016 ) showed that it has significant impact on motivations for more driving times . Retailers like Zara has implemented systematic dynamic markdown pricing strategy . Caro & Gallien ( 2012 ) studied the clearance pricing for Zara to increase the revenue generated by each item while maintaining a large number of items to sell . Kroger is now testing electronic price tag at one store in Kentucky ( Nicas ( 2015 ) ) . Online retailers have a stronger desire for dynamic pricing strategies due to more complex operations . Amazon.com sells 356 million products ( 562 million now ) . Walmart.com sells 4.2 million products according to a 2017 estimate1 . Taobao.com , the biggest E-commerce platform in China , sells billions of products at present . Operation specialists have to set prices for these items periodically to remain competitive , which will be mission impossible when the number of items goes this high . As a result , Amazon has implemented automatic pricing systems and it is reported that Amazon.com can change prices every 15 minutes2 . Chen et al . ( 2016 ) studied the pricing strategies for Amazon.com empirically and derived significant factors for pricing . In this paper , we proposed a reinforcement learning approach to address the dynamic pricing problem for online retailers . The scenario we consider is how to dynamically price for different goods on Tmall.com , the largest business-to-consumer retailer in China , spun off from Taobao.com . There 1https : //www.scrapehero.com/how-many-products-does-walmart-com-sell-vs-amazon-com 2https : //www.whitehouse.gov/sites/default/files/docs/Big Data Report Nonembargo v2ṗdf are many difficulties for pricing on such an E-commerce platform . First , the market environment is impossible to be quantified . Revenues for the same product could change dramatically due to unpredictable fluctuation of the daily customer traffic , the change of other products ’ prices or even comments from the previous buyers . Second , it would lead to non-convergence policies if the reward function is not set properly under such complicated environment . Third , it is not applicable to apply the pricing models online directly , since a slightly inappropriate price online could quickly cause large capital loss . Therefore , the pre-training and evaluation aiming this cold start problem becomes necessary . Fourth , unlike recommendation system , it is impossible to do online A/B testing , because it is illegal to expose different prices at the same time to different customers . That is an obstacle to evaluate the performances of different pricing policies during the field experiment . To overcome these difficulties , we propose a framework for dynamic pricing with DRL to optimize long-term revenue . This paper has several contributions . First of all , we are the first to apply DRL for both discrete and continuous pricing problem on real-world E-commerce platform , rather than traditional discrete one using Q-learning ( Maestre et al . ( 2018 ) ) or within simulated environment . Second , we found that the revenue conversion rate and its difference are more suitable as the reward function rather than the revenue ( Maestre et al . ( 2018 ) , Kim et al . ( 2016 ) , Schwind & Wendt ( 2002 ) ) due to its convex nature . Third , very few assumptions are made : the demand function is not defined in advance and not necessarily time-invariant ; the revenue function is not necessarily convex or concave ; the customer behaviour could be strategic . Finally , we carry out large-scale field experiments lasting for months with thousands of SKUs products priced by our DRL models . Our field experiments prove the effectiveness of our dynamic pricing framework on E-commerce platform , which is the first work of this kind with such achievement . The remaining of this paper is organized as follows : The next section lists some related work in dynamic pricing problem . Section 3 introduces approaches we designed for dynamic pricing , where the problem is modeled as a Markov Decision Process model . Both discrete pricing action model and continuous pricing action model are proposed . In section 4 , the results from both offline and online experiments are introduced , which validate our approach of the problem . The conclusions and future work directions are summarized in section 5 . 2 LITERATURE REVIEW . Much research has been done in dynamic pricing for decades . We refer to den Boer ( 2015 ) for a comprehensive review for recent developments . It combines two research fields : ( 1 ) statistical learning , specifically applied to the problem to estimate demand and ( 2 ) price optimization . Most of previous research has focused on the cases where a functional relationship between price and demand is assumed to be known to decision makers . Cournot ( 1897 ) is acknowledged to be the first to mathematically describe the price-demand relation of products and solve the mathematical problem to achieve the optimal revenue . However , it assumed that the relationship is static over time which usually does not hold true in reality . Evans ( 1924 ) assumed that the demand is not only a function of price , but also the time-derivative of price , leading to a dynamic demand function of price over time . Kamrad et al . ( 2005 ) introduced a stochastic model to capture demand uncertainty while optimizing the prices . Gallego & Van Ryzin ( 1994 ) considered constraints like limited inventories and a finite planning horizon . In practice , it is often difficult to describe the demand beforehand . Much recent research focuses on the dynamic pricing with unknown demand function . Researchers first addressed the problem by parametric approaches . Bertsimas & Perakis ( 2006 ) assumed parametric families of demand functions to be learned over time . Farias & Van Roy ( 2010 ) proposed an approach to learn from the historical purchase data . Harrison et al . ( 2012 ) utilized Bayesian dynamic pricing policies to address demand uncertainties . However , revenue may depart from the optimal due to mis-specifying the demand family . Therefore , much recent research mainly revolves around non-parametric approaches . Besbes & Zeevi ( 2009 ) , Besbes & Zeevi ( 2015 ) , Wang et al . ( 2014 ) looked deep inside learning while earning approaches ; and Wang et al . ( 2014 ) announced to have the smallest gap so far . However , they all assumed the revenue function is strictly concave and differentiable , which could not hold true in E-commerce retail industry as shown in section 3.1 . With the development of computation , reinforcement learning ( RL ) is introduced to address dynamic problems . Kephart et al . ( 2000 ) demonstrated the possibility of using Q-learning to express anticipated future-discounted profits for possible prices , forming a so called pricebot to adjust prices in response to changing market conditions . Schwind & Wendt ( 2002 ) used Temporal Difference for Information products ’ dynamic pricing from a yield management view . Raju et al . ( 2003 ) formulated single seller and two sellers dynamic pricing problems and employ different RL algorithms in a simulated context . Kutschinski et al . ( 2003 ) used different types of asynchronous multi-agent RL methods to determine the competitive pricing strategy in the market scenario . Vengerov ( 2007 ) and Kim et al . ( 2016 ) utilized reinforcement learning to optimize prices in the energy market . Maestre et al . ( 2018 ) suggested Q-learning with neural networks approximation to maintain revenue while improving fairness in a simulated environment . All these previous works , however , are carried out in simulations with simplified market settings , where the reward defined with revenue work out well . And DNNs are only used as approximators for discrete prices . It is not the case for real-world market . 3 METHODOLOGY . We now consider how to build up decision making models for the dynamic pricing problem we discussed above . We first represent the problem as a Markov Decision Process ( MDP ) . The agent periodically changes prices of the products as its action after observing environment state . The new environment state could then be observed and the reward could also be received . Each pricing episode reaches its end if the product is out of stock . The model is pre-trained by historical sales data and previous specialists ’ pricing actions , which is also used for offline evaluation . The framework is shown in Figure 1 . 3.1 PROBLEM FORMULATION . We mainly consider two kinds of dynamic pricing applications for E-commerce platform , named markdown pricing and daily pricing in the rest of this work . These two kinds of pricing applications could cover most dynamic pricing scenarios on E-commerce platform . For both markdown pricing and daily pricing , we define n products labeled by i = 1 , 2 , ... , n individually . The prices are decided to be modified or kept at discrete time steps t = 1 , 2 , ... , T , while between these time steps the prices will be fixed referring to the model of market in Kutschinski et al . ( 2003 ) , Madhavan ( 2000 ) and O ’ hara ( 1995 ) . The distance between two time steps is define by a hyperparameter d. For markdown pricing the supply is limited , so the pricing process for certain product reaches its end if it is out of stock , while the supply in daily pricing is regarded as unlimited . The dynamic pricing process on E-commerce platform is formed as Markov Decision Process ( MDP ) referring works from Kim et al . ( 2016 ) , Vengerov ( 2007 ) , Raju et al . ( 2003 ) etc . At each time step t , the pricing agent observes si , t describing the state for product i , and takes an action ai , t . Then the agent receives the reward ri , t for that action as well as the observation for the new state si , t+1 . These four elements form the transition ( si , t , ai , t , ri , t , si , t+1 ) , which could be simplified as ( s , a , r , s′ ) . Intuitively we want a small d to make pricing actions reacting in time or archiving continuous pricing . However , precisely describing the change of the environment may need a certain time period for observing . Changing price rapidly could also break price image for the product and even cause credit issue on E-commerce platform . Since our experiments would change the prices online , we set this period carefully after discussions with professional pricing managers . In the rest of this work , pricing period d is settled as one day . Therefore , time step t also represents day t. State space . Here in our model , each product i is priced separately , with four different groups of features at time step t to describe the state si , t : price features , sales features , customer traffic features and competitiveness features . Price features contain the actual payment for this product , the discount rate , the coupons , etc . Sales features contain the sales volume , revenue , etc . Customer traffic features contain the time the page of the product i has been viewed ( PV ) , the number of unique visitors viewed the product ( UV ) , the number of buyers for product i , etc . The comments and the states of the similar products contribute to competitiveness features . The explanations for some core features are given in Table 1 ( in Appendix ) . Action space . We also define the action space for each product i separately . We use the maximum price Pi , max and the minimum price Pi , min of the product i during a certain number of periods in the history to define the upper bound and lower bound . It assumes that the pricing framework should not output a price out of this area . The pricing space could be discrete or continuous for different applications . When it is discrete , each action stands for a price range . Reward function . We compared different ways to define the immediate reward ri , t . The revenue is not suitable as we mentioned above , the customer traffic could obviously change without forewarning and then influence the revenue . Therefore , there may not be a clear and explainable relationship between price and revenue like in traditional retail industry . Defining reward function using revenue could severely mislead the agent and end up with non-convergence . But on E-commerce platform , there are links between price and revenue conversion rate , ri , t = revenuei , t/uvi , t , where uvi , t represents the number of unique visitors viewed the product i between time step t and t − 1 . In some part of this work , we also use profit conversion rate , dividing by ui , t , if we have the knowledge of the inventory cost . To prove this idea , we analyze different categories of products about the distribution of revenues and their conversion rates on different prices . Here , we demonstrate the results for over 3300 SKUs of shampoos and over 4800 SKUs of candies selling on Tmall.com , about their average revenue and revenue conversion rate in different price levels within 3 months in Figure 2 . As shown in Figure 2 that , revenue conversion rate is more convex than revenue itself . In the online experiments , using revenue conversion rate as reward function works fine in markdown pricing application when 1 ) there is a very clear and accurate stock determining the life-cycle of the pricing process ; 2 ) most of the markdown products are low-sales-volume luxuries , having low but sensitive revenue conversion rates with prices . However , in another pricing applications in this work , daily pricing for fast moving customer goods ( FMCGs ) , supply is adequate and the stock could be regarded as unlimited . A clear end point for each pricing process could hardly be defined . More importantly , the average sales volumes for these FMCGs are lot higher than the luxuries in markdown pricing . In this case , despite we find the relationship between prices and average revenue conversion rate through a certain period , this relationship may not keep steady at different time in this period . Another investigation for two different kinds of products about the trends of their price level with revenue conversion rate for 90 days ( shown in Figure 3 ) reveals this phenomena . We could see from Figure 3 ( a ) that , for these luxuries , when the price drops , the revenue conversion rate goes up , especially around day 25 , 45 , 65 and 85 . However , for FMCGs in ( b ) , there is no such relationship , but the revenue conversion rate fluctuates with an individual frequency around a week . Therefore , with the revenue conversion rate used as reward function for these FMCGs , the convergence of the model could not be guaranteed . Comparing two phenomenon in Figure 2 and Figure 3 , we define a different reward function , using the difference of the revenue conversion rates ( DRCR ) in Eq. ( 1 ) . ri , t = revenuei , t uvi , t − revenuei , t−τ uvi , t−τ ( 1 ) where τ represents the length of the time to compare the revenue conversion rate . The idea behind this definition is that , we hope to give the agent an positive signal , if it successfully raise the revenue conversion rate with its pricing action . This definition of reward function solves the convergence problem in FMCGs daily sales , while could also work out fine in markdown pricing .

Summary: This paper presents a reinforcement learning approach to dynamically set the price of items on sale on an e-commerce website, based on a state description consisting of several kinds of features: price, sales, customer traffic and competitiveness. The actions (possible prices) are constrained to lie in item-specific lower and upper bounds. Against a proposed method relying a continuous action space (an implementation of the Deep Deterministic Policy Gradient of Lillicrap et al., 2015), alternatives relying on multi-armed bandits and Deep Q Networks are evaluated. Experimental results from an online deployment are presented.