# Outsmarting Human Design in Airline Revenue Management
## 5 Experiments
### Estimating Only the Price Sensitivity under Unconstrained Capacity
Even though the capacity is finite, \(C=50\), given the very low arrival rate \(\
u_{*}=4/22\), it is extremely unlikely that the flight's capacity will ever exhaust even if the lowest fare (the fare for which purchase probability is one) \(f_{0}=\$50\) is selected at all time steps (\(\Pr\{50\,\text{bookings}\mid A_{0},\ldots,A_{\text{T}-1}=f_{0}\}<10^{-16}\)). Therefore, from the optimization perspective, the capacity is practically infinite, allowing us to implement the heuristic method as presented in [33]. Moreover, we perform model calibration at every time step, and we evaluate both the heuristic method from [33] and RL within the interval \(F_{5}\in[2.1,3.8]\). In this interval, the revenue-maximizing fare if the true demand behavior were known covers almost the entirety of the fare structure. In theory, the system does not know anything about the evaluation interval, and any positive value of the price sensitivity could be assumed. However, to avoid extreme evaluations for the price sensitivity parameter, we limit its estimates to be within the range \(F_{5}\in[1.5,4.3]\) (this parameter clipping applies to all methods: RL, RMS, and heuristic). Safety mechanisms such as this one are often present in real-world systems. In Figure 6, we illustrate the relationship of these two intervals with the revenue-maximizing policy when the true price sensitivity of the demand is known by the system at all time steps.  
Algorithm 1 requires two inputs. The first input is the value that the model parameters may assume. Given our settings, we have two natural choices: we can either choose to train under the evaluation interval or under the \"clipped\" interval. We choose the clipped interval (\(F_{5}\in[1.5,4.3]\)), because the two most extreme price points are never optimal in the evaluation interval, and RL could easily learn how to exploit this fact. For example, if the estimation of the price sensitivity parameter is too unreliable to be useful, RL could still learn that the most extreme price points should never be chosen (as illustrated in Figure 6), whereas the heuristic methods would require to be informed of this fact explicitly. One could argue that this is an advantage, but we believe that the comparison is fairer between heuristic and RL if the agent cannot know the true range of the evaluation interval during training. With respect to the second input, the episode horizon, we can choose any arbitrary value larger than the booking horizon \(\text{T}=22\). If this value is too small, the agent might never actually observe the long-term consequences of its choices. We find that \(H=20\times\text{T}=440\) time steps is appropriate for our goals.  
We focus our analysis on the revenue performance, represented by the average collected revenue normalized with respect to the revenue-maximizing policy (which knows  
Figure 6: An illustration of the clipped and evaluation intervals and their corresponding revenue-maximizing policies.  
the true demand price sensitivity at every time step) and the random policy (that selects fares from a uniformly random distribution). Furthermore, we also look at the mean square error (MSE) of the estimated price sensitivity (representing the demand model quality), which is given by  
\[MSE\doteq\frac{1}{n}\sum_{i=0}^{n-1}(\phi_{i}-\phi_{*})^{2},\]  
where \(\phi_{i}\) is a sample estimation of the price sensitivity. We perform each evaluation run following the same training procedure from in Algorithm 1, that is, first we set the true demand behavior parameters, then we warm up the historical database using the random policy, and finally we roll out each method's policy for 440 steps (the computation of the average reward and the update of the neural network weights are not performed during the evaluation). In contrast to the training algorithm, we discard the first \(3\times\mathrm{T}=66\) time steps of data, because we want to compare each method in its stable regime.  
We separate the analysis into two distinct parts. In the first part, we compare how each method behaves within the evaluation interval, then we perform a more detailed analysis of the final pricing policy obtained for each method.  
In Figure 7, we set the true frat5 parameter to the fixed values of the evaluation interval, and, for each frat5, we compute the average revenue performance and the price sensitivity MSE for 3565 independent evaluation runs. For the heuristic method, we choose a constant value for the trade-off parameter \(\eta=2197\) (see Equation (1)), which is the value that maximizes the expected revenue for the evaluation interval. In Figure 7 (left), we observe that RL outperforms both RMS and the heuristic method in the entire interval, except for \(F_{S}^{*}=3.8\). The reason why RL does not perform as well as the heuristic method for \(F_{S}^{*}=3.8\) will be explained in the second experiment of this section, when we illustrate the final policy obtained for both methods. In Figure 7 (right), we plot the MSE over the estimation of the price sensitivity. We see that all methods have larger values for price sensitivity MSE for lower values of frat5, which decreases as the frat5 increases. The same effect is observed and explained in [33]. In short, this is because the true arrival rate parameter \(\
u_{*}\) is known by the system. As the frat5 increases, so does the optimal fare, and the higher the selected price is, the more information is obtained about the customers' price sensitivity. Moreover, we see that RL also produces lower price sensitivity MSE than the RMS and the heuristic method. The RL agent pricing policy displays a better estimation of the demand price sensitivity, and it also generates more revenue on average than the heuristic method and RMS.  
Figure 7: The comparison between the average performance of the revenue management system (RMS), the heuristic method from [33] and reinforcement learning (RL) (99% confidence level). The system’s performance is better the higher the average normalized revenue is (the best theoretical response that requires the perfect knowledge of the demand parameters of all time steps is normalized to be one). On the right chart, we show the estimation mean squared error of demand model, which indicates that smaller errors are better.  
In Figure 8, we show the pricing policy obtained through Monte-Carlo roll-outs for both heuristic and RL methods. When true frat5 \(F_{5}^{*}=2.71\) (Figure 8a), RL shows a large performance advantage with respect to the heuristic method (see Figure 7) and also a smaller price sensitivity MSE. The reason is clear when comparing the two policies. RL tends to price closer to the true optimal fare, that is, it prices fares $110, $130, and $150 more frequently (83.6%) than the heuristic method (69.9%).