# Outsmarting Human Design in Airline Revenue Management
## 5 Experiments  
We separate our experiments into two distinct scenarios. In the first one, we compare the RL agent to the heuristic method defined in Equation (1) while estimating a single demand-model parameter (price sensitivity \(\phi\)). The heuristic method can be adapted to the single-leg problem under the assumption of unconstrained capacity (i.e., infinite inventory capacity) [33]. Even though the unconstrained capacity is an unrealistic assumption for most real-world scenarios in airline RMs, this experiment provides a baseline to measure the effectiveness of the RL agent when solving the EWL in the single-leg problem. In the second, we compare the solution obtained by RL with the standard RMS (which behaves as certainty equivalent pricing, as described in Section 2) when capacity is constrained, and we assume the system must estimate two model parameters from historical booking data (arrival rate \(\
u\), and price sensitivity \(\phi\)). This scenario aims to demonstrate that our method can learn how to solve more complex settings (such as controlling the uncertainty over two model parameters), and that it can also learn how to optimize pricing under capacity constraints (which is not possible with the work developed in [33]). Figure 4: The artificial neural network architecture consists of two separate networks. The critic network approximates the value function, and the actor network approximates the policy function. The long short-term memory (LSTM) encoder takes the remaining capacity inputs \(c^{[i]}\), and the LSTM decoder takes the remaining time inputs \(\tau^{[i]}\), and it outputs the probability of selecting each fare in the fare structure \(\tau^{[i]}\) for each active flight. Throughout the experiments, we assume that each flight has the capacity \(C=50\) (representing, for example, the number of seats in the business cabin), the fare structure has 10 price points \(\mathcal{F}=\{\$50,\$90,\ldots,\$230\}\), the booking horizon has \(\mathrm{T}=22\) time steps, the demand follows the exponential model, and the historical database keeps the booking data for the most recent T flights. Furthermore, we refer to the price sensitivity in terms of the fare ratio of the lowest fare at which the purchase probability is 50% [57], abbreviated as _frat5_, which is defined as  
\[F_{5}\doteq\ln(2)/\phi+1.\]  
The frat5 greatly simplifies interpretation, because it has a linear relationship to revenue-maximizing fare under the assumption of the exponential demand model. As illustrated in Figure 5, for the RL training set up we use the proximal policy optimization algorithm (PPO) [48] because of its simplicity and stability, and we use RLLib [58] to distribute training computation in a cluster with 4 GPUs and 24 CPUs. The training hyperparameters set for the PPO algorithm for the two investigated scenarios can be found in Table 2. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Experiment** & **Hyperparameter** & **Value** \\ \hline \multirow{4}{*}{unconstrained capacity} & train batch size & \(102\frac{\text{eps.}}{\text{CPU}}\times 440\frac{\text{time steps}}{\text{ep.}}\times 23\) CPUs \(=\) 1,032,240 \\  & learning rate (\(\alpha\)) & \(3\times 10^{-5}\) \\  & entropy coefficient & 0.005 \\  & (see [48]) & \\  & value function clip & 30 \\  & (see [48]) & \\  & eligibility trace & 0.15 \\  & (see [59]) & \\ \hline \multirow{4}{*}{constrained capacity} & train batch size & \(175\frac{\text{eps.}}{\text{CPU}}\times 440\frac{\text{time steps}}{\text{ep.}}\times 23\) CPUs \(=\) 1,771,000 \\  & learning rate (\(\alpha\)) & \(3\times 10^{-6}\) \\ \cline{1-1}  & entropy coefficient & 0.015 \\ \cline{1-1}  & value function clip & 30 \\ \cline{1-1}  & eligibility trace & 0.1 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Training hyperparameters. “ep.” abbreviates episode. The RLLib implementation of PPO dedicates one CPU to the master thread, which is not used for generating experience. Figure 5: The proximal policy optimization algorithm training architecture for earning while learning.