# Outsmarting Human Design in Airline Revenue Management
## 6 Discussion  
Perhaps one of the most important keys to the success of RL was the simplification the dual optimization aspect of the EWL problem into a single optimization dimension, which is the revenue maximization. Instead of describing the EWL problem as an explicit requirement of balancing revenue maximization and model learning common to heuristic methods [2; 33; 33], we ask the RL agent to maximize only revenue, and RL has the flexibility to trade-off between earning and learning as it finds best suitable. In Figure 11, we show how RL trades earning and learning throughout its training. This figure is computed according the same experimental settings which the RMS must use to estimate both arrival rate and price sensitivity, and the true demand behavior parameters are sampled uniformly in the evaluation interval. Prior to the start of training, RL starts following a near-random policy at the bottom right, and, as training progresses, RL policy improves its revenue performance (earning) at the expense of the demand behavior parameter's accuracy (learning). At some point in training, RL displays the same revenue performance of RMS while presenting a better estimation of the demand model parameters. As training continues, RL is able to further improve revenue by sacrificing the demand model quality,  
Figure 10: Comparison between revenue management system (RMS) (**left**) and reinforcement learning (RL) (**right**) roll out policies for \(\
u_{s}=70/22\), \(F_{5}^{*}=2.9\). The reinforcement learning policy performs better (89.9%) than the revenue management system policy (80.6%) with respect to the revenue performance but has worse mean squared errors for both demand model parameters. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Experiment**} & \multirow{2}{*}{**Method**} & **Average** & **Price Sensitivity** & **Arrival Rate** \\  & & **Normalized** & **MSE** & **MSE** \\ \hline \multirow{3}{*}{Unconstrained capacity} & RMS (CEP) & 0.702 \(\pm\) 0.007 & 0.0401 \(\pm\) 0.0026 & — \\  & heuristic [33] & 0.783 \(\pm\) 0.003 & 0.0148 \(\pm\) 0.0008 & — \\  & RL & 0.868 \(\pm\) 0.002 & 0.0109 \(\pm\) 0.0005 & — \\ \hline \multirow{3}{*}{Constrained capacity} & RMS & 0.862 \(\pm\) 0.003 & 0.0072 \(\pm\) 0.0002 & 0.162 \(\pm\) 0.003 \\  & RL & 0.912 \(\pm\) 0.002 & 0.0090 \(\pm\) 0.0003 & 0.219 \(\pm\) 0.004 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Summary of experimental results. ultimately converging towards the point of equilibrium represented by the dot at the end of its trajectory, where it cannot further improve revenue. Since controlling model uncertainty is required to succeed in the task, we leave the mission of identifying _when_ and _how much_ price experimentation should be conducted to the agent. Such an approach has strong connections to the \"reward is enough\" hypothesis [13], which states that any associate abilities of intelligence (e.g., social intelligence, perception, knowledge representation, planning, etc.) can be understood as sub-serving the maximization of reward (i.e., the measure of success in the task). In our case, we claim that controlling the quality of the estimated demand model is the associated ability needed by the agent to achieve the maximization of reward (or revenue). We trust RL to find out the solution to all the complexities of the EWL problem from nothing more than the standard RMS raw inputs, without any human guidance. Even though we did not investigate how our method behaves in the face of market shocks, we believe that the adaptation should be straightforward. In principle, to train the agent to react to market shocks, we can simply introduce simulated examples of market shocks by making the true demand behavior parameters a function of time \(\psi_{*}(t)\) in the inner loop of Algorithm 1.