# Outsmarting Human Design in Airline Revenue Management
## 4 Revisiting Earning While Learning through Reinforcement Learning  
To solve the single-leg problem, the standard approach followed by legacy RMSs breaks down the problem of optimizing T active flights into T smaller optimization problems, each flight being optimized with DP as described in Section 3, by making use of Equation (2), as illustrated in Figure 3 (left). This approach would be correct if each flight was truly _independent_ of the others. However, the data collected for each flight are aggregated and used for model calibration, which in turn is used for optimization in future time steps. Thus, the pricing decision of each flight has longstanding consequences for the quality of future model calibration that goes beyond the flight's departure. Furthermore, each price decision contributes to the quality of the estimation of future demand models, making each decision a small part of a whole, suggesting that some collaboration across flights may be needed. In fact, the policy collecting data in legacy RMSs is rather a result of a set of T uncoordinated policies, each seeking to maximize its own revenue independently. To address this problem, we want to give the agent the ability to specify a unified policy that can coordinate pricing decisions across flights and that maximizes revenue beyond individual flights. Therefore, we propose to compute a policy that outputs the price decisions across the active flights altogether as illustrated in Figure 3 (right). We denote the \(i\)-th active flight on the selling horizon with superscript \([i]\), such that \(S_{i}^{[i]}\) implies the inventory state at time step \(t\) for the \(i\)-th active flight on the selling horizon. The agent's action is a tuple containing the selected rare for each one of the T active flights, i.e., \(A_{t}=(A_{t}^{[0]},\dots,A_{t}^{[\text{T}-1]})\). Similarly, the observation space is defined as the tuple of the inventory state for each one of the T active flights and the current estimated model parameters, i.e., \(S_{t}=(S_{t}^{[0]},\dots,S_{t}^{[\text{T}-1]},\psi)\). A natural consequence of this choice is that we need to redefine the reward signal as the sum of the immediate revenue obtained for  
Figure 3: Comparison between episodic and continuing formulations for revenue maximization. each one of the active flights, i.e., \(R_{t}=\sum_{i=0}^{\mathrm{T}-1}r(S_{t}^{[t]},A_{t}^{[t]})\), addressing the second point of maximizing the combined revenue across all flights. Our goal is to find the pricing policy that maximizes the combined revenue. Note that the dimensionality of the problem makes it impossible to solve it with exact methods such as DP. The cardinality of the action space alone is \(|\mathcal{A}|=|\mathcal{F}|^{\mathrm{T}}\), which makes it impractical even for toy problems. The observation space has a discrete component with size \((\mathrm{T}\cdot\mathrm{C})^{\mathrm{T}}\) and a continuous component representing the estimated model parameters \(\psi\). The transition probabilities, which we enumerate in Section 3.1, is now one set too large (\(|\mathcal{S}|\cdot|\mathcal{A}|\)) to be computed exactly. These characteristics make this problem very appealing to address with RL, because RL approximates the policy and value functions, enabling us to represent large state-action spaces, and it uses a simple and cheap sample model to perform trajectory sampling, in contrast to the expensive computation of the transition probabilities required by DP. Among the many RL methods in literature, algorithms that use policy parameterization, such as actor-critic methods, are well suited to large action spaces [51]. In Algorithm 1, we present the training pseudocode of the actor-critic framework to the EWL problem. We compute a parameterized policy \(\pi(a|s;\theta)\) and a parameterized value function \(v(s;w)\) as usual (see Section 3.2).