# Outsmarting Human Design in Airline Revenue Management
## 4 Revisiting Earning While Learning through Reinforcement Learning
The training environment uses a sample model, where each active flight interacts with a simulated demand following the ordinary assumptions (Poisson arrivals, negative exponential purchase probability), and the estimation of the demand model parameters is also performed also as usual [33; 43]. Perhaps the only difference from the standard methods is how the total return is computed. When we redefined the state-action space and the reward signal to contain the aggregation of the state and rewards for all active flights, the \"episodic\" concepts of the starting and terminal states from Section 3.1 disappeared with it. At each time step, the RL agent observes the sequence of the inventory states and the latest model parameters, and it selects the fares for each active flight. This continues from any particular starting point without termination. The continuing nature of the task suggests the use of the average reward defined in Equation (5). ```
Input: The training range \(\psi_{*}\in[\psi_{\min},\psi_{\max}]\) and the episode horizon \(H\gg\mathrm{T}\)
1 Loop forever
2 initialize an empty training batch \(\mathcal{B}\);
3 sample \(\psi_{*}\) uniformly within training range;
4 warmup historical booking database according to \(d(f;\psi_{*})\) and while following the random policy;
5 for\(t=1,\ldots,H-1\)do
6 estimate \(\hat{\psi}\) from historical booking data;
7 set \(S_{t}=(S_{t}^{[0]},\ldots,S_{t}^{[\mathrm{T}-1]},\hat{\psi})\);
8 select \(A_{t}=(A_{t}^{[0]},\ldots,A_{t}^{[\mathrm{T}-1]})\sim\pi(a|S_{t};\theta)\);
9 simulate demand response according to \(d(f=A_{t};\psi_{*})\);
10 observe \(S_{t+1}\) and \(R_{t+1}=\sum_{i=0}^{\mathrm{T}-1}r(S_{t}^{[i]},A_{t}^{[i]})\);
11 store \(S_{t}\), \(A_{t}\), \(R_{t+1},S_{t+1}\) to \(\mathcal{B}\);
12
13 end for
14
15 recompute the rewards \(R_{i}^{\prime}=R_{i}-r(\pi)\,\forall R_{i}\in\mathcal{B}\), see Equation (6);
16 update \(\theta\) and \(w\) according to Equations (3) and (4), respectively, using the collected experience \(\mathcal{B}\);
17
18 end for
```  
**Algorithm 1**Actor-critic for EWL. The way the EWL problem is formulated for the RL agent is related to multitask reinforcement learning (each set of true demand parameters define a new transition probabilities and thus to a new task) and partial observability (the true demand parameters are unknown to the system). The RL agent is not informed which task it is solving, and it needs to discover which task it is solving along the way through the unreliable estimated demand parameters. In fact, we borrow some ideas from these fields, such as the use of universal function approximators (inputs of the value function) [52], and the use of a (supervised) state-estimator function (which corresponds to the forecasting module that computes the estimated model parameters) [53]. With regard to the modeling of the policy and the value functions in Algorithm 1, we suggest the use of artificial neural networks. We recognize that there may be many possible designs for the artificial neural network (ANN), and finding the best one is not the focus of our work. We find that the architecture illustrated in Figure 4, which uses a long short-term memory (LSTM) [54], the Luong-style attention mechanism [55] following the encoder-decoder architecture [56], delivers stable training and good performance results. Perhaps the most important aspect of our design is the use of the LSTM decoder in the actor network. The number of output units grows linearly with respect to the number of parallel active flights (i.e., for each active flight, the agent must compute the probability of selecting each fare in the fare structure), and it can quickly cause the ANN to have too many outputs to learn. By using an LSTM decoder, the ANN can share the learned parameters between the outputs of each active flight, allowing a more compact and efficient representation.