Year: 2018

Abstract:
Representation learning becomes especially important for complex systems with multimodal data sources such as cameras or sensors. Recent advances in reinforcement learning and optimal control make it possible to design control algorithms on these latent representations, but the field still lacks a large-scale standard dataset for unified comparison. In this work, we present a large-scale dataset and evaluation framework for representation learning for the complex task of landing an airplane. We implement and compare several approaches to representation learning on this dataset in terms of the quality of simple supervised learning tasks and disentanglement scores. The resulting representations can be used for further tasks such as anomaly detection, optimal control, model-based reinforcement learning, and other applications.

heading: 1 INTRODUCTION
text: In order to act in real world scenarios and control a complex system such as an airplane, car, or industrial facility, an automated agent needs to process complex high-dimensional data coming from different domains: video feeds from cameras, LIDAR sensors on a car, altitude and speed sensors on an airplane, sensors related to the internal state of the system, and so on. An important problem in this regard would be to map this rich stream of multimodal information into a lower-dimensional space that compresses all modalities into a uniform latent representation (embedding); the agent could then use this embedding to learn or otherwise construct control algorithms. Thus, representation learning lies at the heart of optimal control for complex systems with multimodal features.
Over the last decade, deep neural networks have surpassed other methods in processing nearly all modalities of high-dimensional unstructured data, including images, natural language texts, sounds, and time series. One of the most important properties of neural networks that has made the deep learning revolution possible is their ability to extract meaningful low-dimensional representations of raw unstructured input data. Representation learning with deep neural networks is a large and wellestablished area of research (Kingma & Welling, 2013; Hinton & Salakhutdinov, 2006; Radford et al., 2015). Latent features learned by deep neural networks find applications in various domains, including reinforcement learning for complex systems (Zhang et al., 2018; Watter et al., 2015; van Hoof et al., 2016). In these works, the authors often propose special techniques and architectures to design the latent space in different ways suitable for further use: make the latent space locally linear, capture the dynamics, and so on. Designing such feature extractors is a complex task, usually done by hand. Moreover, it is hard to compare different architectures, to a large extent because it is far from obvious how to measure the quality of resulting representations. One reason for that is that the best metric for the quality of learned representations would be the quality of the final task in question, which is hard to obtain in reinforcement learning due to the sheer scale of this final task.
A common way to deal with such problems is to use a unified dataset and a unified set of benchmarks, such as, for example, ImageNet and the ILSVRC benchmark in computer vision (Russakovsky et al., 2015). Such unified datasets might also prove useful for transfer learning tasks. However, the field of reinforcement learning for complex systems is yet to agree on a common representative large-scale dataset. In this work, we present such a multimodal dataset for the representation learning problem together with a unified benchmark framework for feature extractors suitable for a comprehensive comparative evaluation of different feature extractors. This dataset has been gathered with largescale computer simulations based on the X-Plane simulator and consists of data streams from various
sensors along with images taken from the frontal camera of the plane. We propose and implement several different metrics for comparison between extracted features. We also survey, implement, and compare different neural architectures for learning multimodal state representations.
The paper is organized as follows. In Section 2 we survey related work on representation learning and evaluation of representations. Section 3 presents the X-Plane dataset and explains its characteristic features. In Section 4, we present the various representation learning models that we have implemented and compared on this dataset. Section 5 shows the evaluation metrics and presents a large-scale comparison across different models, and Section 6 concludes the paper.

heading: 2 RELATED WORK
text: Modern agent control methods commonly use techniques based on deep learning as feature extractors to deal with complex multimodal data, either explicitly (Zhang et al., 2018; Watter et al., 2015) or implicitly (Mnih et al., 2015). Explicit techniques separate representations of the learning environment from the agents operating in this enviroment; the environment representation can be learned either independently (Zhang et al., 2018) or in a common end-to-end architecture (Parisi et al., 2017). There are several major approaches to constructing such models (Lesort et al., 2018): (1) autoencoders that reconstruct the input observation data, producing the latent representation between encoder and decoder; (2) forward models that predict the next state either in the latent space or in the raw data, basing the prediction on the current latent representation; (3) inverse models that use two states to predict the actions between them; this approach can be combined with forward models; (4) models with prior knowledge of the system that impose additional constraints on the latent space according to some fundamental system properties such as causality, temporal continuity, or some more specific knowledge about the system (Jonschkowski & Brock, 2015); (5) multimodal temporal fusion models such as the Memory Fusion Network (Zadeh et al., 2018).
If there is no additional data labeling available such as, e.g., a list of factors, the most direct way to measure the quality of representations would be by evaluating the final quality of solving the main task that a model can achieve based on this representation. However, in real world cases, in particular in reinforcement learning, the main task is often hard to solve and unstable to train, so it cannot be consistently used to evaluate latent representations.
Therefore, many indirect ways have been proposed to measure the quality of representations that introduce proxies that can be expected to lead to better solutions of the final control problem. The most common indirect approaches include (see also a comprehensive survey by Lesort et al. (2018)): (1) task performance, the most intuitive metric, where representation quality is measured by the quality of performing some other relatively simple task, e.g., by predicting some available target variables with simple models that take the latent representation as input (Higgins et al., 2016; van Hoof et al., 2016); (2) KNN-MSE, proposed by Lesort et al. (2017), measures the degree of preservation of the same neighbors between the latent space and the ground truth: KNN-MSE(I) = 1k ∑ I′∈KNN(I,k) ‖φ(I)− φ(I ′)‖ , where I is the initial raw input, I ′ is a neighbour of I , and φ is the feature extractor; KNN-MSE is a good metric in situations where the distance in the original input space is well-defined but becomes hard to apply for highly variable multimodal data; (3) a similar approach with humans in the loop is to evaluate whether similar input states according to human evaluation do indeed map to close representations in the latent space (Sermanet et al., 2018); however, for complex multimodal inputs this is again inapplicable since a human would not have an intuitive notion of similarity between two sets of several hundred sensor readings; (4) disentanglement scores (Eastwood & Williams, 2018; Higgins et al., 2016) measure the disentanglement (mutual independence) of extracted features; if there are some known generative factors, these metrics assess whether individual elements of the latent representation capture individual generative factors independently; we will consider such metrics in detail in Section 5.1.
As a representative multimodal dataset for a complex system, we have used the X-Plane flight simulator (Laminar Research, 2018), well-known for its faithful simulation of all systems of an aircraft. It has already been used to solve optimal control problems for aircraft; e.g., Bittar et al. (2014) develop separate control laws for stable and maneuvering flight, and Garcia & Barnes (2009) use X-Plane to simulate a system of several unmanned aerial vehicles. However, to the best of our knowledge this is the first attempt to produce a large-scale dataset for representation learning from X-Plane or any other flight simulator.

heading: 3 THE X-PLANE DATASET
text: We present the X-Plane Dataset (Dataset, 2018; Code, 2018) as a benchmark for evaluating the quality of state representations (embeddings), where the main problem is to encode the state of some complex system, learn a mapping from high-dimensional multimodal data to a low-dimensional latent representation. We have used the X-Plane simulation environment because it is extremely accurate and can produce a lot of useful and different sensor data. In total, we have recorded 8011 landings, with mean duration of a landing of 115 seconds. For every landing, the dataset contains the readings of 1090 sensors arranged in time series recorded with with a frequency of 5 frames per second, together with the corresponding image taken from the camera located at the front of the airplane; the images are recorded at the same frequency. Table 1 summarizes the various groups of sensors recorded in the dataset, showing the number of sensors in a group together with a brief description. The total dataset size is 93GB uncompressed; it contains sensor readings and about 7M 256 × 256 images, joined into per-flight videos for better compression. To ensure diversity in the dataset, we recorded landings with random perturbations and in different environments: with different airports and runways, time of day, weather conditions etc. We have used at most 12 perturbations (abrupt changes in the environment such as, e.g., wind gusts, malfunction of various systems in the aircraft, and so on) applied during each flight; Table 2 summarizes various failures and perturbations. We have used a custom control unit based on the Boeing 737 guide (Brady, 2014) in order to make automatic landings stable.
During dataset collection, for every landing we chose a random airport and a random runway and spawned a plane in a random position, usually at a distance of 3-5 miles from the runway. There are, in total, 114 different runways in 70 airports used in the dataset. Then we applied different landing conditions—landing speed, flaps position, time of day, visibility, precipitation and so on— and enabled the autopilot that lowered the plane on the glide slope path. To achieve a better landing, we disable the autopilot before touchdown, increase the pitch of the plane, and after touchdown we decrease the pitch and enable reverse thrust. At the end of the flight, there are three possible situations: successful landing, aircraft crash, or time limit reached. We set a strict time limit of 160 seconds for every flight. We have also done an airport-stratified split of the dataset into 4 parts: training set for feature extractors, validation set used for early stopping, training set for benchmark supervised learning models, and test set for scoring the benchmark models. We have made the dataset available at (Dataset, 2018).

heading: 4 MODELS
text: For experimental evaluation of different representation learning models, we used a wide variety of different models, from basic autoencoders to dynamic actions-aware encoders. In this section, we present these models, from simple to more complex ones. Our models were constructed with four core building blocks: standard recurrent LSTM cell (Hochreiter & Schmidhuber, 1997), onedimensional convolutions (Conv1D), ConvLSTM (Shi et al., 2015), and an attention cell (Bahdanau et al., 2014). The attention cell A(x) operates as follows (see Fig. 2):
vi = ReLU(Wxi + b), s = softmax(c >V + b), a = Vs, Out = (x,a),
where x is the input, W is a d × N matrix for embedding size d and input dimension N , V is the matrix with columns vi, c is a vector of size d, s are attention weights. We considered the following specific models (Table 4 lists all models with brief descriptions and numerical results).
One-dimensional autoencoders. A simple autoencoder that trains to reconstruct a given set of timesteps (Fig. 2a). We used four types of autoencoders (AE) in the comparison: autoencoder with 1-layer LSTM cells for encoder and decoder, with 2-layer LSTM cells, with a 6-layer convolutional autoencoder with kernel size 7. In LSTM autoencoders, after the encoder block we used simple averaging of vectors from all timestamps in the time series, using the result as embedding. In Conv1D, after the encoder block we used max-pooling to get a single vector from the time series; in the decoder, it is copied the necessary number of times and fed to another Conv1D block. In our experiments, we used 6 layers of Conv1D with kernel size 7.
Image Autoencoders. We trained two image autoencoders for reconstructing individual pictures from the flight time series. The first autoencoder uses a PCA encoder and a PCA decoder. To obtain
the features from a flight sample, we apply the encoder for each image in the sequence and compute the average of all resulting feature vectors. The second architecture is a bit more complicated: its encoder contains the ResNet34 (He et al., 2016) architecture with an additional fully connected layer whose size varies depending on the embedding size d, and whose decoder contains fifteen two-dimensional convolutional layers. In contrast to the PCA model, instead of averaging the feature vectors for each flight we used the standard deviation (this class of models worked better empirically in our experiments).
Autoregression Models. These models are trained to predict a state at time t + k given some subset of states up to time t. In our experiments, we trained autoregression models to predict with k = 1 (next time step) and k = 30. In this task, as an encoder we have compared LSTM, Conv1D, ConvLSTM, and attention-based architectures, with a simple linear layer after each.
Context Models. These models employ ideas similar to the word2vec CBOW model (Mikolov et al., 2013). Given a short sequence of timesteps, the model trains to predict several timesteps before and after the sequence; in our experiments, we used window size of 10 before and 10 after given states. To train context models, we have used LSTM, attention cells, and Conv1D blocks. After an encoder block, we repeated an embedding 2C times, where C is the context size, and then applied a simple linear layer to each vector with different weights for each of the C timestamps.
Multimodal Temporal Encoder. The multimodal temporal encoder (MTE), introduced by Yang et al. (2017), is a state of the art model intended to fuse inputs from different modalities (see Fig. 3). MTE uses an LSTM unit for each modality and shares hidden states across these units. By doing this, MTE forces the model to learn a fused representation across modalities; formally, it adds a correlation loss that computes the correlation between projections of differ-
ent modalities Lcorr(H1t , H 2 t ) =
∑M i=1(h 1 ti−H1t )(h
2 ti−H2t )√∑M
i=1(h 1 ti−H1t )2 ∑M i=1(h 2 ti−H2t )2 . The final loss is calculated as
L = ∑M
i=1 Lrecon,i − λLcorr, where M is the batch size, h j t is the representation of modality j
at timestep t, Hjt = {h j ti}Mi=1, H
j t = 1 M ∑M i h j ti, λ is a constant hyperparameter, and ht is the
final embedding (see Fig. 3). The sensors data is encoded with a three-layer fully connected neural network with dropout and ReLU activations, and a pretrained ResNet18 architecture for the images.
The Dynamics Model. This approach, proposed by Zhang et al. (2018), introduces a way to decouple the training process (for a reinforcement learning task) into learning the dynamics model and the reward model. The dynamics model (shown in Fig. 4) is trained using a combination of four loss functions: reconstruction loss Lt,recon(θenc, θdec) = (st − ŝt)2, state loss Lt,state(θfor, θdec) = (st+1 − ŝt+1)2, forward loss Lt,for(θfor, θenc) = (zt+1 − ẑt+1)2, and inverse loss (with a trainable LSTM unit) Lt,inv(θinv) = (at − ât)2. The final loss is computed as L(θdynamics) = ∑T t=0(λdec(Lt,recon + Lt,state) + λforLt,for + λinvLt,inv), where ẑt+1, ht = ffor(zt, at, ht−1; θfor), ât = finv(zt, zt+1; θinv), ŝt+1 = fdec(ẑt+1; θdec), st is the state at time t, at is the action at time t, ht is the hidden state, and zt is the latent representation at time t; λdec, λfor, and λinv are (constant) hyperparameters. If the dynamics change, we need only to re-train the LSTM unit and can keep the encoder and decoder unchanged, with the assumption that already learned representation contains all the necessary information about the new dynamics. Since the original approach was presented only for inputs with a single modality, we expanded this idea for the multimodal case. To encode data from the sensors, we used a three-layer fully connected neural network with dropouts and ReLU activations. For the images, we used a pretrained ResNet18 architecture. After that, a linear layer was used to concatenate outputs of encoders for each modality into the embeddings, and the final embedding results by averaging over the time steps.

heading: 5 EXPERIMENTAL EVALUATION

heading: 5.1 EVALUATION FRAMEWORK
text: As part of the dataset package, we have implemented and made available the framework which is designed to make a comprehensive evaluation of learned representation based on a number of fixed predefined tests. The main idea behind our framework is to combine the two main approaches to measuring representation quality. First, we measure quality of representations by using them as features for a number of simple tasks. This approach was used, in particular, in the Black Box Learning Challenge (Goodfellow et al., 2013) and Unsupervised and Transfer Learning Challenge (Guyon et al., 2011), whose main objective was to learn a good representation from rich unlabeled data, and representations were evaluated on supervised learning tasks that were not known to the participants.
The second approach is to evaluate the disentanglement in representations. For this we use a QEDR (Quantitative Evaluation of Disentangled Representations) framework proposed by Eastwood & Williams (2018). The idea is that the ideal representation of data is a vector of separated and independent generative factors for the data (perhaps scaled and permuted). The matrix R, where Rij is the relative importance of code (representation) variable ci in predicting generative factor zj , is used to compute three evaluation metrics for the quality of a representation. Disentanglement is a measure of the degree to which a representation factorizes the factors of variation in original data; for a code variable ci it is defined as Di = 1 −H(Pi.) where H(Pi.) is the entropy of the pseudodistribution Pi.; Pij = Rij/ ∑ k Rik denotes the “probability” of ci being important for predicting
generative factor zj , and total disentanglement is computed as the weighted average ∑
i ρiDi, where ρi = ∑ j Rij/ ∑ kj Rkj is the relative code variable importance. Completeness is a measure of the degree to which a factor of variation in the original data is captured by a single code variable; for a factor zj it is defined asCj = 1−H(P̃.j) whereH(P̃.j) is the entropy of the pseudo-distribution P̃.j ; total completeness is the average of Cj . Informativeness measures the amount of information that a representation captures about the underlying factors of variation; following Eastwood & Williams (2018), we use normalised root-mean-square (NRMSE) as the informativeness metric. We use failure scores as generative factors since we know the ground truth for them and they are mutually independent. We define Rij = |Wij |, where W is the weight matrix of lasso regression learned on the representation vector to predict the vector of factors.
Our evaluation framework is shown in Fig. 5. We assume that a feature extractor constructs a single vector representation for a time series of sensor readings. We have implemented disentanglement,
completeness, and informativeness scores computed on 12 failures scores as generative factors and 5 supervised learning targets, including: wind speed, turbulence power, vertical acceleration at the time of landing (a quality of touchdown), autoregression (given t states, predict state t+ 1), failure classification (in dataset generation we randomly applied various failures to the airplane, and the task is to find out whether a failure is present in a given time series). To evaluate supervised learning, we first apply the provided feature extractors for every training example, then train simple models (namely, a three-layer fully connected neural network with ReLU activations) on these features for each target, and finally compute the accuracy on the hold-out set; these accuracies are the final performance metrics. We have made the code for the evaluation pipeline available at (Code, 2018).

heading: 5.2 EVALUATION RESULTS
text: Table 4 shows the main results of our evaluation study; in this section we interpret these quantitative results. The models were trained on the X-Plane dataset with regression tasks evaluated on a holdout set; time series lengths varied from 25 to 75 during training. To establish a simple baseline for our models we use the mean target value over the training set as a prediction. In Table 4, the “MEAN baseline” row shows the absolute MSE values for MEAN and the other rows are normalized to the MEAN baseline. First, we see that different regression tasks have very different models doing well on them; e.g., most models cannot outperform even the mean baseline for wind regression, and models that show good results on wind regression perform poorly on auto regression and vice versa. The effect of images on the results is contradictory: we have trained TS Regression and Dynamics models on three types of data (sensors, images, and both) each, and for TS Regression adding images improves scores on benchmarks, while for the Dynamics model the results deteriorate. The results show that embedding size d needs to be tuned for each model separately; some models even “explode” and give unstable results (very large errors) for some values of d; there is no general correlation between d and benchmark scores. It appears that the Dynamics model trained on sensors only is the best tradeoff between efficiency and quality since it is by far the fastest model. Both disentanglement and completeness scores are small, perhaps because we used only a small subset of generative factors to compute them. The disentanglement score is better for smaller d, which shows that information is packed more efficiently in this case. Performance of the models that use images strongly depends on the quality of the images themselves. For example, if the horizon and runway are clearly visible (as in Table 5a), models with images outperform models that do not use them (Table 5), while on night-time noisy images (Table 5b) models that use images lose.

heading: 6 CONCLUSION
text: In this work, we have presented the dataset for learning state representation and a unified evaluation framework to measure the quality of multimodal joint representations produced by different encoders, through both secondary supervised learning tasks and quantitative metrics of disentanglement and informativeness quality. This work represents an attempt to help advance the research in model-based reinforcement learning and state representation learning by providing a unified. At the same time, the large-scale comparison between baseline and state of the art models that we perform in this work has produced some interesting results by itself. In general, we believe that this dataset can become the new standard for evaluation in representation learning for complex systems, and we hope it will inspire many novel techniques to advance the state of the art that we have attempted to establish and quantify in our practical evaluation.

Review summaries:
The work releases a large-scale multimodal dataset recorded from the X-Plane simulation, as a benchmark dataset to compare various representation learning algorithms for reinforcement learning. The authors also proposed an evaluation framework based on some simple supervised learning tasks and disentanglement scores. The authors then implemented and compared several representation learning algorithms using this dataset and evaluation framework.