Learning State Representations in Complex Systems with Multimodal Data

1 INTRODUCTION . In order to act in real world scenarios and control a complex system such as an airplane , car , or industrial facility , an automated agent needs to process complex high-dimensional data coming from different domains : video feeds from cameras , LIDAR sensors on a car , altitude and speed sensors on an airplane , sensors related to the internal state of the system , and so on . An important problem in this regard would be to map this rich stream of multimodal information into a lower-dimensional space that compresses all modalities into a uniform latent representation ( embedding ) ; the agent could then use this embedding to learn or otherwise construct control algorithms . Thus , representation learning lies at the heart of optimal control for complex systems with multimodal features . Over the last decade , deep neural networks have surpassed other methods in processing nearly all modalities of high-dimensional unstructured data , including images , natural language texts , sounds , and time series . One of the most important properties of neural networks that has made the deep learning revolution possible is their ability to extract meaningful low-dimensional representations of raw unstructured input data . Representation learning with deep neural networks is a large and wellestablished area of research ( Kingma & Welling , 2013 ; Hinton & Salakhutdinov , 2006 ; Radford et al. , 2015 ) . Latent features learned by deep neural networks find applications in various domains , including reinforcement learning for complex systems ( Zhang et al. , 2018 ; Watter et al. , 2015 ; van Hoof et al. , 2016 ) . In these works , the authors often propose special techniques and architectures to design the latent space in different ways suitable for further use : make the latent space locally linear , capture the dynamics , and so on . Designing such feature extractors is a complex task , usually done by hand . Moreover , it is hard to compare different architectures , to a large extent because it is far from obvious how to measure the quality of resulting representations . One reason for that is that the best metric for the quality of learned representations would be the quality of the final task in question , which is hard to obtain in reinforcement learning due to the sheer scale of this final task . A common way to deal with such problems is to use a unified dataset and a unified set of benchmarks , such as , for example , ImageNet and the ILSVRC benchmark in computer vision ( Russakovsky et al. , 2015 ) . Such unified datasets might also prove useful for transfer learning tasks . However , the field of reinforcement learning for complex systems is yet to agree on a common representative large-scale dataset . In this work , we present such a multimodal dataset for the representation learning problem together with a unified benchmark framework for feature extractors suitable for a comprehensive comparative evaluation of different feature extractors . This dataset has been gathered with largescale computer simulations based on the X-Plane simulator and consists of data streams from various sensors along with images taken from the frontal camera of the plane . We propose and implement several different metrics for comparison between extracted features . We also survey , implement , and compare different neural architectures for learning multimodal state representations . The paper is organized as follows . In Section 2 we survey related work on representation learning and evaluation of representations . Section 3 presents the X-Plane dataset and explains its characteristic features . In Section 4 , we present the various representation learning models that we have implemented and compared on this dataset . Section 5 shows the evaluation metrics and presents a large-scale comparison across different models , and Section 6 concludes the paper . 2 RELATED WORK . Modern agent control methods commonly use techniques based on deep learning as feature extractors to deal with complex multimodal data , either explicitly ( Zhang et al. , 2018 ; Watter et al. , 2015 ) or implicitly ( Mnih et al. , 2015 ) . Explicit techniques separate representations of the learning environment from the agents operating in this enviroment ; the environment representation can be learned either independently ( Zhang et al. , 2018 ) or in a common end-to-end architecture ( Parisi et al. , 2017 ) . There are several major approaches to constructing such models ( Lesort et al. , 2018 ) : ( 1 ) autoencoders that reconstruct the input observation data , producing the latent representation between encoder and decoder ; ( 2 ) forward models that predict the next state either in the latent space or in the raw data , basing the prediction on the current latent representation ; ( 3 ) inverse models that use two states to predict the actions between them ; this approach can be combined with forward models ; ( 4 ) models with prior knowledge of the system that impose additional constraints on the latent space according to some fundamental system properties such as causality , temporal continuity , or some more specific knowledge about the system ( Jonschkowski & Brock , 2015 ) ; ( 5 ) multimodal temporal fusion models such as the Memory Fusion Network ( Zadeh et al. , 2018 ) . If there is no additional data labeling available such as , e.g. , a list of factors , the most direct way to measure the quality of representations would be by evaluating the final quality of solving the main task that a model can achieve based on this representation . However , in real world cases , in particular in reinforcement learning , the main task is often hard to solve and unstable to train , so it can not be consistently used to evaluate latent representations . Therefore , many indirect ways have been proposed to measure the quality of representations that introduce proxies that can be expected to lead to better solutions of the final control problem . The most common indirect approaches include ( see also a comprehensive survey by Lesort et al . ( 2018 ) ) : ( 1 ) task performance , the most intuitive metric , where representation quality is measured by the quality of performing some other relatively simple task , e.g. , by predicting some available target variables with simple models that take the latent representation as input ( Higgins et al. , 2016 ; van Hoof et al. , 2016 ) ; ( 2 ) KNN-MSE , proposed by Lesort et al . ( 2017 ) , measures the degree of preservation of the same neighbors between the latent space and the ground truth : KNN-MSE ( I ) = 1k ∑ I′∈KNN ( I , k ) ‖φ ( I ) − φ ( I ′ ) ‖ , where I is the initial raw input , I ′ is a neighbour of I , and φ is the feature extractor ; KNN-MSE is a good metric in situations where the distance in the original input space is well-defined but becomes hard to apply for highly variable multimodal data ; ( 3 ) a similar approach with humans in the loop is to evaluate whether similar input states according to human evaluation do indeed map to close representations in the latent space ( Sermanet et al. , 2018 ) ; however , for complex multimodal inputs this is again inapplicable since a human would not have an intuitive notion of similarity between two sets of several hundred sensor readings ; ( 4 ) disentanglement scores ( Eastwood & Williams , 2018 ; Higgins et al. , 2016 ) measure the disentanglement ( mutual independence ) of extracted features ; if there are some known generative factors , these metrics assess whether individual elements of the latent representation capture individual generative factors independently ; we will consider such metrics in detail in Section 5.1 . As a representative multimodal dataset for a complex system , we have used the X-Plane flight simulator ( Laminar Research , 2018 ) , well-known for its faithful simulation of all systems of an aircraft . It has already been used to solve optimal control problems for aircraft ; e.g. , Bittar et al . ( 2014 ) develop separate control laws for stable and maneuvering flight , and Garcia & Barnes ( 2009 ) use X-Plane to simulate a system of several unmanned aerial vehicles . However , to the best of our knowledge this is the first attempt to produce a large-scale dataset for representation learning from X-Plane or any other flight simulator . 3 THE X-PLANE DATASET . We present the X-Plane Dataset ( Dataset , 2018 ; Code , 2018 ) as a benchmark for evaluating the quality of state representations ( embeddings ) , where the main problem is to encode the state of some complex system , learn a mapping from high-dimensional multimodal data to a low-dimensional latent representation . We have used the X-Plane simulation environment because it is extremely accurate and can produce a lot of useful and different sensor data . In total , we have recorded 8011 landings , with mean duration of a landing of 115 seconds . For every landing , the dataset contains the readings of 1090 sensors arranged in time series recorded with with a frequency of 5 frames per second , together with the corresponding image taken from the camera located at the front of the airplane ; the images are recorded at the same frequency . Table 1 summarizes the various groups of sensors recorded in the dataset , showing the number of sensors in a group together with a brief description . The total dataset size is 93GB uncompressed ; it contains sensor readings and about 7M 256 × 256 images , joined into per-flight videos for better compression . To ensure diversity in the dataset , we recorded landings with random perturbations and in different environments : with different airports and runways , time of day , weather conditions etc . We have used at most 12 perturbations ( abrupt changes in the environment such as , e.g. , wind gusts , malfunction of various systems in the aircraft , and so on ) applied during each flight ; Table 2 summarizes various failures and perturbations . We have used a custom control unit based on the Boeing 737 guide ( Brady , 2014 ) in order to make automatic landings stable . During dataset collection , for every landing we chose a random airport and a random runway and spawned a plane in a random position , usually at a distance of 3-5 miles from the runway . There are , in total , 114 different runways in 70 airports used in the dataset . Then we applied different landing conditions—landing speed , flaps position , time of day , visibility , precipitation and so on— and enabled the autopilot that lowered the plane on the glide slope path . To achieve a better landing , we disable the autopilot before touchdown , increase the pitch of the plane , and after touchdown we decrease the pitch and enable reverse thrust . At the end of the flight , there are three possible situations : successful landing , aircraft crash , or time limit reached . We set a strict time limit of 160 seconds for every flight . We have also done an airport-stratified split of the dataset into 4 parts : training set for feature extractors , validation set used for early stopping , training set for benchmark supervised learning models , and test set for scoring the benchmark models . We have made the dataset available at ( Dataset , 2018 ) .

Summary: The work releases a large-scale multimodal dataset recorded from the X-Plane simulation, as a benchmark dataset to compare various representation learning algorithms for reinforcement learning. The authors also proposed an evaluation framework based on some simple supervised learning tasks and disentanglement scores. The authors then implemented and compared several representation learning algorithms using this dataset and evaluation framework.