A Systematic Comparison of Deep Learning Architectures in an Autonomous Vehicle

Self-driving technology is advancing rapidly — albeit with significant challenges and limitations. This progress is largely due to recent developments in deep learning algorithms. To date, however, there has been no systematic comparison of how different deep learning architectures perform at such tasks, or an attempt to determine a correlation between classification performance and performance in an actual vehicle, a potentially critical factor in developing self-driving systems. Here, we introduce the first controlled comparison of multiple deep-learning architectures in an end-to-end autonomous driving task across multiple testing conditions. We used a simple and affordable platform consisting of an off-the-shelf, remotely operated vehicle, a GPU-equipped computer, and an indoor foamrubber racetrack. We compared performance, under identical driving conditions, across seven architectures including a fully-connected network, a simple 2 layer CNN, AlexNet, VGG-16, Inception-V3, ResNet, and an LSTM by assessing the number of laps each model was able to successfully complete without crashing while traversing an indoor racetrack. We compared performance across models when the conditions exactly matched those in training as well as when the local environment and track were configured differently and objects that were not included in the training dataset were placed on the track in various positions. In addition, we considered performance using several different data types for training and testing including single grayscale and color frames, and multiple grayscale frames stacked together in sequence. With the exception of a fully-connected network, all models performed reasonably well (around or above 80%) and most very well (∼95%) on at least one input type but with considerable variation across models and inputs. Overall, AlexNet, operating on single color frames as input, achieved the best level of performance (100% success rate in phase one and 55% in phase two) while VGG-16 performed well most consistently across image types. Performance with obstacles on the track and conditions that were different than those in training was much more variable than without objects and under conditions similar to those in the training set. Analysis of the model’s driving paths found greater consistency within vs. between models. Path similarity between models did not correlate strongly with success similarity. Our novel pixelflipping method allowed us to create a heatmap for each given image to observe what features of the image were weighted most heavily by the network when making its decision. Finally, we found that the variability across models in the driving task was not fully predicted by validation performance, indicating the presence of a ‘deployment gap’ between model training and performance in a simple, real-world task. Overall, these results demonstrate the need for increased field research in self-driving. 1Center for Complex Systems and Brain Sciences, Florida Atlantic University, 777 Glades Road, Boca Raton, FL 33431, USA 2College of Computer and Information Science, Northeastern University, 360 Huntington Ave, Boston, MA 02115, USA 3Department of Ocean and Mechanical Engineering, Florida Atlantic University, 777 Glades Road, Boca Raton, FL 33431, USA † mteti@fau.edu