## 5 Deep Learning for Navigation
### An Overview of Deep Learning
However, for dynamic visual tasks or complex cross-modal tasks such as Visual Question Answering (VQA), the CNNs domain is usually represented only at a coarse frame level without explicitly revealing the temporal structure pertaining to a full-motion video or a textual element [125]. The hidden layers of a multilayer neural network learn to represent the network's inputs in a way that makes it easy to predict the target outputs. This is nicely demonstrated by training a multilayer neural network to predict the next word or video in a sequence from a local context of earlier one. The powerful dynamic system is another famous architecture of deep learning networks named Recurrent Neural Network(RNN), which takes its previous decisions back in current event and focuses on the connotative information about the history of all the past elements of the sequence. It was, nevertheless, proved that training RNNs is problematic because the back-propagated gradients either grow or shrink at each time step, so over many time steps they typically explode or vanish [126]. Thanks to advances in encoder-decoder architecture and ways of training them, RNNs have first been introduced mostly to predict the next text content in natural language processing and translate online speech-to-speech [127; 128]. Although the main purpose of RNNs is to learn long-term dependencies, this architecture has trouble learning and storing temporal information for long periods. To correct for that, the long short-term memory (LSTM) network was first proposed by using special hidden units to remember long-time inputs [129]. This unit called the memory cell acts like an accumulator or a gated leaky neuron: it has a connection to itself at the next time step that has the same weight, so it copies its own real-valued state and accumulates the external signal, but this self-connection is multiplicatively gated by another unit that learns to decide when to clear the content of the memory. LSTM networks or related forms of gated units, such as Gated Recurrent Unit (GRU) [130], have currently proved to perform more effective than conventional RNNs. In recent years, some researchers have made different proposals to contract RNNs with a memory module or an attention module. Proposals include the Neural Turing Machine [131] and Attention Network [132], which have yielded excellent performance on standard question-answering and video-storying tasks [133; 134]. The theoretical and empirical evidence shows that RNNs can be applied to regression problems and complex tasks. Together with CNN, it can model the series-relationship of signals and achieves a good application effect in the recognition of the video captioning model [135]. This method also has been successful in tasks such as classifying the type of the movement in videos [136]. Figure 10: Single-crop top-1 validation accuracies for top scoring single-model architectures. We believe that the end-to-end systems in autonomous driving can benefit from the work in spatio-temporal deep network in video, that could be plausible proposals for localization of the textual elements and temporal characteristics of motion. A major recent practical success of the CNNs-RNNs combining is dynamic object detection and tracking in natural image for mobile robots or vehicle compared to conventional computer vision techniques. In recent years, companies such as Mobileye, NVIDIA and Google are using such methods in their real-time vision or autonomous navigation systems for smartphones, cameras, robots and self-driving cars [137]. A simple CNNs-RNNs technique has demonstrated advantages in pedestrian tracking and road identification task shown in the following Figure 11.  
We can see that the study of robots or self-driving cars has been involved in many steps. Firstly, abundant video and other multi-sensors data can be obtained and preprocessed as input of the intelligent perception system. Then, CNNs and RNNs will be alternate to extract the spatial and temporal motion or other features of different objects.