## 5 LSTM-Based Model
### Advanced Model with Image Inputs  
Given the limitation of the basic model, more features from the Waymo Open Dataset should be incorporated and one of the most critical data that can be used is the camera image, which perfectly records the changing environment of the AV. It is obvious that a raw camera image cannot be fed into the basic model directly as an image is a two-dimensional data point while the input is required to be a one-dimensional vector. Thus, converting the raw image into a vector whose elements represent the key content of the image is necessary. Naturally, CNN is a popular and excellent image feature extractor that converts an image into a flatten vector. We add one CNN in front of the \"encoder\" module. Training the CNN module with the \"encoder\" module and the \"decoder\" module as a whole can yield an end-to-end advanced  
Figure 4: The basic architecture is relatively straightforward as only 12 features are provided. The information is first encoded by the “encoder” module and an intermediate latent vector, which contains the key value as well as the relationship between the elements, is produced. Then, this vector is stacked in order to match the prediction frame numbers and fed to the “decoder” module and generated a new representation of the original input. The new representation is used to train the final dense layer to predict the acceleration. Given one video clip with a frame-length of 10, the input is the vector consists of 12 features from these 10 frames. The output is the acceleration for the next five frames starting from the end of the video clip. The “encoder” module contains 128 LSTM-cells and the “decoder” module contains 128 LSTM-cells. The optimizer of this network is RMSprop with learning rate 0.0001 and the dropout is used for each LSTM-cell with probability 0.2  
model, much more complex than the basic model. However, training this model from scratch can be time-consuming because of the complexity of a deep CNN module. One quick but efficient solution is to use a pre-trained CNN module to convert the raw image into a vector without any fine-tuning, as the existing popular pre-trained CNN networks are mostly well trained on massive datasets and should generalize well. One of the best CNN networks is the ResNetV2 [9] and the output of its second last layer can be a high-quality representation of the input image. With such an image feature extractor, we decide to use the front camera image in the Waymo Open Dataset as the major input because this camera captures the most critical environment of an AV and contains key information to determine its future acceleration. In other words, image embedding receives and processes inputs from a front camera through the ResNetV2 and outputs an embedding result:  
\[\mathbf{X}_{embedding}=f_{ResNetV2}(\mathbf{I}_{front},\theta_{pretrained})\]  
where \(\mathbf{X}_{embedding}\in\mathbb{R}^{1576}\) is the 1576-dimensional embedding result, \(f_{ResNetV2}\) is the ResNetV2 model with the second last layer as its output, \(\theta_{pretrained}\) is the parameter for the pretrained ResNetV2. With the embedding image and the 12 basic features, the idea of the advanced model can be illustrated by this equation:  
\[\mathbf{O}_{acceleration}=f_{advance}(\mathbf{X}_{embedding},\mathbf{X}_{feature},\theta)\]  
where \(\mathbf{O}_{acceleration}\in\mathbb{R}^{3}\) is a three-dimensional output containing accelerations along the \(x\)-axis, the \(y\)-axis, and the \(z\)-axis, \(\mathbf{X}_{embedding}\in\mathbb{R}^{1576}\) is the embedding image, \(\mathbf{X}_{feature}\in\mathbb{R}^{12}\) and \(f_{advance}\) is the proposed advanced model with its parameter \(\theta\). The architecture of such an advanced model is similar to the previous basic model. An \"encoder-decoder\" structure is maintained to learn the information hidden in the input features. The difference is that the front camera images are treated as additional inputs.