# Improving Air Pollution Prediction System through Multimodal Deep Learning Model Optimization
## 4 Deep Learning Models
### Multimodal Deep Learning  
Multimodal refers to utilizing two or more modalities, including text and image/text and sound/numerical data and image. The process of constructing and learning a deep learning model is called multimodal deep learning [15]. Multimodal deep learning allows us to correlate relationships between different modalities and solve various problems. In this paper, we construct a multimodal deep learning model for learning different data relationships according to time series characteristics through numerical and image data. However, since numerical and image data have different characteristics, we need to integrate these data. There are several data integration methods [16]. First, the integration of data dimensions is a method of embedding data of different characteristics and extracting data with the same characteristics, as shown in Figure 6. Consequently, data of different characteristics are projected into a single feature space shared. Second, integration between learned representations is a method of combining learned representations with different neural networks, as shown in Figure 7. In this paper, we use the integration of data dimension and integration between learned representations. For example, we train using CNN for image data and LSTM for time series data. Representations learned through each neural network are connected and combined with the hidden layer, and learning is performed by finding the optimal weight connected to the hidden layer. Figure 6: Integration of data dimensions: embedding data of different characteristics (image and numerical data), extracting them into data with identical characteristics, and projecting them into a shared feature space. In this paper, we constructed the multimodal deep learning model, as shown in Figure 8. First, the numerical and image data are preprocessed so that data from the past 5 h can be used as input. However, the two data dimensions do not match even if the same preprocessing is performed over the preceding 5 h. Therefore, we extract the feature map for image data through a CNN-based model to preserve spatial information of the image and reshape the extracted feature map to fit the form of numerical data so that the two data can be integrated to have the same dimensional characteristics. Figure 8: Multimodal deep learning model: First, a feature map of image data is extracted through CNN/VGGNet. Then, the feature map and numerical data are reconstructed to have time series characteristics and merged through concatenate layer. Finally, the merged data is processed by the LSTM AutoEncoder. Figure 7: Integration between learned representations: It is a method of combining representations learned with different neural networks (CNN, LSTM). The representations learned through each neural network are connected to and combined with the hidden layer, and learning is performed in a way that finds the optimal weight connected to the hidden layer. We used the LSTM AutoEncoder model for numerical data processing. The GRU model [17], another compact time series data handling model, is excluded because our parameters increase as numerical data and image data are merged. In this situation, the LSTM model outperforms the GRU model. Furthermore, we used the basic CNN and VGGNet models (VGG16, VGG19) to compare performance differences according to network depth. Then, we merge the numerical data with the reshaped feature map (representations learned through the CNN-based model) via the concatenate layer. However, the data are merged so that the dimension may increase and the learning may not be good. The merged multimodal data have a time series characteristic of the past 5 h. Once the multimodal data are integrated, we feed the integrated data to the LSTM AutoEncoder in our model to reflect the dimensionality reduction and time series characteristics in the preprocessing stage.