## 2 Related Works
### Application of Multimodal Deep Learning
#### 2.2.2 Application of Multimodal Techniques in Other Fields  
Multimodal deep learning has been explored across diverse fields. [PERSON], [PERSON], et al. [17] conducted an extensive study on multimodal deep learning, incorporating modalities such as image, video, text, audio, body gestures, facial expressions, and physiological signals. Among those, there have been studies applying multimodal deep learning combining images and text, including visual-based referred expression understanding and phrase localization [18; 19], as well as image and video captioning [20; 21; 22], text-to-image generation [23; 24; 25], and visual question answering (VQA) [26; 27; 28; 29; 30]. [PERSON] et al. [26] developed a multimodal deep learning model for the visual question answering (VQA) task. They employed a model with two channels: an image and a text (question) channel. The image channel used VGGNet for feature extraction, while the text channel utilized LSTM to embed the textual data. They then combined the two channels' features through element-wise multiplication to transform the data into a shared space and obtain the final classification result. [PERSON] et al. [31], [PERSON] et al. [32], [PERSON] et al. [33], and [PERSON] et al. [34] also developed multimodal models using both visual and text information. [PERSON] et al. [31] proposed a multimodal deep learning model that combines image data and text metadata to retrieve social media posts containing important information about floods. Their model uses a convolutional neural network (CNN) for extracting visual features and a bidirectional LSTM network for extracting semantic features from the text metadata. The experimental evaluation showed that combining these two types of information improves the performance of retrieving flood-related posts. [PERSON] et al. [32] proposed a multimodal deep learning approach to support interpreting the intended message of information graphics. An information graphic refers to the visual representation such as lines or bar graphs in a document or popular media. The multimodal model uses pixel and text data to classify information graphics into six intention categories validated as useful for people with visual impairments. For this, the model utilizes CNN\(-\)based layers for processing pixel information graphics and a bag of words (BOW) for handling OCR-extracted textual information. The authors showed that the proposed multimodal method performs the best among the compared methods. [PERSON] et al. [33] considered the visual and text information extracted from images to classify digitized documents. Their approach involves processing the visual and text information separately using CNN\(-\)based layers. First, they trained a CNN\(-\)based layer for visual information with grayscale document images resized at 384 \(\times\) 384. For textual information, they used Tesseract OCR to extract text from images and conducted text embedding using FastText to feed the results to the CNN\(-\)based layer. Finally, they concatenated the image and text CNN\(-\)based layers. The authors validated their approach on the Tobacco3482 and RVL-CDIP datasets. The experimental evaluation showed that their multimodal approach outperforms the existing single modality approaches. [PERSON] et al. [34] introduced CB-Fake, a multimodal deep learning framework designed to identify fake news at the early stage by analyzing both the textual and visual content of the news article. The CB-Fake model incorporates CapsNet to extract crucial visual features from images and employs the pre-trained BERT language model to capture context-rich textual features from news articles. These features are synergistically fused to create a richer data representation, subsequently fed into a classification layer to determine whether the news is fake or real. Some studies considered other modalities besides visual and textual information. [PERSON] et al. [35] proposed an approach for classifying music genres from different data modalities, including audio, text, and images.