# Recurrent Neural Network-Based Multimodal Deep Learning for Estimating Missing Values in Healthcare
## 5 Conclusions  
This study proposed a recurrent neural network-based multimodal deep learning for estimating missing values. By using the deep learning of soft computing, the proposed method solves both the data redundancy problem caused by data collection from heterogeneous devices and the problem of missing values caused by changing user situations. The multimodal design eases the requirement that the deep learning model has to be relearned when the mobile devices change. By selecting the representative values, the data redundancy caused by heterogeneous devices is removed. For the selection of a representative value, the mean, which features the simplest computation and is easily applicable to any situation, is used.  
Using the data from which redundant items have been removed, an RNN is learned according to variables. Thus, it is possible to avoid the issue of relearning an existing model according to a device change or the addition of a new device. In addition, it is possible to smoothly learn the time series data that are collected in diverse time steps from heterogeneous devices. For estimating the missing value, each learned RNN is connected in a multimodal deep learning structure to learn the influence of variables on each other. An FCN is used to apply this influence on each one's output values through weights. Thus, even if one variable has a missing value, it is possible to estimate an output value that is approximate to the missing value through other variables. For the assessment of the proposed method, the RMSE was used to evaluate the representative value selection and the RMSE and turnaround time to evaluate the estimation methods. The effect of sparse coding on the models' performance was also evaluated.  
According to the evaluation in terms of representative value selection, when a representative value was selected by using the mean or median, the most stable service was achieved. As a result of the evaluation according to the estimation method, the accuracy of the RNN-based multimodal deep learning method is 3.91%p higher than that of the SVD method. According to the evaluation in terms of estimation methods, the RNN-based multimodal deep learning had the smallest error. Nevertheless, the large number of operations in deep learning caused a big difference between the turnaround time of the proposed method and other methods. Therefore, sparse coding was applied to the learned models to reduce computation and decrease turnaround time. Sparse coding caused a loss of accuracy. For this reason, the model that had a higher accuracy rate than the other estimation models and achieved the shortest turnaround time was selected. By using the proposed missing value estimation method that uses RNN-based multimodal deep learning in a heterogeneous healthcare platform, it is possible to solve the problems that heterogeneous devices and deep learning generate. Nevertheless, it is necessary to preprocess learning data in the integration process of RNNs in the multimodal deep learning structure. If an RNN for a new variable is added, relearning for the FCN is required. These problems can be alleviated by combining deep learning techniques and machine learning techniques. Research and experiments on them are required. It is planned in the future research to solve this problem and study further the scalability of multimodal deep learning.  
Conceptualization, J.-C.K.; methodology, J.-C.K. and K.C.; validation, J.-C.K. and K.C.; writing--original draft preparation, J.-C.K. and K.C.; writing--review and editing, J.-C.K. and K.C.; visualization, J.-C.K. and K.C.; supervision, K.C.; project administration, K.C.; funding acquisition, K.C. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the GRRC program of Gyeonggi province [GRRC KGU 2020-B03, Industry Statistics and Data Mining Research].  
Data Availability Statement: Data sharing not applicable.  
The authors declare no conflict of interest.