# Recurrent Neural Network-Based Multimodal Deep Learning for Estimating Missing Values in Healthcare
###### Abstract  
This estimation method operates by integrating the input values that are redundantly collected from heterogeneous devices through the selection of a representative value and estimating missing values by using a multimodal RNN. Users use a heterogeneous healthcare platform mainly in a mobile environment. Users who pay a relatively large amount of attention to healthcare possess various types of healthcare devices and collect data through their mobile devices. The collected data may be duplicated depending on the types of these devices. This data duplication causes an ambiguity issue in that it is difficult to determine which value among multiple data should be taken as the user's actual value. Accordingly, it is necessary to create a neural network structure that considers the data value at the time previous to the current time. RNNs are appropriate for handling data with a time series characteristic. To learn an RNN-based neural network, learning data that have the same time step are required. Therefore, an RNN in which one variable becomes single-modal was designed for each learning run. In the RNN, a cell is a gated recurrent unit (GRU) cell that presents sufficient accuracy in the small resource environment of mobile devices. The RNNs that are learned according to the variables can each operate without additional learning, even if the situation of the user's mobile device changes. In a heterogeneous environment, missing values are generated by various types of errors, including errors caused by battery charge and discharge, sensor failure, equipment exchange, and near-field communication errors. The higher the missing value ratio, the greater the number of errors that are likely to occur. For this reason, to achieve a more stable heterogeneous health platform, missing values must be considered. In this study, a missing value was estimated by means of multimodal deep learning; that is, a multimodal deep learning method was designed with one neural network that was connected with each learned single-modal RNN using a fully connected network (FCN). Each RNN input value delivers mutual influence through the weights of the FCN, and thereby, it is possible to estimate an output value even if any one of the input values is missing. According to the evaluation in terms of representative value selection, when a representative value was selected by using the mean or median, the most stable service was achieved. As a result of the evaluation according to the estimation method, the accuracy of the RNN-based multimodal deep learning method is 3.91%p higher than that of the SVD method. estimation; missing value; multimodal; deep learning; heterogeneous; health +
Footnote †: journal: Medical  
+++ ==WARNING: Truncated because of repetitions==
convergence technology in various areas, such as society, science, and industry, has been used to develop a health platform. Accordingly, health data are collected through many different routes, such as electronic medical records (EMRs), personal health records (PHRs), and life logs [3]. In addition, through the digitalization of medical records, new data are accumulated continuously. Health data vary continuously, and their variables and attributes are diverse [4]. Many health data have a time series characteristic. In other words, the data continuously occur and change as time elapses. A soft computing-based heterogeneous healthcare platform observes and analyzes how multiple variables, such as weather, nutrition, and activity, influence health over time [1; 2; 3; 4; 5]. The influence of these variables on personal health differs according to the individual's living habits, family history, and disease status, and the range of collected variables differs according to the user's surroundings, devices, and personal situation [6]. Multimodality describes the integrated environment of data from multiple sensors or sources for a single object [7]. In a health platform, user data are collected in multimodal form: temperature, humidity, GPS, travel range, PHRs, and EMRs.