# Recurrent Neural Network-Based Multimodal Deep Learning for Estimating Missing Values in Healthcare
## 2 Related Work
### Data Characteristics in Heterogeneous Healthcare Platform  
A heterogeneous healthcare platform is an environment where various health devices such as health bands, smartwatches, blood glucose meters, and weight scales collect data from smartphones. Private health data are recorded by the heterogeneous devices for managing user health, such as drug administration and diet. It supports decision making by collecting the health-related data collected by users as observation targets. Variables that can be collected differ depending on the device, and even the same variable may change the observed value depending on the wearing area. Different types of devices collect dataon a single object, resulting in redundancy. The diverse data collected in a heterogeneous healthcare platform have a variety of characteristics according to the variables. Variables, such as weather, ECG data, EMG data, and temperature, have the time series characteristic of continuous numerical data [4]. EMR and PHR data, as unstructured data, have diverse categories and attributes, and feature a mixture of strings and numbers [5; 15]. Weather, x-ray, and endoscopy data are collected in the format of videos or images [16]. In addition to these, there are numerous variables that influence human health. It is impossible to collect all the variables using the current technologies. Thus, soft computing is used to find the optimal answer using the given variables. ANN is classified into DNN, CNN, and RNN according to the characteristics of the neural network. RNN is most suitable for healthcare data with time series features. When the variables show the time series feature, most frequently, RNNs are used to predict a change in the current state of healthcare [17]. Each variable is collected in a different cycle, and therefore a basic time step is ambiguous in RNN learning. Therefore, the variables not collected in the same cycle learn the RNN separately. To consider the hidden associations of multiple variables, the neural network structure needs to integrate each neural network [18]. Unstructured data, such as EMR and PHR data, are analyzed through natural language processing, text mining, and association analysis. For this purpose, the data need to be preprocessed in line with the output [19]. Image or video data are applied to decision making by using a neural network specialized for classification or clustering, as shown in a CNN. The data sets used in a health platform are the wearable stress and affect detection (WESAD) data set [20], the mobile health (mHealth) data set [21], the diabetes data set [22], the bike sharing data set [23], and the health news data set [24]. The WESAD data set [20] contains data collected from devices worn on the wrist and the chest. It includes ECG, electrodermal activity, EMG, respiration, body temperature, 3-axis acceleration, and blood amount pulse data. The mHealth data set [21] contains the data of body movement and bio signals generated by 12 types of physical activities. The data were recorded via mobile devices. The 12 types of physical activities include walking, stair climbing, biking, running, and jumping. Their repeated count or practice time were recorded. The data, such as x-axis acceleration, y-axis acceleration, z-axis acceleration, ECG, and magnetic field, were recorded using 24 sensors. The diabetes data set [22] contains the data collected through the EMRs and paper records of diabetic patients. In the case of an EMR, the time-stamp for an event is automatically recorded by the device's internal watch. In the case of paper records, a certain time is fixed, and then the medical record is filled at breakfast, lunch, dinner, and sleep time. The variables of the data set are date, time, code, and value. The code consists of the blood sugar before a meal, the blood sugar after a meal, the general meal amount, and the amount of insulin. The bike sharing data set [23] contains the data collected by a bike sharing system, including, date, time, travel route, weather, temperature, humidity, and wind velocity.