# Adaptive Imputation of Irregular Truncated Signals with Machine Learning
## 3 Materials and Methods
### Automatic Determination of the Type of Missing Data  
Distinguishing between the different types of missing data programmatically has long proven to be a difficult task. For MCAR data, the randomness implies that there is no pattern or information that can be used to identify the missingness as MCAR without additional external knowledge. Identifying MAR data often involves the use of statistical models that aim to infer relationships between the probability of missingness and other observed variables. The downside to using these models is that they are often complex and laden with assumptions. The types of inferences performed by these statistical models may be difficult to recreate programmatically, because they often involve nuanced judgements that may not be possible without subjective interpretation or additional context about the data, which may include domain-specific knowledge that isn't always available. MNAR data is generally considered the most complex type of missing data to detect, because understanding why certain data is missing often requires access to the same data that's unavailable, which makes it almost impossible to identify MNAR data without resorting to making assumptions about the missing data mechanism or obtaining external information. Additionally, statistical tests designed to identify MCAR data, like [PERSON]'s MCAR test, have limitations. They can indicate that the data is not MCAR, but that can't accurately distinguish between MAR and MNAR without further, more detailed analysis. Creating accurate models for the missing data mechanism, especially to identify MAR and MNAR, requires complex and comprehensive statistical models. These models must consider all observed variables, a task that is not only computationally intensive but also fraught with assumptions that may not accurately reflect the underlying missing data mechanism. The scheme we used for this research to determine the type of missing data involves the use of the Missingno package for Python. This package enables flexible and easy-to-use missing data visualizations and utilities that all a user to get a quick visual summary of the completeness of a given dataset. Once of the visualizations available in this package is the correlation heatmap, which shows how strongly the presence or absence of one variable affects the presence of another by visualizing the nullity correlation scores for the variables  
Figure 2: (**a**) MCAR data, (**b**) MAR data, (**c**) MNAR data. Each type of missing data was introduced with a percentage of 10%. The gray blocks in the figure represent data that is present, and the white lines represent missing data. containing missing data. Sample heatmaps of the dataset after each type of missing data has been introduced are shown in Figure 3. Figure 4 is a plot showing how the nullity correlation scores change as the percentage of missing data increases from 1% to 99% for each of the three types of missing data. The correlation heatmaps show that there is no correlation score for the MCAR data, which makes sense given that the missingness of MCAR data is completely random and does not depend on the values of any variables observed or unobserved. The moderate correlation score for MAR data also makes sense as in our setup, the missingness in the \"Vibration\" variable is systematically related to the observed values in \"Temperature, but not the unobserved values or its own missing values. The high correlation score found in MNAR data indicates a direct relationship between the missingness of the data and its own values, which in this case represents that both \"Pressure\" and \"Vibration\" are more likely to be missing when their values are extreme. Based on the information depicted in Figure 4, we devised a scheme to automatically determine the type of missing data based on the nullity correlation scores. In our method, if the average correlation score is greater than or equal to 0 but less than 0.1, then the missing data is classified as MCAR.