Missing Data Infill with Automunge
Year: 2021

Abstract:
Missing data is a fundamental obstacle in the practice of data science. This paper 1 surveys a few conventions for imputation as available in the Automunge open 2 source python library platform for tabular data preprocessing, including “ML infill” 3 in which auto ML models are trained for target features from partitioned extracts 4 of a training set. A series of validation experiments were performed to benchmark 5 imputation scenarios towards downstream model performance, in which it was 6 found for the given benchmark sets that in many cases ML infill outperformed for 7 both numeric and categoric target features, and was otherwise at minimum within 8 noise distributions of the other imputation scenarios. Evidence also suggested 9 supplementing ML infill with the addition of support columns with boolean integer 10 markers signaling presence of infill was usually beneficial to downstream model 11 performance. We consider these results sufficient to recommend defaulting to 12 ML infill for tabular learning, and further recommend supplementing imputations 13 with support columns signaling presence of infill, each as can be prepared with 14 push-button operation in the Automunge library. Our contributions include an 15 auto ML derived missing data imputation library for tabular learning in the python 16 ecosystem, fully integrated into a preprocessing platform with an extensive library 17 of feature transformations, with a novel production friendly implementation that 18 bases imputation models on a designated train set for consistent basis towards 19 additional data. 20

Missing data is a fundamental obstacle in the practice of data science. This paper1 surveys a few conventions for imputation as available in the Automunge open2 source python library platform for tabular data preprocessing, including “ML infill”3 in which auto ML models are trained for target features from partitioned extracts4 of a training set. A series of validation experiments were performed to benchmark5 imputation scenarios towards downstream model performance, in which it was6 found for the given benchmark sets that in many cases ML infill outperformed for7 both numeric and categoric target features, and was otherwise at minimum within8 noise distributions of the other imputation scenarios. Evidence also suggested9 supplementing ML infill with the addition of support columns with boolean integer10 markers signaling presence of infill was usually beneficial to downstream model11 performance. We consider these results sufficient to recommend defaulting to12 ML infill for tabular learning, and further recommend supplementing imputations13 with support columns signaling presence of infill, each as can be prepared with14 push-button operation in the Automunge library. Our contributions include an15 auto ML derived missing data imputation library for tabular learning in the python16 ecosystem, fully integrated into a preprocessing platform with an extensive library17 of feature transformations, with a novel production friendly implementation that18 bases imputation models on a designated train set for consistent basis towards19 additional data.20
1 Introduction21
Missing data is a fundamental obstacle for data science practitioners. Missing data refers to feature22 sets in which a portion of entries do not have samples recorded, which may interfere with model23 training and/or inference. In some cases, the missing entries may be randomly distributed within the24 samples of a feature set, a scenario known as missing at random. In other cases, certain segments of a25 feature set’s distribution may have a higher prevalence of missing data than other portions, a scenario26 known as missing not at random. In some cases, the presence of missing data may even correlate27 with label set properties, resulting in a kind of data leakage for a supervised training operation.28
In a tabular data set (that is a data set aggregated as a 2D matrix of feature set columns and collected29 sample rows), missing data may be represented by a few conventions. A common one is for missing30 entries to be received as a NaN value, which is a special numeric data type representing “not a31 number”. Some dataframe libraries may have other special data types for this purpose. In another32 configuration, missing data may be represented by some particular value (like a string configuration)33 associated with a feature set.34
When a tabular data set with missing values present is intended to serve as a target for supervised35 training, machine learning (ML) libraries may require as a prerequisite some kind of imputation36 to ensure the set has all valid entries, which for most libraries means all numeric entries (although37 there are some libraries that accept designated categoric feature sets in their string representations).38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
Conventions for imputation may follow a variety of options to target numeric or categoric feature sets39 [Table 1], many of which apply a uniform infill value, which may either be arbitrary or derived as a40 function of other entries in the feature set.41
Other, more sophisticated conventions for infill may derive an imputation value as a function of42 corresponding samples of the other features. For example, one of many learning algorithms (like43 random forest, gradient boosting, neural networks, etc.) may be trained for a target feature where44 the populated entries in that feature are treated as labels and surrounding features sub-aggregated45 as features for the imputation model, and where the model may serve as either a classification or46 regression operation based on properties of the target feature.47
This paper is to document a series of validation experiments that were performed to compare48 downstream model performance as a result of a few of these different infill conventions. We crafted a49 contrived set of scenarios representing paradigms like missing at random or missing not at random50 as injected in either a numeric or categoric target feature selected for influence toward downstream51 model performance. Along the way we will offer a brief introduction to the Automunge library for52 tabular data preprocessing, particularly those aspects of the library associated with missing data infill.53 The results of these experiments summarized below may serve as a validation of defaulting to ML54 infill for tabular learning even when faced with different types of missing data, and further defaulting55 to supplementing imputations with support columns signaling presence of infill.56
Our contributions include an auto ML derived missing data imputation library for tabular learning57 in the python ecosystem, fully integrated into a preprocessing platform with an extensive library of58 feature transformations, extending the ML imputation capabilities of R libraries like MissForest [1]59 to a more production friendly implementation that bases imputation models on a designated train set60 for consistent basis towards additional data.61
2 Automunge62
Automunge [2], put simply, is a python library platform for preparing tabular data for machine63 learning, built on top of the Pandas dataframe library [3] and open sourced under a GNU GPL64 v3.0 license. The interface is channeled through two master functions: automunge(.) for the initial65 preparation of training data, and postmunge(.) for subsequent efficient preparation of additional “test”66 data on the train set basis. In addition to returning transformed data, the automunge(.) function also67 populates and returns a compact dictionary recording all of the steps and parameters of transformations68 and imputations, which dictionary may then serve as a key for consistently preparing additional data69 in the postmunge(.) function on the train set basis.70
Under automation the automunge(.) function performs an evaluation of feature set properties to71 derive appropriate simple feature engineering transformations that may serve to normalize numeric72 sets and binarize (or hash) categoric sets. A user may also apply custom transformations, or even73 custom sets of transformations, assigned to distinct columns. Such transformations may be sourced74 from an extensive internal library, or even may be custom defined. The resulting transformed data log75 the applied stages of derivations by way of suffix appenders on the returned column headers.76
Missing data imputation is handled automatically in the library, where each transformation applied77 includes a default imputation convention to serve as a precursor to imputation model training, one78 that may also be overridden for use of other conventions by assignment.79
Included in the library of infill options is an auto ML solution we refer to as ML infill, in which a80 distinct model is trained for each target feature and saved in the returned dictionary for a consistent81 imputation basis of subsequent data in the postmunge(.) function. The model architecture defaults to82 random forest [4] by Scikit-Learn [5], and other auto ML library options are also supported.83
The ML infill implementation works by first collecting a ‘NArw’ support column for each received84 feature set containing boolean integer markers (1’s and 0’s) with activations corresponding to entries85 with missing or improperly formatted data. The types of data to be considered improperly formatted86 are tailored to the root transformation category to be applied to the column, where for example87 for a numeric transform non-numeric entries may be subject to infill, or for a categoric transform88 invalid entries may just be special data types like NaN or None. Other transforms may have other89 configurations, for example a power law transform may only accept positive numeric entries, or an90 integer transform may only accept integer entries.91
This NArw support column can then be used to perform a target feature specific partitioning of the92 training data for use to train a ML infill model [Fig 1]. The partitioning segregates rows between those93 corresponding to missing data in the target feature verses those rows with valid entries, with the target94 feature valid entries to serve as labels for a supervised training and the other corresponding features’95 samples to serve as training data. Feature samples corresponding to the target feature missing data96 are grouped for an inference operation. Note that for cases where a transformation set has prepared97 a target input feature in multiple configurations, those derivations other than the target feature are98 omitted from the partitions to avoid data leakage. A similar partitioning is performed for test data99 sets for ML infill imputation, although in this case only the rows corresponding to entries of missing100 data in the target feature are utilized for inference. As a further variation available for any of the101 imputation methods, the NArw support columns may themselves be appended to the returned data102 sets as a signal to training of entries that were subject to infill.103
There is a categorization associated with each preprocessing transformation category to determine104 the type of ML infill training operation, for example a target feature set derived from a transform that105 returns a numeric form may be a target for a regression operation or a target feature set derived from106 a transform that returns an ordinal encoding may be a target for a classification operation. In some107 cases a target feature may be composed of a set of more than one column, like in the case of a set108 returned from a one-hot encoding. For cases where a learner library does not accept some particular109 form of encoding as valid labels there is a conversion of the target feature set for training and an110 inverse conversion after any inference, for example it may be necessary to convert a binarized target111 feature set to one-hot encoding or ordinal encoding for use as labels in different auto ML frameworks.112
As may be particularly beneficial in cases with high prevalence of missing data across features, the113 sequential training of feature imputation models may be iterated through repeated rounds of imputa-114 tions. For instance in the first round of model trainings and imputations the models’ performance115
may be slightly degraded by high prevalence of missing data populated with the initial transformation116 function imputation conventions in surrounding features, but after that first round of imputations a117 second iteration of model trainings may have slight improvement of performance due to the presence118 of ML infill imputations, and similarly ML infill may benefit from any additional iterations of model119 trainings and imputations. In each iteration the sequence of imputations between columns are applied120 in an order from features with highest prevalence of missing data to least. The library defaults to a121 single round of imputations, with the option to specify an additional iteration quantity.122
The final trained models for each target feature, as derived from properties of a designated train set123 passed to the automunge(.) function, are collectively saved and returned to the user in a dictionary124 that may serve as a key for consistent imputation basis to additional data in the postmunge(.) function,125 with such dictionary also serving as a key for any applied preprocessing transformations.126
3 Preprocessing127
The utility of the library extends well beyond missing data infill. Automunge is intended as a platform128 for all of the tabular learning steps following receipt of tidy data [6] (meaning one column per129 feature and one row per sample) and immediately preceding the application of machine learning. We130 found that by integrating the imputations directly into a preprocessing library, benefits included that131 imputations can be applied to returned multi-column categoric representations like one-hot encodings132 or binarized encodings, can account for potential data leakage between redundantly encoded feature133 sets, and can accept raw data as input as may include string encoded and date-time entries with only134 the minimal requirement of data received in a tidy form.135
Under automation, Automunge normalizes numeric sets by z-score normalization and binarizes136 categoric sets (where binarize refers to a multi-column boolean integer representation where each137 categoric unique entry is represented by a distinct set of zero, one, or more simultaneous activations).138 We have a separate kind of binarization for categoric sets with two unique entries, which returns a139 single boolean integer encoded column (available as a single column by not having a distinct encoding140 set for missing data which is instead grouped with the most common entry). High cardinality categoric141 sets with unique entry count above a configurable heuristic threshold are instead applied with a hashing142 trick transform [7, 8], and for highest cardinality approaching all unique entries features are given a143 parsed hashing [9] which accesses distinct words found within entries. Further automated encodings144 are available for date-time sets in which entries are segregated by time scale and subject to separate145 sets of sine and cosine transforms at periodicity of time scale and additionally supplemented by146 binned activations for business hours, weekdays, and holidays. Designated label sets are treated a147 little differently, where numeric sets are left un-normalized and categoric sets are ordinal encoded (a148 single column of integer activations). All of the defaults under automation are custom configurable.149
A user need not defer to automation. There is a built in extensive library of feature transformations to150 choose from. Numeric features may be assigned to any range of transformations, normalizations, and151 bin aggregations [10]. Sequential numeric features may be supplemented by proxies for derivatives152 [10]. Categoric features may be subject to encodings like ordinal, one-hot, binarization, hashing, or153 even parsed categoric encoding [11] with an increased information retention in comparison to one-hot154 encoding by a vectorization as a function of grammatical structure shared between entries. Categoric155 sets may be collectively aggregated into a single common binarization. Categoric labels may have156 label smoothing applied [12], or fitted smoothing where null values are fit to class distributions. Data157 augmentation transformations [10] may be applied which make use of noise injection, including158 several variants for both numeric and categoric features. Sets of transformations to be directed at a159 target feature can be assembled which include generations and branches of derivations by making use160 of our “family tree primitives” [13], as can be used to redundantly encode a feature set in multiple161 configurations of varying information content. Such transformation sets may be accessed from those162 predefined in an internal library for simple assignment or alternatively may be custom configured.163 Even the transformation functions themselves may be custom defined with only minimal requirements164 of simple data structures. Through application statistics of the features are recorded to facilitate165 detection of distribution drift. Inversion is available to recover the original form of data found166 preceding transformations, as may be useful to recover the original form of labels after inference.167
Or of course if the data is received already numerically encoded the library can simply be applied as168 a tool for missing data infill.169
4 Code Demonstration170
Jupyter notebook install and imports are as follows:171
!pip install Automunge172 173 from Automunge import *174 am = AutoMunge()175
The automunge(.) function accepts as input a Pandas dataframe or tabular Numpy array of training176 data and optionally also corresponding test data. If any of the sets include a label column that header177 should be designated, similarly with any index header or list of headers to exclude from the ML infill178 basis. For Numpy, headers are the index integer and labels should be positioned as final column.179
import pandas as pd180 df_train = pd.read_csv('train.csv')181 df_test = pd.read_csv('test.csv')182 labels_column = '<labels_column_header>'183 trainID_column = '<ID_column_header>'184
These data sets can be passed to automunge(.) to automatically encode and impute. The function185 returns 10 sets (9 dataframes and 1 dictionary) which in some cases may be empty based on parameter186 settings, we suggest the following optional naming convention. The final set, the “postprocess_dict”,187 is the key for consistently preparing additional data in postmunge(.). Note that if a validation set188 is desired it can be partitioned from df_train with valpercent and prepared on the train set basis.189 Shuffling is on by default for train data and off by default for test data, the associated parameter190 is shown for reference. Here we demonstrate with the assigncat parameter assigning the root191 category of a transformation set to some target column which will override the default transform192 under automation. We also demonstrate with the assigninfill parameter assigning an alternate193 infill convention to a column. The ML infill and NArw column aggregation are on by default, their194 associated activation parameters are shown for reference. Note that if the data is already numerically195 encoded and user just desires infill, they can pass parameter powertransform = 'infill'.196
train, train_ID, labels, \197 val, val_ID, val_labels, \198 test, test_ID, test_labels, \199 postprocess_dict = \200 am.automunge(df_train,201 df_test = df_test,202 labels_column = labels_column,203 trainID_column = trainID_column,204 valpercent = 0.2,205 shuffletrain = True,206 assigncat = {'or23' : ['<parsed_categoric_target_column>'] },207 assigninfill = {'modeinfill' : ['<infill_target_column>'] },208 MLinfill = True,209 NArw_marker = True)210
A list of columns returned from some particular input feature can be accessed with211 postprocess_dict['column_map']['<input_feature_header>']. A report classifying the212 returned column types (such as continuous, boolean, ordinal, onehot, binary, etc.) and their groupings213 can be accessed with postprocess_dict['columntype_report'].214
If the returned train set is to be used for training a model that may go into production, the postpro-215 cess_dict should be saved externally, such as with the pickle library.216
We can then prepare additional data on the train set basis with postmunge(.).217
test, test_ID, test_labels, \218 postreports_dict = \219 am.postmunge(postprocess_dict,220 df_test)221
5 Related Work222
The R ecosystem has long enjoyed access to missing data imputation libraries that apply learned223 models to predict infill based on other features in a set, such as MissForest [1] and mice [14], where224 MissForest differs from mice as a deterministic imputation built on top of random forest and mice225 applies chained equations with pooled linear models and sampling from a conditional distribution.226 One of the limitations of these libraries are that the algorithms must be run through both training227 and inference for each separate data set, as may be required if test data is not available at time of228 training, which practice may not be amenable to production environments. Automunge on the other229 hand bases imputations on a designated train set, returning from application a collected dictionary of230 feature set specific models that can then be applied as a key for consistently preparing additional data231 on the train set basis.232
Automunge’s ML infill also differs from these R libraries by providing multiple auto ML options233 for imputation models. We are continuing to build out a range that currently includes Catboost [15],234 AutoGluon [16], and FLAML [17] libraries. Our default configuration is built on top of Scikit-Learn235 [5] random forest [4] models and may be individually tuned to each target feature with grid or random236 search by passing fit parameters to ML infill as lists or distributions.237
There are of course several other variants of machine learning derived imputations that have been238 demonstrated elsewhere. Imputations from generative adversarial networks [18] may improve239 performance compared to ML infill (at a cost of complexity). Gaussian copula imputation [19] has240 a benefit of being able to estimate uncertainty of imputations. There are even imputation solutions241 built around causal graphical models [20]. Towards the other end of complexity spectrum, k-Nearest242 Neighbor imputation [21] for continuous data is available in common frameworks like Scikit-Learn.243
Being built on top of the Pandas library, there is an inherent limitation that Automunge operations are244 capped at in-memory scale data sets. Other dataframe libraries like Spark [22] have the ability to245 operate on distributed datasets. We believe this is not a major limitation because the in memory scale246 is only associated with datasets passed to automunge(.) to serve as the basis for transformations and247 imputations. Once the basis has been established, transformations to any scale of data can be applied248 by passing partitions to the postmunge(.) function. We expect there may be potential to parallelize249 such an operation with a library like Dask [23] or Ray [24], such an implementation is currently250 intended as a future direction of research.251
Another limitation associated with Pandas dataframes is that operations take place on the CPU. There252 are emerging dataframe platforms like Rapids [25] which are capable of GPU accelerated operations,253 which may particularly be of benefit when you take account for the elimination of a handoff step254 between main and GPU memory to implement training. Although the Pandas aspects of Automunge255 are CPU bound, the range of auto ML libraries incorporated are in some cases capable of GPU256 training for ML infill.257
There will always be a simplicity advantage to deep learning libraries like Tensorflow [26] or PyTorch258 [27] which can integrate preprocessing as a layer directly into a model’s architecture, eliminating the259 need to consider preprocessing in inference. We believe the single added inference step of passing260 data to the postmunge(.) function is an acceptable tradeoff because by keeping the preprocessing261 operations separate it facilitates a ML framework agnostic tabular preprocessing platform.262
6 Experiments263
Some experiments were performed to evaluate efficacy of a few different imputation methods in264 different scenarios of missing data. To amplify the impact of imputations, each of two data sets265 were pared down to a reduced set of the top 15 features based on an Automunge feature importance266 evaluation [11] by shuffle permutation [28]. (This step had the side benefit of reducing the training267 durations of experiments.) The top ranked importance categoric and numeric features were selected268 to separately serve as targets for injections of missing data, with such injections simulating scenarios269 of both missing at random and missing not at random.270
To simulate cases of missing not at random, and also again to amplify the impact of imputation, the271 target features were evaluated to determine the most influential segments of the features’ distributions272
[29], which for the target categoric features was one of the activations and for the target numeric273 features turned out to be the far right tail for both benchmark data sets.274
Further variations were aggregated associated with either the ratio of full feature or ratio of distribution275 segments injected with missing data, ranging from no injections to full replacement.276
Finally, for each of these scenarios, variations were assembled associated with the type of infill277 applied by Automunge, including scenarios for defaults (mean imputation for numeric or distinct278 activations for categoric), imputation with mode, adjacent cell, and ML infill. The ML infill scenario279 was applied making use of the CatBoost library to take advantage of GPU acceleration.280
Having prepared the data in each of these scenarios with an automunge(.) call, the final step was to281 train a downstream model to evaluate impact, again here with the CatBoost library. The performance282 metric applied was root mean squared error for the regression applications. Each scenario was283 repeated 68 or more times with the metrics averaged to de-noise the results.284
Finally, the ML infill scenarios were repeated again with the addition of the NArw support columns285 to supplement the target features.286
7 Results287
The results of the various scenarios are presented [Fig 2, 3, 4, 5]. Here the y axis are the performance288 metrics and the x axis the ratio of entries with missing data injections, which were given as {0, 0.1,289 0.33, 0.67, 1.0}, where in the 0.0 case no missing data was injected and with 1.0 the entire feature or290 feature segment was injected. Because the 0.0 cases had equivalent entries between infill types, their291 spread across the four infill scenarios are a good approximation for the noise inherent in the learning292 algorithm. An additional source of noise for the other ratios was from the stochasticity of injections,293 with a distinct set for each trial. Consistent with common sense, as the injection ratio was ramped up294 the trend across infill scenarios was a degradation of the performance metric.295
We did find that with increased repetitions incorporated the spread of the averaged performance296 metrics were tightened, leading us to repeat the experiments at increased scale for some improved297 statistical significance.298
For the missing at random injections [Fig 2, 3], ML infill was at or near top performance across both299 data sets, although the spread between imputations was not extremely pronounced. In most of the300 setups, mode imputation and adjacent cell trended as reduced performance in comparison to ML infill301 or the default imputations (mean for numeric sets and distinct activation set for categoric).302
For not at random injections to the right tail of numeric sets [Fig 4], it appears that ML infill had a303 pronounced benefit to the Ames Housing data set [30], especially as the injection ratio increased,304 and more of an intermediate performance to the Allstate Claims data set [31]. We speculate that ML305 infill had some degree of variability across these demonstrations due to correlations (or lack thereof)306
between the target feature and the other features, without which ML infill may struggle to establish a307 basis for inference. In the final scenario of not at random injections to categoric [Fig 5] we believe308 default performed well because it served as a direct replacement for the single missing activation.309
An additional comparable series of injections were conducted with ML infill and the added difference310 of appending the NArw support columns corresponding to the target columns for injections. Again311 these NArw support columns are the boolean integer markers for presence of infill in the corresponding312 entries which support the partitioning of sets for ML infill. The expectation was that by using these313 markers to signal to the training operation which of the entries were subjected to infill, there would314 be some benefit to downstream model performance. For many of the scenarios the visible impact was315 that supplementing with the NArw support column improved the ML infill performance, demonstrated316 here for missing at random [Fig 6, 7] and missing not at random [Fig 8, 9] with the other imputation317 scenarios shown again for context.318
8 Discussion319
One of the primary goals of this experiment was to validate the efficacy of ML infill as evidenced320 by improvements to downstream model performance. For the Ames Housing benchmark data set,321 there was a notable demonstration of ML infill benefiting model performance in the scenario of322 the numeric target column with not at random injections at increased injection ratios, and also to323 a lesser extent with missing at random injections. We speculate an explanation for this advantage324 towards the numeric target columns may partly be attributed to the fact that the downstream model325 was also a regression application, so that the other features selected for label correlation may by326 proxy have correlations with the target numeric feature. The corollary is that the more mundane327 performance of ML infill toward the categoric target columns may be a result of these having less328 correspondence with the surrounding features. The fact that even in these cases the ML infill still fell329 within noise distribution of the other imputation scenarios we believe presents a reasonable argument330 for defaulting to ML infill for tabular learning.331
Note that as another argument for defaulting to ML infill as opposed to static imputations is that the332 imputation model may serve as a hedge against imperfections in subsequent data streams, particularly333 if one of the features experiences downtime in a streaming application for instance.334
The other key finding of the experiment was the pronounced benefit to downstream model performance335 when including the NArw support column in the returned data set as a supplement to ML infill. This336 finding was consistent with our intuition, which was that increased information retention about infill337 points should help model performance. Note there is some small tradeoff, as the added training set338 dimensionality may increase training time. Another benefit to including NArw support columns may339
be for interpretability in inspection of imputations. We recommend including the NArw support340 columns for model training based on these findings, with the one caveat that care should be taken to341 avoid inclusion in the data leakage scenario where there is some kind of correlation between presence342 of missing data and label set properties that won’t be present in production.343
9 Conclusion344
Automunge offers a push-button solution to preparing tabular data for ML, with automated data345 cleaning operations like normalizations, binarizations, and auto ML derived missing data imputation346 aka ML infill. Transformations and imputations are fit to properties of a designated train set, and with347 application of automunge(.) a compact dictionary is returned recording transformation parameters348 and trained imputation models, which dictionary may then serve as a key for consistently preparing349 additional data on the train set basis with postmunge(.).350
We hope that these experiments may serve as a kind of validation of defaulting to ML infill with351 supplemented NArw support columns in tabular learning for users of the Automunge library, as352 even if in our experiments the material benefits towards downstream model performance were not353 demonstrated for all target feature scenarios, in other cases there did not appear to be any material354 penalty. Note that ML infill can be activated for push-button operation by the automunge(.) parameter355 MLinfill=True and the NArw support columns included by parameter NArw_marker=True. Based356 on these findings these two parameter settings are now cast as defaults for the Automunge platform.357
Acknowledgments358
A thank you owed to those facilitators behind Stack Overflow, Python, Numpy, Scipy Stats, PyPI,359 GitHub, Colaboratory, Anaconda, VSCode, and Jupyter. Special thanks to Scikit-Learn and Pandas.360
References361
[1] Daniel J. Stekhoven, Peter Bühlmann. MissForest - nonparametric missing value imputation for mixed-type362 data (2011) arXiv:1105.0828363
[2] Author(s) (2021) Automunge, GitHub repository (Please see supplemental material)364
[3] W. McKinney. Data structures for statistical computing in python. Proceedings of the 9th Python in Science365 Conference, pages 51–56, 2010.366
[4] L. Breiman. Random Forests. Machine Learning, 45(1), 2001.367
[5] Pedregosa et al., Scikit-learn: Machine Learning in Python, JMLR 12, pp. 2825-2830, 2011.368
[6] H. Wickham. Tidy data. Journal of Statistical Software, 59(10), 2014.369
[7] John Moody. Fast Learning in Multi-Resolution Hierarchies. NIPS Proceedings, 1989370
[8] Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, Josh Attenberg. Feature Hashing for371 Large Scale Multitask Learning. ICML Proceedings, 2009372
[9] Author(s) Hashed Categoric Encodings with Automunge (2020) (Please see preprint in supplemental material)373
[10] Author(s) Numeric Encoding Options with Automunge (2020) (Please see preprint in supplemental material)374
[11] Author(s) Parsed Categoric Encodings with Automunge (2020) (Please see preprint in supplemental375 material)376
[12] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, Zbigniew Wojna. Rethinking the Inception377 Architecture for Computer Vision. IEEE conference on computer vision and pattern recognition, 2016378
[13] Author(s) Specification of Derivations with Automunge (2020) (Please see preprint in supplemental379 material)380
[14] Stef van Buuren, Karin Groothuis-Oudshoorn. mice: Multivariate Imputation by Chained Equations in R381 (2011) https://www.jstatsoft.org/article/view/v045i03382
[15] Anna Veronika Dorogush, Vasily Ershov, Andrey Gulin. CatBoost: gradient boosting with categorical383 features support (2018) arXiv:1810.11363384
[16] Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola.385 AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020) arxiv:2003.06505386
[17] Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu. FLAML: A Fast and Lightweight AutoML Library387 (2019) arXiv:1911.04706388
[18] Jinsung Yoon, James Jordon, Mihaela van der Schaar. GAIN: Missing Data Imputation using Generative389 Adversarial Nets (2018 International Conference of Machine Learning), arXiv:1806.02920390
[19] Yuxuan Zhao, Madeleine Udell. Missing Value Imputation for Mixed Data via Gaussian Copula (KDD391 2020), arXiv:1910.12845392
[20] K. Mohan, J. Pearl. Graphical Models for Processing Missing Data (2019), arXiv:1801.03583393
[21] Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastie, Robert Tibshirani, David394 Botstein and Russ B. Altman. Missing value estimation methods for DNA microarrays, BIOINFORMATICS395 Vol. 17 no. 6, 2001 Pages 520-525.396
[22] Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui397 Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker,398 Ion Stoica. Apache Spark: a unified engine for big data processing. Communications of the ACM, 59(11), 2016399
[23] Dask Development Team. Dask: Library for dynamic task scheduling (2016) https://dask.org400
[24] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih401 Elibol, Zongheng Yang, William Paul, Michael I. Jordan, Ion Stoica. Ray: A Distributed Framework for402 Emerging AI Applications. 13th USENIX Symposium on Operating Systems Design and Implementation403 (2018), arXiv:1712.05889404
[25] Rapids Development Team. Open GPU Data Science | RAPIDS https://rapids.ai405
[26] Abadi, Martín, Barham P, Chen J, Chen Z, Davis A, Dean J, et al. Tensorflow: A system for large-scale406 machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (2016) p.407 265–83.408
[27] Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan,409 Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison,410 Alban and Kopf, Andreas and Yang, Edward and DeVito, Zachary and Raison, Martin and Tejani, Alykhan and411 Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith. PyTorch: An412 Imperative Style, High-Performance Deep Learning Library. NeurIPS Proceedings, 2019413
[28] Terrence Parr, Kerem Turgutlu, Christopher Csiszar, and Jeremy Howard. Beware default random forest414 importances. Explained.ai (blog), 2018. https://explained.ai/rf-importance/.415
[29] Author(s) Automunge Influence (2020) (Please see preprint in supplemental material)416
[30] Dean De Cock. Ames, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression417 Project, Journal of Statistics Education, Volume 19, Number 3 (2011)418
[31] Kaggle: Allstate Claims Severity, https://www.kaggle.com/c/allstate-claims-severity419
Checklist420 1. For all authors...421
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contribu-422 tions and scope? [Yes]423
(b) Did you describe the limitations of your work? [Yes] Please see discussions in section 5 Related424 Work425
(c) Did you discuss any potential negative societal impacts of your work? [Yes] A Broader Impacts426 discussion is provided as Appendix C427
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]428 2. If you are including theoretical results...429
(a) Did you state the full set of assumptions of all theoretical results? [N/A]430 (b) Did you include complete proofs of all theoretical results? [N/A]431
3. If you ran experiments...432 (a) Did you include the code, data, and instructions needed to reproduce the main experimental433 results (either in the supplemental material or as a URL)? [Yes] Please see jupyter notebooks434 provided with supplemental material435
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were436 chosen)? [Yes] We noted that missing data injections were random for each trial. Performance437 was evaluated on a 25% validation split. We used hyperparameter defaults for learning.438
(c) Did you report error bars (e.g., with respect to the random seed after running experiments439 multiple times)? [Yes] We noted that since scenarios for 0% injection are comparable between440 imputation methods, their spread may serve as a proxy for noise inherent in the operation.441
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs,442 internal cluster, or cloud provider)? [No] Our experiments did not require significant compute.443
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...444 (a) If your work uses existing assets, did you cite the creators? [Yes]445 (b) Did you mention the license of the assets? [Yes] We note licenses of supporting packages in the446 read me document included in the github repository folder within the supplemental material.447 (c) Did you include any new assets either in the supplemental material or as a URL? [Yes]448 (d) Did you discuss whether and how consent was obtained from people whose data you’re us-449 ing/curating? [N/A]450 (e) Did you discuss whether the data you are using/curating contains personally identifiable informa-451 tion or offensive content? [N/A]452 5. If you used crowdsourcing or conducted research with human subjects...453
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?454 [N/A]455
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB)456 approvals, if applicable? [N/A]457
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on458 participant compensation? [N/A]459

Review summaries:
The paper introduces a library for data wrangling. This include, in particular, feature process and missing value imputation. The library is described and demonstrated to be effective at the targeted tasks.

The paper presents the automunge library for missing data imputation and preprocessing for supervised learning in Python. The automunge library implements ML-based infill similar to the established MissForest and MICE algorithms.  The authors discuss several imputation methods, as well as the automunge API, and evaluate the impact of different imputation strategies on two regression datasets, using a CatBoost model as the final machine learning model.  The authors conclude that ML infill with an indicator feature for missing values (called NArw in paper) achieves the best performance for continuous features, and that for categorical features, that all methods perform similarly.

The paper introduces ‘ML infill’ , which is a Python implementation of a sequential imputation technique. More precisely, this method iteratively considers each feature $X_j$ with missing values as a target feature, fits a predictive model based on the remaining features as input variables and the observed values of $X_j$ as labels, and finally predicts the missing entries of $X_j$. ‘ML infill’ is provided as part of an open source Python library, Automunge, whose goal is to prepare data to feed it to a machine learning algorithm (e.g. feature transformation, and imputation of missing values). The paper provides experiments comparing the impact of basic imputations on downstream regression tasks, and recommend defaulting to ‘ML infill’ and using the missingness indicator as additional predictor.