Missing Data Infill with Automunge
Authors: Nicholas Teague
Venue: NeurIPS 2021 Submitted (2021)
Keywords: missing data
tabular
infrastructure

Abstract: Missing data is a fundamental obstacle in the practice of data science. This paper surveys a few conventions for imputation as available in the Automunge open source python library platform for tabular data preprocessing, including "ML infill" in which auto ML models are trained for target features from partitioned extracts of a training set. A series of validation experiments were performed to benchmark imputation scenarios towards downstream model performance, in which it was found for the given benchmark sets that in many cases ML infill outperformed for both numeric and categoric target features, and was otherwise at minimum within noise distributions of the other imputation scenarios. Evidence also suggested supplementing ML infill with the addition of support columns with boolean integer markers signaling presence of infill was usually beneficial to downstream model performance. We consider these results sufficient to recommend defaulting to ML infill for tabular learning, and further recommend supplementing imputations with support columns signaling presence of infill, each as can be prepared with push-button operation in the Automunge library. Our contributions include an auto ML derived missing data imputation library for tabular learning in the python ecosystem, fully integrated into a preprocessing platform with an extensive library of feature transformations, with a novel production friendly implementation that bases imputation models on a designated train set for consistent basis towards additional data.

Missing Data Infill with Automunge
Anonymous Author(s)
Affiliation Address email
Abstract
Missing data is fundamental obstacle in the practice of data science. This paper surveys a few conventions for imputation as available in the Automunge open source python library platform for tabular data preprocessing, including "ML infill" in which auto ML models are trained for target features from partitioned extracts of training A series of validation experiments were performed benchmark imputation scenarios towards downstream model performance, in which it was found for the given benchmark sets that in many cases ML infill outperformed for both numeric and categoric target features, and was otherwise at minimum within noise distributions of the other imputation scenarios. Evidence also suggested supplementing MI. infill with the addition of support columns with boolean integer markers signaling presence of infill was usually beneficial to downstream model performance. We consider these results sufficient to recommend defaulting to ML infill for tabular learning, and further recommend supplementing imputations with support columns signaling presence of infill, each as can be prepared with push-button operation the Automunge library. Our contributions include an auto ML derived missing data imputation library for tabular learning in the python ecosystem, fully integrated into a preprocessing platform with an extensive library of feature transformations, with novel production friendly implementation that bases imputation models on a designated train set for consistent basis towards additional data.
2
11
13
15
17
19
Introduction
21
22 Missing data is fundamental obstacle for data science practitioners, Missing data refers to feature sets in which portion of entries do not have samples recorded, which may interfere with model 23
24 training and/or inference. In some cases, the missing entries may be randomly distributed within the
25 samples of feature set, scenario known as missing random. In other cases, certain segments of
26 feature set's 's distribution may have higher prevalence of missing data than other portions, scenario
27 known as missing not at random. some cases, the presence of missing data may even correlate with label set properties, resulting in a kind of data leakage for supervised training operation.
29 In tabular data set (that is data set aggregated a 2D matrix of feature set columns and collected sample rows), missing data may be represented by few conventions. A common one is for missing entries be received as NaN value, which is a special numeric data type representing "not number". Some dataframe libraries may have other special data types for this purpose. In another
30
31
32
33 configuration, missing data may be represented by some particular value (like a string configuration) associated with feature set.
34
35
When tabular data with missing values present is intended serve as a target for supervised training, machine learning (ML) libraries may require as prerequisite some kind of imputation ensure the set has all valid entries, which for most libraries means all numeric entries (although there are some libraries that accept designated categoric feature sets in their string representations).
38
37
38
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

Under automation the automunge()) function performs an evaluation of feature set properties to 72 derive appropriate simple feature engineering transformations that may serve to normalize numeric sets and binarize (or hash) categoric sets. A user may also apply custom transformations, or even 74 custom sets of transformations, assigned to distinct columns. Such transformations may be sourced from an extensive internal library. even may be custom defined. The resulting transformed data log 39 Conventions for imputation may follow a variety of options to target numeric or categoric feature sets
40 [Table 1I, many of which apply uniform infill value, which may either be arbitrary or derived as
41 function of other entries in the feature set.
Table 1: Imputation Conventions
Imputation Value
Numeric
Categoric
mean median
mode
adjacent cell
arbitrary (e.g. 0 or 1)
distinct activation MI. infill
42
Other, more sophisticated conventions for infill may derive an imputation value as a function of corresponding samples of the other features. For example, one many learning algorithms (like random forest, gradient boosting, neural networks, etc.) may be trained for target feature where the populated entries in that feature are treated as labels and surrounding features abb-aggeeatted as features for the imputation model, and where the model may serve as either a classification or regression operation based on properties of the target feature.
43
44
45
46
47
48
This paper is to document a series of validation experiments that were performed to compare downstream model performance contrived set of scenarios representing paradigms like missing at random or missing not at random as injected in either numeric or categoric target feature selected for influence toward downstream model performance. Along the way we will offer a brief introduction to the Automunge library for tabular data preprocessing, particularly those aspects of the library associated with missing data infill. The results of these experiments summarized below may serve validation of defaulting to ML infill for tabular learning even when faced with different types of missing data, and further defaulting to supplementing imputations with support columns signaling presence of infill.
49
result of few of these different infill conventions. We crafted
50
51
52
53
54
55
56
57
Our contributions include in the python ecosystem, fully integrated into a preprocessing platform with an extensive library of feature transformations, extending the ML imputation capabilities of R libraries like MissForest [1]
ML derived missing data imputation library for tabular learning
58
59
80 to more production friendly implementation that bases imputation models on a designated train set for consistent basis towards additional data.
61
Automunge
82
Automunge [2]. put simply, is python library platform for preparing tabular data for machine learning, built on top of the Pandas dataframe library [3] and open sourced under a GNU GPL v3.0 license. The interface is channeled through two master functions: automunge(.) for the initial preparation of training data, and postmunge(.) for subsequent efficient preparation of additional "test" data the train set basis. In addition to returning transformed data, the automunge(.) function also populates and returns compact dictionary recording all of the steps and parameters of transformations 89 and imputations, which dictionary may then serve a key for consistently preparing additional data in the postmunge(.) function on the train set basis.
64
66
68
70
76 the applied stages derivations by way of suffix appenders on the returned column headers.
Missing data imputation is handled automatically in the library, where each transformation applied 78 includes a default imputation convention to serve as a precursor to imputation model training, one that may also be overridden for use of other conventions by assignment.

---

80
Included in the library of infill options is an auto ML solution we refer to as ML infill, in which distinct model trained for each target feature and saved in the returned dictionary for a consistent imputation basis of subsequent data in the postmunge(.) function. The model architecture defaults to random forest [4] by Scikit-Learn [5], and other auto MI. library options are also supported.
81
85
84
The ML infill implementation works by first collecting a 'NArw' support column for each received feature set containing boolean integer markers (I's and 0's) with activations corresponding to entries with missing or improperly formatted data. The types of data to be considered improperly formatted are tailored to the root transformation category be applied to the column, where for example for a numeric transform non non-numeric entries may be subject to infill, or for a categoric transform invalid entries may just be special data types like NaN or None. Other transforms may have other configurations, for example power law transform may only accept positive numeric entries, or an integer transform may only accept integer entries.
85
86
87
88
86
90
91
92
This NArw support column can then be used to perform a target feature specific partitioning of the training data for corresponding to missing data in the target feature verses those rows with valid entries, with the target feature valid entries serve labels for supervised training and the other corresponding features' samples to serve as training data. Feature samples corresponding to the target feature missing data has prepared
93
train ML infill model [Fig The partitioning segregates rows between those
94
95
96
97
are grouped for an inference operation. Note that for cases where transformation target input feature in multiple configurations, those derivations other than the target feature are omitted from the partitions to avoid data leakage. A similar partitioning performed for test data sets for MI infill imputation, although in this case only the rows corresponding to entries of missing data in the target feature are utilized for inference. Ás further variation available for any of the
98
99
101
102
imputation methods, the NArw support columns may themselves be appended to the returned data
103
sets as signal to training of entries that were subject to infill.
ML infill
X
target column(s)
training data
model
llbels
features
(ifill
predict
Figure ML Infill partitioning
104
There is categorization associated with each preprocessing transformation category to determine the type of MI. infill training operation, for example a target feature set derived from a transform that returns a numeric form may be target for a regression operation or target feature set derived from transform that returns ordinal encoding may be target for classification operation. In some cases a target feature may be composed of set of more than one column, like the case of a set returned from one-hot encoding. For cases where learner library does not accept some particular form of encoding as valid labels there a conversion of the target feature set for training and an inverse conversion after any inference, for example it may be necessary to convert a binarized target feature set to one-hot encoding or ordinal encoding for use as labels in different auto ML frameworks.
106
108
110
112
As may be particularly beneficial in cases with high prevalence of missing data across features, the sequential training of feature imputation models may be iterated through repeated rounds of imputa- tions. For instance in the first round of model trainings and imputations the models' performance
114

---

116
may be slightly degraded by high prevalence of missing data populated with the initial transformation function imputation conventions in surrounding features, but after that first round of imputations a second iteration of model trainings may have slight improvement of performance due to the presence of ML. infill imputations, and similarly MI. infill may benefit from any additional iterations of model trainings and imputations. In each iteration the sequence of imputations between columns are applied
117
118
119
120
121
in an order from features with highest prevalence of missing data to least. The library defaults to a single round of imputations, with the option to specify an additional iteration quantity.
122
123
The final trained models for each target feature, as derived from properties of a designated train set passed to the automunge(.) function, are collectively saved and returned to the user in a dictionary that may serve as key for consistent imputation basis to additional data in the postmunge(.) function, with such dictionary also serving as a key for any applied preprocessing transformations.
124
125
128
3
Preprocessing
127
128
The utility of the library extends well beyond missing data infill. Automunge is intended a platform for all of the tabular learning steps following receipt of tidy data [6] (meaning one column per feature and one row per sample) and immediately preceding the application of machine learning. We found that by integrating the imputations directly into a preprocessing library, benefits included that imputations can be applied to returned multi-column categoric representations like one-hot encodings
129
130
131
132
133
or binarized encodings, can account for potential data leakage between redundantly encoded feature sets, and can accept raw data as input as may include string encoded and date-time entries with only the minimal requirement of data received in a tidy form.
134
135
136
Under automation, Automunge normalizes numeric sets by z-score normalization and binarizes categoric sets (where binarize refers to a multi-column boolean integer representation where each categoric unique entry is represented by distinct set of zero, one, more simultaneous activations).
137
138
139
We have a separate kind of binarization for categoric sets with two unique entries, which returns a single boolean integer encoded column (available as a single column by not having distinct encoding
140
141
set for missing data which is instead grouped with the most common entry). High cardinality categoric sets with unique entry count above a configurable heuristic threshold are instead applied with a hashing
142
143
trick transform [7, 8]. and for highest cardinality approaching all unique entries features are given a parsed hashing [9] which accesses distinct words found within entries. Further automated encodings are available for date-time sets in which entries are segregated by time scale and subject to separate sets of sine and cosine transforms at periodicity of time scale and additionally supplemented by binned activations for business hours, weekdays, and holidays. Designated label sets are treated a little differently, where numeric sets are left un-normalized and categoric sets are ordinal encoded (a single column of integer activations). All of the defaults under automation are custom configurable.
144
145
146
147
148
150
user need not defer to automation. There is built in extensive library of feature transformations to choose from. Numeric features may be assigned to any range of transformations, normalizations, and bin aggregations [10]. Sequential numeric features may be supplemented by proxies for derivatives [10]. Categoric features may be subject to encodings like ordinal, one-hot, binarization, hashing, or even parsed categoric encoding [11] with an increased information retention in comparison to one-hot encoding by vectorization as function of grammatical structure shared between entries. Categoric sets may be collectively aggregated into a single common binarization. Categoric labels may have label smoothing applied [12], or fitted smoothing where null values are fit to class distributions. Data augmentation transformations [10] may be applied which make use of noise injection, including several variants for both numeric and categoric features. Sets of transformations to be directed at target feature can be assembled which include generations and branches of derivations by making use of our "family tree primitives"
151
152
153
154
155
156
157
158
159
160
[13], can be used to redundantly encode feature set in multiple configurations of varying information content. Such transformation sets may be accessed from those predefined in an internal library for simple assignment or alternatively may be custom configured. Even the transformation functions themselves may be custom defined with only minimal requirements of simple data structures. Through application statisties of the features are recorded to facilitate detection of distribution drift. Inversion is available to recover the original form of data found preceding transformations, as may be useful recover the original form of labels after inference.
162
164
165
186
167
168
Or of course if the data is received already numerically encoded the library can simply be applied as tool for missing data infill.

---

Code Demonstration
170
Jupyter notebook install and imports are follows:
172
!pip install Automunge
173
174
from Automunge import
175
am AutoMunge ()
The automunge(.) function accepts input Pandas dataframe tabular Numpy array of training 177 data and optionally also corresponding test data. If any of the sets include lahel column that header should be designated, similarly with any index header list of headers exclude from the ML infill basis. For Numpy, headers are the index integer and labels should be positioned as final column.
178
179
import pandas pd 181
df train pd. read_eearyyll c trriin. csv') if_test = pd.reed_cspp ('test. csv') labels_col column clabels_collme coluuc_eaeeseuee HaDD_colum__heeeeeeee_
189
MhiinD_coolmme
185
These data sets can be passed automunge(.) automatically encode and impute. The function returns 10 sets (9 dataframes and dictionary) which in some cases may be empty based parameter settings, we suggest the following optional naming convention. The final set, the "postprocess __dct",,i""""
186
187
is the key for consistently preparing additional data in postmunge()). Note that if validation set desired can partitioned from ur_irain with valpercent and prepared on train set basis. Shuffling is on by default for train data and off by default for test data, the associated parameter shown for reference. Here we demonstrate with the assigncat parameter assigning the root category of a transformation set to under automation. We also demonstrate with the assigninf apiinfill parameter assigning an alternate infill convention to column. The ML infill and NArw column aggregation are by default, their associated activation parameters are shown for reference. Note that if the data is already numerically encoded and user just desires infill, they can pass parameter povertransform 'infill'.
189
191
target column which will override the default transform
193
195
197 train, train_ID, labels, val, val_ID, fal_labels, test, test_ID, test_labell, postprocess_dict-
199
201
am automunge (df_trainn
df_test = df_test,
203
dabels_colume labels_colume
trainID_colume trainID_colume
205
valpercent 0.2,
shuffletrain True,
207
assigncat = or23'' : ['<parsed_catee caweeoooic.taggetcoooloooooo }, assigninfill '' ('modeinfill' s<infiill_target.com
}
209
MLinfill True,
210
NArw_marker True)
211 list of columns returned from some particular input feature can be accessed with pastprocessddictt colllun'oo____oo returned column types (such as continuous, boolean, ordinal, onehot, binary, etc.) and their groupings can be accessed with postprocess _ecesss__cc dict ( "colunntype_repon cetoopperrrrrr'_._
['<input_feattun beature_hhaerr1] 'J. A report classifying the
213
215 If the returned train is be used for training model that may go into production, the postpro- cess dict should be saved externally, such as with the pickle library.
216
We then prepare additional data the train basis with postmunge()).
218 test, test_ID, test_ lest_labels, costreports_dice =
220
am. postmunue (postprocess_ df_test)
5

---

222
5
Related Work
223
The R ecosystem has long enjoyed access to missing data imputation libraries that apply learned models to predict infill based on other features in a set, such as MissForest [1] and mice [14), where MissForest differs from mice as deterministic imputation built on top of random forest and mice applies chained equations with pooled linear models and sampling from a conditional distribution. One of the limitations of these libraries are that the algorithms must be run through both training and inference for each separate data set, as may be required if test data is not available at time of training. which practice may not be amenable to production environments. Automunge on the other hand bases imputations on a designated train set, returning from application a collected dictionary of feature set specific models that can then be applied as key for consistently preparing additional data on the train set basis.
224
225
226
228
230
232
233
Automunge's ML infill also differs from these R libraries by providing multiple auto ML options for imputation models. We are continuing to build out range that currently includes Catboost [15], AutoGluon [16], and FLAML [17] libraries. Our default configuration is built on top of Scikit-Learn [5] random forest [4] models and may be individually tuned to each target feature with grid or random search by passing fit parameters to ML infill as lists or distributions.
234
236
238
There are of course several other variants of machine learning derived imputations that have been demonstrated elsewhere. Imputations from generative adversarial networks [18] may improve performance compared ML infill (at a cost of complexity). Gaussian copula imputation [19] has a benefit of being able to estimate uncertainty of imputations. There are even imputation solutions built around causal graphical models [20]. Towards the other end of complexity spectrum, k-Nearest Neighbor imputation [21] for continuous data is available in common frameworks like Scikit-Learn.
239
240
241
242
243
244
Being built on top of the Pandas library, there is an inherent limitation that Automunge operations are capped at in-memory scale data sets. Other dataframe libraries like Spark [22] have the ability to operate on distributed datasets. We believe this is not a major limitation because the in memory scale is only associated with datasets passed to automunge(.) to serve as the basis for transformations and imputations. Once the basis has been established. transformations to any scale of data can be applied by passing partitions to the postmunge(.) function. We expect there may be potential to parallelize such an operation with a library like Dask [23] or Ray [24], such an implementation currently intended as future direction of research.
245
248
247
248
249
250
251
252
Another limitation associated with Pandas dataframes is that operations take place on the CPU. There are emerging dataframe platforms like Rapids [25] which are capable of GPU accelerated operations, which may particularly be of benefit when you take account for the elimination of handoff step between main and GPU memory to implement training. Although the Pandas aspects of Automunge are CPU bound, the range of auto ML libraries incorporated are in some cases capable of GPU training for ML infill.
253
254
255
256
257
There will always be a simplicity advantage deep learning libraries like Tensorflow [26) or Py [27] which can integrate preprocessing as a layer directly into a model's architecture, eliminating the need to consider preprocessing in inference. We believe the single added inference step of passing
259
260
261
data to the postmunge(.) function is an acceptable tradeoff because by keeping the preprocessing
282
operations separate it facilitates ML framework agnostic tabular preprocessing platform.
6
Experiments
Some experiments were performed to evaluate efficacy of a few different imputation methods in different scenarios of missing data. To amplify the impact of imputations, each of two data sets were pared down to a reduced set of the top 15 features based on an Automunge feature importance evaluation [11] by shuffle permutation [28]. (This step had the side benefit of reducing the training
265
267
durations of experiments.) The top ranked importance categoric and numeric features were selected
269
to separately serve as targets for injections of missing data, with such injections simulating scenarios of both missing random and missing not random.
270
271
To simulate cases of missing not at random, and also again to amplify the impact of imputation, the target features were evaluated to determine the most influential segments of the features' distributions

---

273
[29], which for the target categoric features was one of the activations and for the target numeric features turned out be the far right tail for both benchmark data sets.
274
275
Further variations were aggregated associated with either the ratio of full feature or ratio of distribution segments injected with missing data, ranging from no injections to full replacement.
276
277
Finally, for each of these scenarios, variations were assembled associated with the type of infill applied by Automunge, including scenarios for defaults (mean imputation for numeric or distinct activations for categoric), imputation with mode, adjacent cell, and ML infill. The ML infill scenario was applied making use of the CatBoost library to take advantage of GPU acceleration.
279
281
Having prepared the data in each of these scenarios with an automunge(.) call, the final step was to train a downstream model to evaluate impact, again here with the CatBoost library. The performance metric applied was root mean squared error for the regression applications. Each scenario was repeated 68 or more times with the metrics averaged to de-noise the results.
283
285
Finally, the ML infill scenarios were repeated again with the addition of the NArw support columns to supplement the target features.
286
7
Results
287
The results of the various scenarios are presented [Fig 2, 3, 4. 5]. Here the y axis are the performance metrics and the x axis the ratio of entries with missing data injections, which were given as {0, 0.1, 0.33, 0.67, 1.0), where in the 0.0 case no missing data was injected and with 1.0 the entire feature feature segment was injected. Because the 0.0 cases had equivalent entries between infill types, their
289
291
282
spread across the four infill scenarios are a good approximation for the noise inherent in the learning algorithm. An additional source of noise for the other ratios was from the stochasticity of injections, with distinct set for each trial. Consistent with common sense, as the injection ratio was ramped up the trend across infill scenarios was a degradation of the performance metric.
293
284
285
We did find that with increased repetitions incorporated the spread of the averaged performance metrics were tightened, leading us to repeat the experiments at increased scale for some improved statistical significance.
297
299
For the missing at random injections [Fig 2. 3], ML infill was at or near top performance across both data sets, although the spread between imputations was not extremely pronounced. In most of the setups, mode imputation and adjacent cell trended as reduced performance in comparison to ML infill
300
301
or the default imputations (mean for numeric sets and distinct activation set for categoric).
275
2319
0100
180
3030
195
Figure 2: Missing at Random Numeric Target Feature
Mosii Pondiin
Hasiing
Xitatt
20:00
4200
oct ee
mide
9-301
-20
0130
3550
2000
34
250
65
Figure 3: Missing at Random CCtteooric Target Feature
303 For not at random injections to the right tail of numeric sets [Fig 4], appears that ML infill had a pronounced benefit to the Ames Housing data set [30], especially as the injection ratio increased, 305 and more of an intermediate performance the Allstate Claims data set [31]. We speculate that MI.
infill had some degree of variability across these demonstrations due to correlations (or lack thereof)
7

---

307
between the target feature and the other features, without which ML infill may struggle to establish basis for inference. the final scenario of random injections to categoric [Fig 5] we believe default performed well because it served as direct replacement for the single missing activation.
308
laassing
Mistate
20500
3030
2330
3123
defoult
4010
00330
MI. intill
21930
250
2333
740
1970
30
02
0.5
13
3923
00
22
04
0.0
10
Figure 4: Not at Random Numeric Target Feature
hot do
Cataggnne
Atttotcc
adjacen:
mode
infil
20002
1530
Rijttct 1229
Figure 5: Not at Random Categoric Target Feature
310
An additional comparable series of injections were conducted with ML infill and the added difference of appending the NArw support columns corresponding the target columns for injections. Again these NArw support columns are the boolean integer markers for presence of infill in the corresponding entries which support the partitioning of sets for ML infill. The expectation was that by using these e
312
313
314
markers to signal to the training operation which of the entries were subjected to infill, there would
315
be some benefit to downstream model performance. For many of the scenarios the visible impact was that supplementing with the NArw support column improved the MI infill performance, demonstrated here for missing at random [Fig 6, 7] and missing not random [Fig 9] with the other imputation
316
317
scenarios shown again for context.
Discussion
319
320
One of the primary goals of this experiment was to validate the efficacy of ML infill as evidenced by improvements downstream model performance. For the Ames Housing benchmark data set, there was a notable demonstration
322
ML infill benefiting model performance in the scenario of the numeric target column with not at random injections at increased injection ratios, and also to 324 a lesser extent with missing at random injections. We speculate an explanation for this advantage towards the numeric target columns may partly be attributed to the fact that the downstream model was also a regression application, so that the other features selected for label correlation may by proxy have correlations with the target numeric feature. The corollary is that the more mundane performance of MI infill toward the categoric target columns may be a result of these having less correspondence with the surrounding features. The fact that even in these cases the MI infill still fell within noise distribution of the other imputation scenarios we believe presents a reasonable argument for defaulting to ML infill for tabular learning.
326
328
329
330
331
332
Note that as another argument for defaulting to ML infill as opposed to static imputations is that the imputation model may serve hedge against imperfections in subsequent data streams, particularly one of the features experiences downtime in streaming application for instance.
334
335
The other key finding of the experiment was the pronounced benefit to downstream model performance when including the NArw support column in the returned data set as a supplement to ML infill. This finding was consistent with our intuition, which was that increased information retention about infill
336
338
points should help model performance. Note there is some small tradeoff, as the added training set dimensionality may increase training time. Another benefit to including NArw support columns may

---

340 be for interpretability in inspection of imputations. We recommend including the NArw support columns for model training based on these findings, with the one caveat that care should be taken to avoid inclusion in the data leakage scenario where there is some kind of correlation between presence 343 of missing data and label set properties that won't 't be present production.
341
Voudi of Kordiin
Iesry
2010
Nitav
detnst
2019
mode
Mine
150
26:00
0.5
00
04
Figure 6: NArw comparison Missing at Random Numeric Target Feature
20300
130
defeutt
7150
7.50
3020
2773
Figure 7: NArw comparison Missing at Random Categoric Target Feature
Met.d Sacker Moxe:
defukk Njjeeet
ned:
WIT
Figure 8: NArw comparison Not at Random
Numeric Target Feature
defsult
acjazen:
mede
NA
Figure 9: NArw comparison Not at Random Cattegoric Target Feature
Conclusion
344
Automunge offers a push-button solution to preparing tabular data for ML, with automated data cleaning operations like normalizations, binarizations, and auto ML derived missing data imputation aka MI. infill. Transformations and imputations are fit to properties of a designated train set, and with application of automunge(.) a compact dictionary is returned recording transformation parameters and trained imputation models, which dictionary may then serve key for consistently preparing additional data on the train set basis with postmunge()).
346
348
350
351 We hope that these experiments may serve as a kind of validation of defaulting to ML infill with supplemented NArw support columns in tabular learning for users of the Automunge library, as even if in our experiments the material benefits towards downstream model performance were not demonstrated for all target feature scenarios, other cases there did not appear to be any material penalty. Note that MI. infill can be activated for push-button operation by the automunge()) parameter MLinfill illTTull and the NArw support columns included by parameter NArw NArw_marrrtaa marker=True,, Based on these findings these two parameter settings now cast defaults for the Automunge platform.
352
353
354
356

---

358
Acknowledgments
359
thank you owed those facilitators behind Stack Overflow, Python, Numpy, Scipy Stats, PyPI, GitHub, Colaboratory, Anaconda, VSCode, and Jupyter. Special thanks to Scikit-Learn and Pandas.
380
References
361
[I] Daniel J. Stekhoven, Peter Bühlmann. MissForest nonparametric missing value imputation for mixed-type data 12011) arXiv:1105.0822
363
364
[2] Author(s) (2021) Automunge, GitHub repository (Please see supplemental material)
385
3J W. McKinney. Data structures for statistical computing in python. Proceedings of the 9th Python Science Conference, pages 51-56, 2010.
366
367 [4] L. Breiman. Random Forests. Machine Learning, 45(1), 2001.
368 [5] Pedregosa et al., Scikit-lcam: Machine Learning Python, JMLR 12, pp. 2825-2830, 2011.
369
[6] H. Wickham. Tidy data, Journal of Statistical Software, 59(10), 2014.
370
77] John Moody, Fast Learning in Multi-Resolution Hicrarchics. NIPS Proceedings, 1989
371
[8] Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, Josh Attenberg. Feature Hashing for Large Scale Multitask Learning. ICML Proceedings, 2009
372
373
[9] Author(s) Hashed Categoric Encodings with Automunge (2020) (Please see preprint in supplemental material)
374
[10] Author(s) Numeric Encoding Options with Automunge (2020) (Please see preprint in supplemental material)
375
[1]] Author(s) Parsed Categoric Encodings with Automunge (2020) (Please see preprint in supplemental
376
material)
377 [12] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. IEEE conference on computer vision and pattern recognition, 2016
[13] Author(s) Specification of Derivations with Automunge (2020) (Please see preprint in supplemental material)
380
381
[14] Stef van Buuren, Karin Oudshhoo.. mice: Multivariate Imputation by Chained Equations (201) I) httpp::/www.....t jtt.taasttttt.ttttt... wttp://rrrrrrreeeeeeieee/e/////e
382
383
[15] Anna Veronika Dorogush, Vasily Ershov, Andrey Gulin, CatBoost: gradient boosting with categorical features support (2018) 11X.v:1810.11134
384
385
[16] Nick Frickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. MattGluun--aabllee Robust and Accurate AutoMI. for Structured Data (2020) anxiv:2003.06505
387
[17] Chi Wang, Qingyun Wu, Markus Weimer, Erkang Zhu. FLAML: A Fast and Lightweight AutoML Library (2019) arX.v:1911.04706
[18] Jinsung Yoon, James Jordon, Mihaela van der Schaar. GAIN: Missing Data Imputation using Generative Adversarial Nets (2018 International Conference of Machine Learning), (2XXv:1806.029))
390
391
[19] Yuxuan Zhao. Madeleine Udell. Missing Value Imputation for Mixed Data Gaussian Copula (KDD 2020), arXiv:1910.1 12845
392
393
[20] K. Mohan, J. Pearl. Graphical Models for Processing Missing Data (2019), arXiv:1801. 1001.03083
394 [21] Olga Troyanskaya, Michael Cantor, Gavin Sherlock, Pat Brown, Trevor Hastic, Robert Tibshirani, David Botseenn and Russ B. Altman. Missing value estimation methods for DNA microarrays, BIOINFORMATICS Pages 520-525.
385
396
Vol. 17 no. 6, 2001
397
[22] Matei Zaharia, Reynold S. Xin, Patrick Wendell, Tathagata Das, Michael Armbrust, Ankur Dave, Xiangrui Meng, Josh Rosen, Shivaram Venkataraman, Michael J. Franklin, Ali Ghodsi, Joseph Gonzalez, Scott Shenker, Ion Stoica. Apache Spark: a unified engine for big data processing. Communications the ACM, 59( 2016
399
400
[23] Dask Development Team. Dask: Library for dynamic task scheduling (2016) attps://dask..
[24] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael Jordan, lon Stoica. Ray: A Distributed Framework for
402
403
Emerging Al Applications. 13th USENIX Symposium on Operating Systems Design and Implementation
(2018), arXiv: 171..0589
10

---

405
[25] Rapids Development Team. Open GPU Data Science I RAPIDS https://rapids. ai
406
[26] Abadi, Martín, Barham P. Chen J. Chen Z, Davis A, Dean J. et al. Tensorflow: A system for large-scale machine learning. 12th USENIX Symposium on Operating Systems Design and Implementation (2016) p.
408
265 83.
409
[27] Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Kopf, Andreas and Yang, Edward and Vito,o, Zachary and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang. Lu and Bai, Junjie and Chintala, Soumith. PyTorch: An Imperative Style, Prrfh-Perffrmance Deep Learning Library. NeurIPS Proceedings, 2019
410
411
412
413
[28] Terrence Parr, Kerem Turgutlu, Christopher Csiszar, and Jeremy Howard. Beware default random forest importances. Explained.ai (blog), 2018. httpe: httpe://expl. //pppe////alaadddd... ai/rf- -erf-iiitottaeee////
415
416
[29] Author(s) Automunge Influence (2020) (Please see preprint in supplemental material)
417
[30] Dean De Cock. Ames, Iowa: Alternative to the Boston Housing Data as an End of Semester Regression Project, Journal of Statistics Education, Volume 19, Number 3 (2011)
418
419
[31] Kaggle: Allstate Claims Severity, https: / //www. http://wwwk.gg.....//
Checklist
421
1. For all authors...
422
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
424
(b) Did you describe the limitations of your work? [Yes] Please see discussions in section 5 Related Work
425
428
(c) Did you discuss any potential negative societal impacts of your work? [Yes] A Broader Impacts discussion is provided as Appendix C
427
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
429
2. If you are including theoretical results...
430
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
432
3. If you ran experiments...
433
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [Yes] Please see jupyter notebooks provided with supplemental material
434
435
436
(b) Did you specify all the training details (e.g., data splits, hyperparameters. how they were chosen)? [Yes] We noted that missing data injections were random for each trial. Performance was evaluated on 25% validation split. We used hyperparameter defaults for learning.
437
438
439
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [Yes] We noted that since scenarios for 0% injection are comparable hetween imputation methods, their spread may serve as proxy for noise inherent in the operation.
440
442
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [No] Our experiments did not require significant compute.
443
444
4. If you are using existing assets (e.g.. code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
446
(b) Did you mention the license of the assets? [Yes] We note licenses of supporting packages in the
447
read me document included in the github repository folder within the supplemental material.
448
(c) Did you include any new assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent obtained from people whose data you're 're us-
450
ing/curating? [N/A]
451
(c) Did you discuss whether the data you are using/curating contains personally identifiable informa- tion or offensive content? [N/A]
453
5. If you used crowdsoureing or conducted research with human subjects...
454
(a) Did you include the full text of instructions given to participants and screenshols, if applicable?
455
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
457
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
11


Reviews:
[{"review_id": "u9pVDrLxTKV", "reviewer": "Reviewer_mGQe", "summary": "The paper presents the automunge library for missing data imputation and preprocessing for supervised learning in Python.\nThe automunge library implements ML-based infill similar to the established MissForest and MICE algorithms.\n\nThe authors discuss several imputation methods, as well as the automunge API, and evaluate the impact of different imputation strategies on two regression datasets, using a CatBoost model as the final machine learning model.\n\nThe authors conclude that ML infill with an indicator feature for missing values (called NArw in paper) achieves the best performance for continuous features, and that for categorical features, that all methods perform similarly.", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the authors address a really important problem. As far as I know, there is no extensive empirical evaluation of imputation methods for supervised learning, a gap that is crucial to close.\nHowever, by restricting the evaluation to two regression datasets, I don't think it's possible for the authors to make very general claims. For the contribution to be empirical, I would expect at least a dozen datasets, ideally CC-18 or OpenML 100 for classification, and hopefully a similar set for regression.\n\nOverall, the contributions of the work are not spelled out very clearly. There seem to be no significant methodological improvements over standard algorithms like MissForest. There is a discussion of the software library, however, the related work does not properly discuss related software work and how automunge fits into the ecosystem, so it seems not to be the focus of the contribution.\nFinally, the empirical evaluation is very limited, as mentioned above.\n\n## Detailed comments about software and related work\nThe core of automunge is ML-based imputation, which is actually implemented in scikit-learn as IterativeImputer.\nThis is not mentioned in the paper, even though it has been part of the library for a significant amount of time.\nIt's unclear what benefits automunge has over the implementation in scikit-learn. For the implementation or methodology to be a  contribution, this needs to be addressed.\n\nAutomunge seems to also allow automatic featurization, but there is very little detail about this in the paper, and there is no empirical evaluation.\n\nIt's unclear to me why automunge does not adhere to the standard scikit-learn interface for transformations (or estimators). In particular, given the interface as described in the paper, it's unclear to me how to use any of the standard cross-validation and grid-search utilities in scikit-learn.\n\n## Notes on empirical evaluation\nIt's unclear how many iterations of ML infill have been applied for the benchmarks; I assume it's the default of 1.\nI think one of the most critical questions is what model to use for infill, and what stopping criterion to use.\nMissForest uses a somewhat odd heuristic for determining when to stop iterative infill. Scikit-learn uses the more natural heuristic of MICE, but this has been observed to never converge when using Random Forest base models.\nOf course it is always possible to grid-search the number of iterations, but that is not an ideal outcome.\n\nThere are no error bars on the plots, which makes it hard to understand how much noise is in these observations. Given the ups and downs, I assume the plots to be quite noisy.\n\nFinally, it's not clear why you would suggest ML infill for categorical variables if encoding it as a separate category behaves as well.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't see potential negative societal impact.\n\nThey authors did not address the limitations of their empirical evaluation, which I consider the main contribution of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "8PYbnJ-kTBi", "reviewer": "Reviewer_uuhM", "summary": "This paper introduce a method called automunge to impute the missing entries in the data.\nThe imputation model is learned from the data and then the automunge imputes the missing values based on the learned model.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I appreciate that the authors provided a nice summary of their method, there are many critical issues in missing data that were not answered. \n1. [What is the underlying missing data assumption?] \nA major issue is: an imputation model reflects our belief/assumption/model on the missing entries given the observed data. Different assumptions lead to different imputation distributions. This paper does not clearly indicate what is the underlying assumptions that they were making, so it is difficult to believe the imputed values. \n\n\n2. [Deterministic vs Stochastic imputation]\nWhen applying an imputation approach, we may impute the missing entries deterministically or stochastically. The later is generally preferred since deterministic imputation often leads to bias in the downstream analysis. See, e.g., \n\nLittle, R. J., & Rubin, D. B. (2019). Statistical analysis with missing data (Vol. 793). John Wiley & Sons.\n\nThis should be discussed in the paper.\n\n\n3. [Lack of details on the imputation method]\nThere is no details about which ML model to be used for imputation. \nBut this is a critical part of the analysis--different model reflects different assumptions on the missingness. \n\n\n4. [Lack of analysis for the automunge]\nThe authors only describe how automunge works but there is no analysis on this approach. For a top ML conference, I don't think this is acceptable.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper is too shallow that many important components and analysis were not explored. See my comments in the main review. I would recommend the authors to provide more details on the derivations, intuitions, and analysis on the automunge.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6Ivi1GlGjMh", "reviewer": "Reviewer_i1mF", "summary": "The paper introduces a library for data wrangling. This include, in particular, feature process and missing value imputation. The library is described and demonstrated to be effective at the targeted tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well written paper that introduce a software library. It does not represent a novel scientific contribution. It also lacks comparison to competing methods, e.g. for missing value imputation. \n\n## Some Detailed Comments\n- In the abstract the authors \"recommend\" using the library they built. This seems to be a largely subjective judgement and is misplaced for a NeurIPS paper.\n- The title is not great, does not give any information other than that the problem is about missing data imputation.\n- l 24: The definition of \"missing at random\" seems wrong. What about \"missing completely at random\"?\n- [DataWig](https://www.jmlr.org/papers/volume20/18-753/18-753.pdf) seems to be direct competition that is not considered/cited. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Possible problems with missing value imputation are not addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "1aUuhruTk1", "reviewer": "Reviewer_Fgob", "summary": "The paper introduces ‘ML infill’ , which is a Python implementation of a sequential imputation technique. More precisely, this method iteratively considers each feature $X_j$ with missing values as a target feature, fits a predictive model based on the remaining features as input variables and the observed values of $X_j$ as labels, and finally predicts the missing entries of $X_j$. ‘ML infill’ is provided as part of an open source Python library, Automunge, whose goal is to prepare data to feed it to a machine learning algorithm (e.g. feature transformation, and imputation of missing values). The paper provides experiments comparing the impact of basic imputations on downstream regression tasks, and recommend defaulting to ‘ML infill’ and using the missingness indicator as additional predictor.", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strengths**\n\nThe paper adresses an important problem for the application of machine learning in real-world settings: learning with messy tabular data. The proposed library is an effort to provide practitioners an automatic one-step preprocessing of messy data (notably when missing values are present) into tidy data that can be fed to off-the-shelves machine learning models.\n\n**Weaknesses**\n\nAs noted by the authors in the Related work, sequential imputation methods that model each feature with missing values as a function of the others are not new (e.g. Mice, missForest, …).\nMore importantly, it seems to me that they are already available in Python thanks to Scikit-Learn’s IterativeImputer. In particular, similarly to the options provided by ‘ML infill’, IterativeImputer needs only be trained on the train set, can use a wide range of machine learning models for imputation, and also provides an option for stacking the missingness indicator with the imputed values.\n\nConcerning the experiments, I think they are not extensive enough to provide recommendations with regards to the type of imputation method that’s best for downstream learning (only two datasets with 15 features selected). Also, it has already been observed by a number of works that in missing not at random settings, adding the missingness indicator as additional features can improve predictions (see for ex [1], [2], [3]).\n\nFor these reasons (availability of IterativeImputer in Scikit learn, weakness of experimental benchmark), I recommend rejection.\nAlso, it seems to me that since the imputation method is not novel, and that its implementation does not imply particular coding tricks, there is little novelty in the contribution.\n\nMinor comment: l24-25, the missing data mechanism described is missing completely at random, which is different from missing at random.    \n[1] Sharafoddini et al A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study.  \n[2] Sperrin et al Missing data should be handled differently for prediction than for description or causal explanation.  \n[3] Josse et al On the consistency of supervised learning with missing values.  ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "-", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents the automunge library for missing data imputation and preprocessing for supervised learning in Python. It has limited novelty. It tries to address the problem that currently there is no extensive empirical evaluation of imputation methods for supervised learning. However, it does not clearly indicate what are the underlying data assumptions, and lack details on the imputation methods. Empirical evaluation is performed on only two regression datasets. This is very limited and its conclusions are not convincing. It also lacks comparison to competing methods.
Decision: Reject