Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: summary: The paper introduces a library for data wrangling. This include, in particular, feature process and missing value imputation. The library is described and demonstrated to be effective at the targeted tasks.
main_review: This is a well written paper that introduce a software library. It does not represent a novel scientific contribution. It also lacks comparison to competing methods, e.g. for missing value imputation. 

## Some Detailed Comments
- In the abstract the authors "recommend" using the library they built. This seems to be a largely subjective judgement and is misplaced for a NeurIPS paper.
- The title is not great, does not give any information other than that the problem is about missing data imputation.
- l 24: The definition of "missing at random" seems wrong. What about "missing completely at random"?
- [DataWig](https://www.jmlr.org/papers/volume20/18-753/18-753.pdf) seems to be direct competition that is not considered/cited. 
limitations_and_societal_impact: Possible problems with missing value imputation are not addressed.
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: The paper presents the automunge library for missing data imputation and preprocessing for supervised learning in Python.
The automunge library implements ML-based infill similar to the established MissForest and MICE algorithms.

The authors discuss several imputation methods, as well as the automunge API, and evaluate the impact of different imputation strategies on two regression datasets, using a CatBoost model as the final machine learning model.

The authors conclude that ML infill with an indicator feature for missing values (called NArw in paper) achieves the best performance for continuous features, and that for categorical features, that all methods perform similarly.
main_review: I think the authors address a really important problem. As far as I know, there is no extensive empirical evaluation of imputation methods for supervised learning, a gap that is crucial to close.
However, by restricting the evaluation to two regression datasets, I don't think it's possible for the authors to make very general claims. For the contribution to be empirical, I would expect at least a dozen datasets, ideally CC-18 or OpenML 100 for classification, and hopefully a similar set for regression.

Overall, the contributions of the work are not spelled out very clearly. There seem to be no significant methodological improvements over standard algorithms like MissForest. There is a discussion of the software library, however, the related work does not properly discuss related software work and how automunge fits into the ecosystem, so it seems not to be the focus of the contribution.
Finally, the empirical evaluation is very limited, as mentioned above.

## Detailed comments about software and related work
The core of automunge is ML-based imputation, which is actually implemented in scikit-learn as IterativeImputer.
This is not mentioned in the paper, even though it has been part of the library for a significant amount of time.
It's unclear what benefits automunge has over the implementation in scikit-learn. For the implementation or methodology to be a  contribution, this needs to be addressed.

Automunge seems to also allow automatic featurization, but there is very little detail about this in the paper, and there is no empirical evaluation.

It's unclear to me why automunge does not adhere to the standard scikit-learn interface for transformations (or estimators). In particular, given the interface as described in the paper, it's unclear to me how to use any of the standard cross-validation and grid-search utilities in scikit-learn.

## Notes on empirical evaluation
It's unclear how many iterations of ML infill have been applied for the benchmarks; I assume it's the default of 1.
I think one of the most critical questions is what model to use for infill, and what stopping criterion to use.
MissForest uses a somewhat odd heuristic for determining when to stop iterative infill. Scikit-learn uses the more natural heuristic of MICE, but this has been observed to never converge when using Random Forest base models.
Of course it is always possible to grid-search the number of iterations, but that is not an ideal outcome.

There are no error bars on the plots, which makes it hard to understand how much noise is in these observations. Given the ups and downs, I assume the plots to be quite noisy.

Finally, it's not clear why you would suggest ML infill for categorical variables if encoding it as a separate category behaves as well.

limitations_and_societal_impact: I don't see potential negative societal impact.

They authors did not address the limitations of their empirical evaluation, which I consider the main contribution of this work.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper introduce a method called automunge to impute the missing entries in the data.
The imputation model is learned from the data and then the automunge imputes the missing values based on the learned model.
main_review: While I appreciate that the authors provided a nice summary of their method, there are many critical issues in missing data that were not answered. 
1. [What is the underlying missing data assumption?] 
A major issue is: an imputation model reflects our belief/assumption/model on the missing entries given the observed data. Different assumptions lead to different imputation distributions. This paper does not clearly indicate what is the underlying assumptions that they were making, so it is difficult to believe the imputed values. 


2. [Deterministic vs Stochastic imputation]
When applying an imputation approach, we may impute the missing entries deterministically or stochastically. The later is generally preferred since deterministic imputation often leads to bias in the downstream analysis. See, e.g., 

Little, R. J., & Rubin, D. B. (2019). Statistical analysis with missing data (Vol. 793). John Wiley & Sons.

This should be discussed in the paper.


3. [Lack of details on the imputation method]
There is no details about which ML model to be used for imputation. 
But this is a critical part of the analysis--different model reflects different assumptions on the missingness. 


4. [Lack of analysis for the automunge]
The authors only describe how automunge works but there is no analysis on this approach. For a top ML conference, I don't think this is acceptable.


limitations_and_societal_impact: This paper is too shallow that many important components and analysis were not explored. See my comments in the main review. I would recommend the authors to provide more details on the derivations, intuitions, and analysis on the automunge.
review_title: null

final_score: rating: 2: Strong rejection
review_content: summary: The paper introduces ‘ML infill’ , which is a Python implementation of a sequential imputation technique. More precisely, this method iteratively considers each feature $X_j$ with missing values as a target feature, fits a predictive model based on the remaining features as input variables and the observed values of $X_j$ as labels, and finally predicts the missing entries of $X_j$. ‘ML infill’ is provided as part of an open source Python library, Automunge, whose goal is to prepare data to feed it to a machine learning algorithm (e.g. feature transformation, and imputation of missing values). The paper provides experiments comparing the impact of basic imputations on downstream regression tasks, and recommend defaulting to ‘ML infill’ and using the missingness indicator as additional predictor.
main_review: **Strengths**

The paper adresses an important problem for the application of machine learning in real-world settings: learning with messy tabular data. The proposed library is an effort to provide practitioners an automatic one-step preprocessing of messy data (notably when missing values are present) into tidy data that can be fed to off-the-shelves machine learning models.

**Weaknesses**

As noted by the authors in the Related work, sequential imputation methods that model each feature with missing values as a function of the others are not new (e.g. Mice, missForest, …).
More importantly, it seems to me that they are already available in Python thanks to Scikit-Learn’s IterativeImputer. In particular, similarly to the options provided by ‘ML infill’, IterativeImputer needs only be trained on the train set, can use a wide range of machine learning models for imputation, and also provides an option for stacking the missingness indicator with the imputed values.

Concerning the experiments, I think they are not extensive enough to provide recommendations with regards to the type of imputation method that’s best for downstream learning (only two datasets with 15 features selected). Also, it has already been observed by a number of works that in missing not at random settings, adding the missingness indicator as additional features can improve predictions (see for ex [1], [2], [3]).

For these reasons (availability of IterativeImputer in Scikit learn, weakness of experimental benchmark), I recommend rejection.
Also, it seems to me that since the imputation method is not novel, and that its implementation does not imply particular coding tricks, there is little novelty in the contribution.

Minor comment: l24-25, the missing data mechanism described is missing completely at random, which is different from missing at random.    
[1] Sharafoddini et al A New Insight Into Missing Data in Intensive Care Unit Patient Profiles: Observational Study.  
[2] Sperrin et al Missing data should be handled differently for prediction than for description or causal explanation.  
[3] Josse et al On the consistency of supervised learning with missing values.  
limitations_and_societal_impact: -
review_title: null

Meta-review: The paper presents the automunge library for missing data imputation and preprocessing for supervised learning in Python. It has limited novelty. It tries to address the problem that currently there is no extensive empirical evaluation of imputation methods for supervised learning. However, it does not clearly indicate what are the underlying data assumptions, and lack details on the imputation methods. Empirical evaluation is performed on only two regression datasets. This is very limited and its conclusions are not convincing. It also lacks comparison to competing methods.
Decision: Reject