Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to jointly optimize imputation and regression to learn from data with missing values, backed by theoretical analysis of impute-then-regress procedures. In particular, the authors show that the impute-then-regress procedure is asymptotically Bayes optimal for almost all (smooth) imputation functions, regardless of the missing data mechanism (including MNAR). However, apart from restrictive special cases, learning the regression function for conditional imputation to obtain a Bayes optimal function is hard due to discontinuities; similar with correcting the imputation for a given oracle function. The authors thus propose to jointly optimize impute-then-regress procedures, in particular chaining a NeuMiss architecture with an MLP. The proposed approach is empirically evaluated on synthetic data with various combinations of imputation and regression methods.
main_review: The paper considers an important problem of learning with missing data, under a general setting with little assumptions on the oracle function or the missingness mechanism, and thus can be relevant in many practical scenarios.

It is also well-written overall. One suggestion for clarity is to formally define “almost all” in Theorem 3.1 sooner.

This work contains interesting and novel theoretical results. In particular, I was surprised that impute-then-regress is Bayes optimal even under MNAR for almost all imputation functions, including a simple constant function. I believe Proposition 4.1 is also an important result, because many existing works on imputation implicitly or explicitly target to learn the conditional imputation and oracle labeling function. 

However, I was less convinced by the conclusion to jointly optimize imputation and regression. According to the authors, for a fixed imputation function, the Bayes optimal regression function may not be continuous, and vice versa. It was not clear how joint optimization addresses this, and whether the joint learning could be confused. The specific choice of NeuMiss was also not very well motivated.

Alternatively, if the goal is not to learn the “best” imputation and “best” regression functions but rather to chain two architectures to optimize predictive performance, how does it compare to simply optimizing one architecture that takes as input the given observations concatenated with missingness information? 

Another recommendation is to also evaluate on real-world benchmark dataset to compare against state-of-the-art learning from missing data.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: In handling missing data, a standard approach is to simply impute and then run a prediction model that requires fully-revealed data. However, this standard "impute-then-regress" approach is largely a hack without theoretical grounding. This paper provides theory to justify this standard approach, with a surprising finding that it turns out that for almost all imputation functions, impute-then-regress with a sufficiently powerful learner is Bayes optimal. Moreover, the paper theoretically justifies why imputation and regression should be done jointly rather than separately as doing so separately can require learning discontinuous functions that are harder to learn. Numerical experiments corroborate the theoretical findings.
main_review: Overall, this paper is very well-written and I find it to constitute a major advance in our understanding of the standard impute-then-regress approach (which shows up in a massive number of applied papers throughout scientific literature but lacks theoretical grounding). The theoretical findings (summarized in my summary above) are surprising and informative of the direction researchers should shift toward in handling missing data.

Strengths:
- extremely well-motivated paper
- compelling theoretical and experimental results
- detailed numerical experiments

Weaknesses:
- especially as the claim of Theorem 3.1 is surprising regarding the "almost all" imputation methods, perhaps it would be instructive providing an example problem setup and imputation method that does not lead to Bayes optimality and relating such an example to what is done in practice (perhaps such an example is contrived and basically all practical imputation methods really do fall under the coverage of Theorem 3.1)
limitations_and_societal_impact: The authors have provided a sufficient answer (see checklist 1(c)).
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper first seeks to study the theoretical properties of impute-then-regress procedures -- which are widely used. They show that most any imputations admit Bayes consistent estimators; however, that estimation may be difficult in certain settings. They show that conditional imputation is not necessarily an optimal strategy, supply bounds on the associated risk, and discuss when it can be expected to perform reasonably well. In particular, conditional imputation under some conditions requires learning a discontinuous regression function for optimal performance. Given all this, they examine the conditions under which imputations exist such that impute-then-regress procedures are optimal. Lastly, they propose a procedure for joint optimization of an imputation and a regression scheme that is empirically shown to 1. outperform common impute-then-regression procedures as well as gradient boosting which naturally can accommodate missingness and 2. perform similarly to oracle estimators in certain settings. 
main_review: **Originality**: I was not familiar with general work on the validity of impute-then-regress schemes, though this submission seems to clearly move beyond the restrictive settings considered in past papers. In general, there is not much related work cited on this concrete task (not that I am sure more exists). The actual method proposed to take advantage of the demonstrated theory -- NeuMiss+MLP -- is not very innovative, though it is not meant to be; the point is simply that joint optimization of how to impute and how to regress is superior. The proof technique of Theorem 3.1. seems to be original, though I did not examine it carefully. 

**Quality**: Claims are backed up by theoretical results, the proofs of which I did not examine closely. There is empirical support for the overall argument of the paper. Thought it is not the focus, it might have been good to touch upon the limitations of NeuMiss. I would also have liked to see a discontinuous $f^*$, though I understand that the Bayes predictor might be harder to compute.

**Clarity**: I thought the paper read very well. A few typos to address before CR, but I like the overall structure. 

**Significance**:  I don't think the prevalence of "first impute, then regress" as a strategy for handling missing data can be understated and for this reason I find the work very compelling. It is possible that in the future the standard shifts towards a joint treatment of the two tasks and this paper provides evidence for why that would be beneficial. It would have been useful to comment on joint schemes beyond the one proposed here. For example, do researchers that don't want to rely on an uninterpretable method for regression have any hope of adopting / adapting the approach here? 
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors look at the impute and regress framework and provide arguments as to Bayes optimality of the process for missing data mechanisms and imputation functions. It gives a general theoretical framework for the impute and regress procedures. Unlike the missing at random setups, they look at more general missing value mechanisms. They also argue that the imputation and regression processes should be properly aligned with each other and also provide a method for optimizing the two processes jointly.
main_review: The paper is well written and provides a well-structured overview of the results. They provide a framework that gives a theoretical background for the impute-then-regress process and they look at a more general setup that does not require any structured missing data mechanisms such as MAR and etc. The authors show that for almost all infinitely differentiable, smooth impute functions, there is an impute-then-regress process that is Bayes optimal. They provide a constructive argument to that effect. The paper also looks at the case of using conditional expectations for the imputation and how this could result in regression functions with discontinuity. The authors also provide a chaining of NeuMiss and multi-layer perceptron to compute the impute-then-regress process. All the claims are clearly stated and properly presented and the math seems to hold as well.
limitations_and_societal_impact: The authors do state that the choice of imputation function impacts the regression step and how the continuity and smoothness are not guaranteed making it hard to approximate. However, as stated in the previous section, they look at this case with the imputation being conditional expectation and also show the existence of continuous imputation functions that lead to Bayes optimality.
review_title: null

Meta-review: The paper studies prediction in the presence of missing values. A common approach to solving this problem is "impute-then-regress", independently imputing missing values and predicting using the imputed data. The authors show that this strategy is almost always Bayes optimal w.r.t. risk, but that a poor imputation may lead to learning a complex prediction function. Several strategies are evaluated and compared empirically on synthetic data. 

All reviewers recognised the value of this work and pointed out its strong motivation and clarity in exposition. 
Decision: Accept (Spotlight)