Kati8/ICLR_2023_2025 huggingface dataset:
authors: Liancheng Fang
authors: Qitian Wu
authors: Philip S. Yu
reviews: 
Comments: The authors combine diffusion processes and Expectation-Maximization (EM) algorithms to propose a novel way to impute missing data when both training and tests sets contain missing data. The proposed solution is shown to target the correct conditional distribution (distribution of missing data conditional on observed ones). Imputation values are computed by taken the expectation with respect to this distribution, which is approximated by the sample mean. Experiments on ten real-world data sets show the benefit of the proposed method, compared to various state-of-the-art imputation algorithms (machine and deep learning algorithms).
Strengths: The authors propose a new method to impute missing data for continuous and discrete inputs. The proposed method appears to be new, with excellent performances. An extensive literature review has been done to present and explain the previous approaches to deal with missing values via imputation. The method is clearly explained, the paper well-written, and the experiments show the benefit of the proposed method.
Weaknesses: I only have three remarks: 
- RMSE and MAE are measures that encourage the imputation to target the (conditional) mean or median. In both cases, the target is not a distribution but a single quantity. Recent works (https://arxiv.org/abs/2002.03860) have shown that such measures do not properly evaluate the correctness of an imputation method. Imputation score (https://arxiv.org/pdf/2106.03742) can be used instead to assess the quality of imputations. As the proposed method generates a distribution and not a single point estimate, it is likely that its performance will be higher with respect to this metric, showing that it is able to recover the underlying distribution of the data. Presenting imputation scores in the tables would definitely improve the strength of the paper, in my opinion.
- The computational performances of DiffPuter should be discussed in the main text. Table 4 is interesting, as it shows that the training time is larger, but not too important. However, the two considered data sets have few features. It would be appealing to consider larger data sets with (i) more observations and/or (ii) more variables to see how the predictive performances and the training time behave. 
- I have trouble understanding the proof of Theorem 1. Notations are confused to me. Adding a table of notations, with exact definitions at the beginning of the Appendix would help. Besides, many approximations are done in the proof : l.730, 731, 750, 753. This results in the theorem being imprecise. For example, nothing is assumed about the quality of the neural network $\varepsilon_{\theta}$. What type of convergence is required for Theorem 1 to be valid? Similarly, in Theorem 1, $\sigma(T)$ is not assumed to be large, whereas it is required in the proof. Please clarify the different assumptions and the proof.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: This paper introduces an adaptation of the EM algorithm for missing data imputation, leveraging advanced diffusion-based models to perform precise density estimation in the M-step and provide robust imputations in the E-step, inspired by the RePaint algorithm. The authors reference theoretical analyses from prior work to support the use of diffusion models for density estimation and prove a theorem demonstrating that E-step samples can be drawn from the true conditional distribution. Extensive empirical evaluations highlight the proposed method’s robustness and superiority over various baseline approaches, many of which do not incorporate the EM algorithm.
Strengths: - Robust imputation method based on EM. 
- Well written and structured. 
- The method is theoretically grounded. 
- The empirical analysis is extensive.
Weaknesses: - Motivation appears to overlook recent work.
- Experimental section lacks fair comparison and clarity.
- Discussion of limitations is lacking.
- Given these weaknesses, the contribution is not strongly justified.


------- Post rebuttal update -------

All the weaknesses were thoroughly addressed in the rebuttal provided by the authors. I appreciate their efforts and the detailed responses, which resolved all my concerns.
Recommendation: 8
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: The missing value imputation is a very important problem in both machine learning and statistics. Although deep generative imputation methods have shown promise, there is still substantial room for improvement, even for matching the performance of some traditional machine learning approaches. This paper introduces DIFFPUTER, a tailored diffusion model combined with the Expectation-Maximization (EM) algorithm for missing data imputation, and shows its promising performance on a variety of datasets,
Strengths: 1. Theoretical analysis: DIFFPUTER’s training step corresponds to the maximum likelihood estimation of data density (M-step), and its sampling step represents the Expected A Posteriori estimation of missing values (E-step). 
2. Extensive experiments that demonstrate the good performance, as compared with existing baselines, of the proposed method across various datasets.
Weaknesses: the computational complexity is not explicitely discussed or compared on the numerical experiments, see details below.
Recommendation: 8
Reviewer confidence: 3
Soundness/correctness: 3
Clarity: 3
reviews: 
Comments: The paper addresses the imputation of data missing completely at random (MCAR) in tabular data, handling both continuous and categorical variables. The authors propose an EM procedure with a conditional diffusion model for imputation, featuring a novel adaptation of the annealing process for better conditioning on observed values. The paper demonstrates strong results across multiple datasets in comparison with leading methods.
Strengths: * The paper is well-written with a clear, thorough, and concise introduction that effectively summarizes key points from previous works
* The authors specifically address the challenges of the problem and provide clever solution to mitigate them
* The paper's main novelty is supported by theoretical proof
* The evaluations are comprehensive, with thorough and convincing ablation studies
Weaknesses: * A major concern regarding evaluations - while the paper claims to use a single hyperparameter setting throughout, it's unclear how hyperparameters for other methods were selected and their sensitivity to these HP. For me, this concern significantly impacts the overall assessment of the paper.

* While the results are impressive, their importance is not clear. A more convincing evaluation would include the effect on downstream tasks, given imputation is only a first step in most pipelines. 

* Another point regarding evaluations, is the sole focus on data missing completely at random. While the MER assumption is important, it is the MNER which is a primary focus in many imputation methods.

* The 0/1 continuous encoding of categorical data is unusual, given that binary data is a known challenge for diffusion models (for example in fields like graph generation). Also, the use of mean is inherently problematic due to common multi-modality in the data

* The novelty of the method compared to other approaches is not clearly articulated in the related works section

### Smaller Issues
* Given the method's novelty isn't specific to tabular data, the related work should include other imputation methods (e.g., image inpainting)
* A simulation study with multiple modes would be valuable, particularly as diffusion-based models should excel in such scenarios
* Despite highlighting the importance of initialization in the EM procedure, the paper doesn't address this point. (Particularly relevant given the naive initial imputation approach)
* It would be interesting to analyze the relationship between delta_t size and ML solution approximation.
* Figure 4 lacks clarity
Recommendation: 6
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3