Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: This paper proposes a framework based on principle components analysis (PCA) to speed up the missing data imputation. It divides the feature sets into two partitions -- the fully observed one and the one that contains missing values. The proposed method applies PCA to the fully observed partition to do dimensionality reduction, followed by the existing imputation methods. The authors further propose to apply PCA to the imputed data to speed up the downstream classification task.
strengths_and_weaknesses: Strengths:
- The paper is overall organized and readers can easily follow.

Weaknesses:
- The major weakness is that the methodological contribution is quite limited. Projecting data into lower dimensional spaces to speed up downstream tasks is not new. In fact, it has been widely regarded as the go-to tool to handle large datasets.
questions: A suggestion for further improvement: the proposed method only applies PCA to the fully observed partition, yet in many important cases, either we have no prior information about which feature is fully observed, or the feature partition with missing value is very large. If the PCA variant to be proposed could directly apply to the partition with missing values, it would much more interesting. Of course, it has to come with theoretical and empirical validations.
limitations: Limitations are not discussed in this paper.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors presented two novel frameworks, PCAI and PIC, that allow for more efficient handling of missing data. There are two use cases, fashion MNIST (imaging data) and Parkinson (voice recording) dataset. 
strengths_and_weaknesses: Strength: easy to understand and implement. 
Weakness: The main assumption is that introducing random missing is not acceptable without sufficient evidence and justification. 
questions: Testing and optimizing imputation is complex and unless one is taking sufficient steps, the results may be biased, and the study will have limited utility and generalizability. 
A comparison of a fashion dataset (in this case imaging data) and a medical dataset (in this case, voice recording) cannot be only about the number of records and number of features. To impute one has to understand the data characteristics, beyond simple statistics. A Deeper discussion of these differences is important if such comparisons are made. 
limitations: The main assumption is that introducing random missing is not acceptable unless the author can show that such data suffers from random missing. For instance, clinical data has missing that is not completely random. Please see the paper published in nature digital medicine on this topic. https://www.nature.com/articles/s41746-021-00518-0 
Similarly, voice recordings and imaging data suffer likely from non at random missingness due to the nature of the data.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors present a PCA based framework for missing data imputation specially efficient for high dimensional data. The experiments indicate that the imputation presents low MSE and, when applied to classification tasks, results in similar or better accuracy values.
strengths_and_weaknesses: The main motivation is relevant and promising. The related work section indicates a well explored field. Nevertheless, the authors should mention and include in the experimental section at least some of the many PCA based approaches to data imputation. See for instance [1,2,3,4] and the references within. Without those comparisons, it is difficult to state the originality of the work.

Since the proposed approach relies on classic PCA and can be coupled with any imputation strategy, it seems to be readily available for practitioners. This is a welcomed advantage of the proposal.

The writing is mostly sound and the manuscript is well organized. However, some steps of the proposed algorithms are difficult to follow due to the lack of details. I will list them in the next part.

References

[1] Qu, Li, et al. "PPCA-based missing data imputation for traffic flow volume: A systematical approach." IEEE Transactions on intelligent transportation systems 10.3 (2009): 512-522.

[2] Folch-Fortuny, Abel, Francisco Arteaga, and Alberto Ferrer. "PCA model building with missing data: New proposals and a comparative study." Chemometrics and Intelligent Laboratory Systems 146 (2015): 77-88.

[3] Folch‐Fortuny, Abel, Francisco Arteaga, and Alberto Ferrer. "Assessment of maximum likelihood PCA missing data imputation." Journal of Chemometrics 30.7 (2016): 386-393.

[4] D’Enza, Alfonso Iodice, Francesco Palumbo, and Angelos Markos. "Single Imputation Via Chunk-Wise PCA." Data Analysis and Rationality in a Complex World (2021): 75.
questions: Some details of the proposed algorithms and experiments can be improved:

- Although it is not explicit in the text, I have inferred that the so-called "partition with missing values" correspond to features with unknown values (columns of the dataset), instead of samples with unknown feature values (rows of the dataset). Am I correct?
- In Algorithm 1, how exactly "R U M" is handled? Column-wise concatenation?
- How exactly the missing data was generated? It is not clear if a 20% missing rate is related to 20% instances with a single random missing feature or if it means that 20% of the total features are missing.
- In the classification experiments, how many data points are used for training and testing? How many independent runs? How hyperparameter selection was performed?

I do not think it is necessary to separate the PCAI and PIC frameworks. It seems to me that PIC is the straightforward application of PCAI on data related to a classification task. Unless I am mistaken, I recommend unifying the presentation of both strategies.

I list additional comments below:

- I believe "Principal Components Analysis" is preferred over "Principle Components Analysis".
- Line 117: "We first conduct dimension reduction on the fully observed partition M..." -> 'F' is the fully observed partition.
- Line 151: The matrix 'W' has not been defined at this point.
- Tab. 4 should include the standard result obtained by the SVM classifier, without PCA.
- I recommend always including in the supervised experiments the simple baseline of simply removing the instances with any missing data.
limitations: The manuscript states sufficiently its limitations.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors propose we propose a framework based on PCA to speed up the imputation process of many available imputation techniques. The proposed methods are named PCAI and PCA-PIC. Experiments are provided on various datasets.
strengths_and_weaknesses: Weaknesses:
1) The authors only compared PCAI strategy with the traditional strategy.
2) "Principle Component Analysis" should be "Principal Component Analysis"

Strengths:
The proposed methods are named PCAI and PCA-PI are novel.
questions: Can the authors explain why the traditional style of imputing on the original missing dataset gives NAs for all missing rates?
limitations: The authors addressee adequately the limitaions of  PCAI and PCA-PI.
review_title: null

Meta-review: metareview: This paper proposes a framework based on principle components analysis (PCA) to speed up the missing data imputation. It divides the feature sets into two partitions -- the fully observed one and the one that contains missing values. The proposed method applies PCA to the fully observed partition to do dimensionality reduction, followed by the existing imputation methods. The authors further propose to apply PCA to the imputed data to speed up the downstream classification task.

The major weakness is that the methodological contribution is quite limited. Projecting data into lower dimensional spaces to speed up downstream tasks is not new. In particular, the main assumption of random missingness has been considered before 10-20 years ago and the more challenging setting of non-random missingness was not considered. Overall the reviewers mostly agree that the contribution is limited.
confidence: Certain
Decision: Reject