Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This paper proposes a method for simultaneously imputing data and training a classifier. Here are my detailed comments:

1. The proposed method is a simple combination of dictionary learning and classification. As can be seen from (5), J_1 and J_2 are the loss function for dictionary learning, and J_0 is the empirical risk for classification (J_0). I did not see much novelty here.

2. There exists a capacity mismatch between data imputation and classifier. The paper claims that the method is proposed for training deep neural networks. However, deep neural networks are capable of modeling very complex distributions (e.g., images), and the dictionary learning only considers a very simple bilinear factorization, which is not of the same modeling capability as neural networks. 

3. The experimental evaluations are very weak. Table 1 even does not provide any meaningful baseline methods, e.g., the so-called "sequential method".

4. Theorem 3.2 requires the dictionary to be RIP, which is a very strong assumption and unlikely to hold in practice, especially when considering D is actually trained by minimizing (5). The proof of Theorem 3.2 is very elementary, ad-hoc, and does not provide any insight to the problem.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This manuscript studies supervised learning with incomplete observation of the features, assuming a low-rank structure in the full set of features. The work tackles the problem with a global cost function that optimizes the classifier on observations reconstructed with a dictionary penalized jointly for sparse coding. Importantly, the dictionary is optimized both for the supervised-learning task (the classifier) and the sparse coding. The manuscript contributes a theoretical argument showing with sparse-recovery arguments that if the data can be decomposed sparsely, the incomplete feature prediction problem can perform as well as the complete one. Finally, it performs a empirical study on simulated data as well as computer-vision problems with random missing pixels as well as occlusions, and using for the classifier logistic regression as well as classic neural network architectures.

The work is interesting, but I have several big-picture comments on the framing of the work. The first one is that framing the problem as a missing values one, referring to the classic missing-values literature, does not seem right to me. Indeed, in the missing-values literature, different features are observed on different samples, which leads to a set of problems and corresponding solutions. The contribution here is a different one. My second big picture comment is that it seems that this is very much related to the classic work "supervised dictionary learning" by Mairal et al. The framing of the Mairal paper is that of a representation-learning architecture: the cost function is there to impose an inductive bias. Added to a linear model, it contributes representation learning, and hence can bring some of the appealing properties of deep architectures. This is very much what the experimental study reveals, where the contributed architecture with latent factors outpeforms the linear model on fully observed data. With these two comments, I would like to know better how the work positions itself to supervised dictionary learning: the paper is cited (albeit with an incorrect citation), but the positioning is not explicit.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 1. Summary
The authors propose a scheme for simultaneous dictionary learning and classification based on sparse representation of the data within the learned dictionary. Their goal is achieved via optimization of a three-part training cost function that explicitly models the accuracy and sparsity of the sparse model, simultaneously with usual classification error minimization. An alternating optimization algorithm is used, alternating between sparse representations and other parameters.

The problem they want to address with this scheme is training over incomplete/partial feature vectors. A clean theoretical statement is provided that provides conditions under which a classifier trained via partial feature vectors would do no better in terms of accuracy, had it been trained on complete feature vectors. The authors claim this condition can be checked after training, although this ability is not validated/illustrated numerically.

2. Decision and Arguments
Weak Accept
a) Very nice mathematical result and justification. The proof is clear. However, why wasn’t the result used in the numerical section? You claim that the condition (4) can be evaluated to test optimality: how do your trained dictionaries compare to say another dictionary learning scheme? After all, this doesn’t seem to inform your actual learning scheme and is not used to evaluate your results. It would be nice to see a numerical validation/illustration of this result. Also is \delta_K really that easy to compute?
b) The numerical results are good—but lack error bars and comparators. I don’t understand why you considered so many comparators on synthetic data and none on more ‘realistic’ benchmark data.
c) Also I don't feel very satisfied doing image examples, it would be more interesting to work on difficult (eg medical) classification problems with large feature vectors

4. Additional Feedback
a) Very well and clearly written with intuitive examples and clean math. My only suggestion is to clarify in the abstract that there are missing *features*-- when I first read "incomplete data" I think of entire data samples that are missing from the training set. That makes no sense, but it became clear when I got to section 2.
b) Please use markers and dashes etc. with *every* line in your plots. It is hard (or impossible for many) to compare as is with such tiny thin lines.

5. Questions
a) Could you comment on statistical significance of your results? For synthetic data it should be easy to perform the experiments on a number of mask realizations and include error bars.
b) Why no comparators on benchmark datasets?
c) The proof of Thm 3.2 is nice—but how reasonable is the assumption that you have two dictionaries each with the exact same RIP constant \delta_K? Can that property be enforced (even approximately) during training? Or is it trivial that, given one such dictionary, there exists a second one? 
d) See 2.a

review_title: Official Blind Review #3

Meta-review: This was a difficult paper to decide, given the strong disagreement between reviewer assessments.  After the discussion it became clear that the paper tackles some well studied issues while neglecting to cite some relevant works.  The significance and novelty of the contribution was directly challenged, yet I could not see a convincing case presented to mitigate these criticisms.  The paper needs to do a better job of placing the work in the context of the existing literature, and establishing the significance and novelty of its main contributions.
Decision: Reject