Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 9: Very Strong Accept: Technically flawless paper with groundbreaking impact on at least one area of AI/ML and excellent impact on multiple areas of AI/ML, with flawless evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper provides a link between compressed sensing, non-parametric learning, and deep classification networks. By formulating classical compressed sensing results in context of deep learning, the authors propose a learning framework for deep generative modeling using compression, known as Non-Parametric learning by Compression with Latent Variables (NPC-LV). Experiments are conducted on image classification on standard benchmarks (MNIST, CIFAR-10, and Fashion MNIST) and demonstrate compelling results compared to prior work. A strong theoretical analysis support the method.  
strengths_and_weaknesses: The key strength of the paper is the original and novel idea of unification of two streams of research: compressed sensing and classification using deep neural networks. The description of distances and metric learning from compressed sensing and reformulation into DL terminology is extremely helpful and useful. The proposed method demonstrates very compelling results on three image classification benchmarks, outperforming supervised learning and non-parametric learning methods, and semi-supervised learning under low data regime. Theoretical results include derivation of equivalence between optimizing latent variable models and shortest code length, using negative evidence lower bound (nELBO) analysis. Empirically, the authors also show an inverse relationship between accuracy and bitrate. 

Overall, the paper is very promising and has few weaknesses. The key limitation seems to be that analysis is focused on VAE and image classification. It would be valuable to understand the relationships of NPC-LV with respect to other popular generative models, e.g., GANs. In computer vision, is is standard practice to learn a good representation, e.g., using VAE or GAN, and re-use the learned space for fine-tuning to many down-stream tasks. In this context, it may be valuable to understand evaluate how good of a representation does NPC-LV learn in comparison to VAE/GAN. However, even with existing analysis and setup, I believe the paper is very promising and would encourage research at the intersection of compressed sensing and DL representation learning, and thus a great contribution to NeurIPS. 
questions: In Table 1, for several experiments on MNIST and FashionMNIST, the best performing methods appears to be self-supervised learning and not supervised learning. Why is this the case, and why isn’t supervised learning always achieving top result?

limitations: Analysis is very thorough and limitations are discussed in section 6.3. Given that one of the limitations is complexity, it would be helpful to report required training/evaluation times and comparison to those of baselines.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Hi all, I am so very sorry for the incorrect review !! 

The authors prropose a strategy for few shot learning by using data compression and nearest neighbor classification.  They focus in particular on a distance based on the kolmogorov complexity. The latter being uncomputable, the authors use different types of compression-based distances.

Given a couple of images, the authors thus compress each individually as well as their aggregation in order to be able to compute the distance.

The authors test their méthodes on multiple datasets : MNIST, FASHIONMNIST and CIFAR10 and show very competitive results.
strengths_and_weaknesses: ## Strengths
The paper propose a novel method based on theoretical grounds and demonstrate its usability with deep learning. The paper seems theoretically sounds. It is very easy to read and the context is well presented for non expert readers. The experimental section support the contribution with competitive results.

## weaknesses
The datasets are known to be easy discriminative tasks... I think that it woul be interesting to see how the method generalizes with more difficult tasks. A deeper focus on representation learning strategies would help get insight about the benefit of the method.
questions: What is the effect of the difficulty of the task ? How such a method compete with other representation learning strategies ?
limitations: Limitations are given and discussed by the authors in section 6.3.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper describes a generative model trained in an unsupervised fashion which utilizes the data distribution to build the compressor. The theory presented in this paper is a minor extension of previous work (see the details below) and the experimental results is not convincing and lack of interpretation and detailed analysis.
strengths_and_weaknesses: The strength of the paper lies in that it proposes a compression-based distance metric and applied it to the few-shot learning.  The experimental results show that the proposed method outperforms some of supervised methods and semi-supervised learning methods on image classification.

The weakness are as follows:

(1)	Compared to the theory proposed in the reference [36], the contribution of this paper seems to be quite incremental as they are all about performing lossless compression with latent variable models. The mathematical framework is largely similar and the intuition behind is the same. The only difference is that in this paper, the compression is applied to the distance metric in few-shot learning setting up which turns out to be a pretty straight-forward extension of [36].
(2)	The experimental results shown in the paper is somewhat not convincing. For instance, the paper claims that the performance of the proposed method is better than supervised learning methods. However, the paper fails to present what components and for what reasons it can outperform supervised learning methods. Because with data samples are fully labeled, supervised learning performances are frequently served as a upper bound of the semi-supervised learning or few-shot learning performances. Without detailed explanations and looking into the insight of the comparison, it is hard to believe the fairness of the comparison.
(namely is the supervised learning methods state-of-the-art, how are the parameter settings of these comparative methods etc)
(3)	The theory presented in this paper are either an extension of the work in [36] or disconnected with the experimental results presented afterward. Namely, I have no idea how to apply these theory to the few shot learning problem and how does these theory support the incremental performance in the experiments. As intuitively, if we compress the distance, we either lose the information and preserve all the information (will not increase the information), so it is not clear where the performance gain comes from. A separate ablation study may help to illustrate the performance gain.

questions: As I pointed out in the weakness of the paper, weak evaluations and lack of interpretations are the main issue of the paper. It will be helpful if the paper can detail the analysis and reveal the insights about why the proposed method can outperform the supervised methods and semi-supervised methods using ablation study. It is also important to make the experimental section more convincing with fair comparison.

Moreover, it is critical to make sure the theory is well connected to the experimental results and support the performance gains.

Last but not least, it is imperative to enhance the uniqueness (novelty) of the paper by explaining the difference of the paper compared to [36] as applying the compression theory to the distance metric in few-shot learning seems to be straight forward and the improvement is incremental.
limitations: N.A.
review_title: null

Meta-review: metareview: This paper proposes a learning framework for deep generative modeling called Non-Parametric learning by Compression with Latent Variables (NPC-LV) with a strong theoretical support. The results on image classification benchmarks look compelling and promising. It's impressive that this unsupervised approach achieves strong results even compared with supervised and semi-supervised approaches. The reviewers unanimously think that this work contains novel and interesting ideas to connect the deep generative modeling, non-parametric learning and compressed sensing in an elegant manner. The reviewers also pointed out that more thorough discussion and investigation on generalization ability of the proposed method to other scenarios beyond VAE and image classification would further strengthen the paper. A good discussion on limitations is presented in the paper. After the rebuttal, most concerns that the reviewers raised have been well addressed. The AC recommends acceptance. 
confidence: Certain
Decision: Accept