Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies a new contrastive loss function for knowledge distillation and self-supervised representation learning. The paper demostrates results on transferring knowledge from a teacher to student using both supervised and self-supervised objectives, and on representation learning based on self-distillation. As prior work has already applied contrastive learning to each of these problems the main novelty of the paper is the exact form of the contrastive loss. The loss is computed over projections of the embeddings into a k-dimensional simplex (probability space) where cross-entropy can be used as the similarity function. This allows for encoding "negatives" into a distribution over k prototypes that are updated with EMA over iterations of training rather than requiring a large batch of negatives during each iteration of training. The results appear to slightly outperform SOTA methods.
main_review: I found this to be an interesting paper. The results look decent, the method makes sense, and the derivations appear to be sound. The paper is clearly written and well presented.

There are however, several weaknesses to the paper. First and foremost, almost all the methods appear in prior work, albeit used in slightly different ways. The use of representations of negatives, updated with momentum, is reminiscent of MoCo; in both cases the basic method -- momentum -- and outcome -- smaller batches of negatives -- are the same, even though how the negatives are represented is different. Prior work has used contrastive losses for knowledge distillation (e.g., CRD), and has used prototypes for contrastive learning (e.g., DINO, SwAV). The Sinkhorn-Knopp method was introduced in SeLa. The present paper is novel in how it puts together these pieces and applies them, but the key technical ideas are all already in the literature, as far as I can tell.

Nonetheless, as an engineered system, the paper achieves solid performance. The ablations and comparisons are thorough, and I'm convincined that the ProtoCPC objective does have benefits over the baselines, although the numerical gains are not large. 

One point the paper makes is that ProtoCPC avoids the large batches required by some of the competing methods, and that this could have advantages in terms of computational efficiency. I would have liked to see a quantification of this point, in terms of total number of network calls, memory consumption, or wall clock time.

Two minor comments:
1. I'm not sure the name ProtoCPC is well-chosen. Contrastive Predictive Coding involved a "predictive" component in the form of the linear transformation W that predicts z_{t+k} from c_t. I don't really see a "prediction" component in present paper, at least not in the same sense as in CPC.
2. The title also could be improved: there is no mention of distillation in the title yet the rest of the paper is all about applications to distillation.
3. “Our method infers” —> “Our method inherits”?
summary_of_the_review: I think this is a fine paper, well executed, and with decent results. However, I think the novelty is minimal compared to prior work that introduced the key methods that are used, and the empirical gains are also not that large. I therefore think this paper is slightly below the threshold of typical ICLR papers in terms of value to the community, although it is above the threshold in terms of sound science.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides a method for combining contrastive learning and clustering (prototypical probabilities) for three knowledge distillation tasks - supervised model compression, self-supervised model compression, and self-supervised learning with self-distillation. 
Traditional contrastive learning methods rely on the similarity of feature vectors from the teacher and student networks using cosine similarity, log bilinear model (van den Oord et al), etc. This paper proposes to project the feature vectors into a probability space via linear prototypes (layer) of the same dimension (i.e, cluster the feature vectors). The proposed method overcomes the disadvantage of having large negative examples for contrastive learning by maintaining an EMA of prior that accounts for the negative examples.

The main contribution of this paper is the ProtoCPC loss that combines the advantages of contrastive predictive coding loss using prototypes over the teacher and student representation. 

The authors empirically show that the proposed method is useful for various distillation tasks on image classification tasks on CIFAR-100 and Imagenet datasets.
main_review: Strength:
(i) The proposed method is well described and seems to work as shown by sufficient experiments in various distillation settings. 
(ii) Fairly well-written paper and easy to follow various distillation settings.

Weakness:
(i) A detailed justification on why momentum-based estimation helps the ProtoCPC loss is missing? 
(ii) Empirically the method seems to work, a critique on why prototypes (clustering) help is missing? Motivation on prototypes would be appreciated.

Questions:
(i) The pseudo-code of the ProtoCPC loss only takes z_t and z_s as input. How are the negative examples provided for momentum estimation? 
(ii) Currently, the prototypes are computed using a linear layer. Are more complex networks helpful or deterimental?
(iii) Is the runtime of the method a criterion to evaluate efficient learning?

Typo:
computation between z_s and z_tjS -> computation between z_s and z_tj

summary_of_the_review: I enjoyed reading the paper and feel it has value to the community. The experimental study justifies the usefulness of the paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a prototypical contrastive predictive coding by combining the prototypical method and contrastive learning. Experiments are done on applications such as supervised/ self-supervised model compression and self-supervised learning by self-distillation to validate its effectiveness.
main_review: Strengths:
This paper is well written and organized.
The proposed method can be integrated into other methods and Experiments are done on different tasks to validate effectiveness.
Weakness: 
The main concern is the novelty of this method. It combines the prototypical method and contrastive learning while these two techniques are well researched in the literature. Also, compared with CRD+KD  the improvement is marginal in the knowledge distillation task.

After rebuttal:
I keep my original rate.  I think this paper is well presented with decent results. 
summary_of_the_review: This paper is well written and experiments are done thoroughly. The main concern is the novelty as it combines existing techniques without substantial findings.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine knowledge distillation (KD) and contrastive learning for distillation tasks. Concretely, it models the critic of a contrastive objective by the prototypical probabilistic discrepancy between two features. The authors then carry out extensive experiments on supervised model compression, self-supervised model compression, and self-supervised learning through self-distillation. The empirical results show that the proposed method outperforms other strong baselines. 
main_review: As for citation, the original method of KD was introduced in "Model compression, KDD 2006". The authors cite this paper in the Related Work section, but it might be more appropriate to also cite it in the Introduction section. 

Please reply to the below questions.

1. Although the paper title called "PROTOTYPICAL CONTRASTIVE PREDICTIVE CODING", the experiments are only conducted on knowledge distillation. Could you provide other evidence of ProtoCPC's advantages (e.g. Mutual Information Estimation and Representation Learning without distillation experiments)? Otherwise, the paper could be called "**Distillation** with PROTOTYPICAL CONTRASTIVE PREDICTIVE CODING".
2. Is it possible to compare CRD with ProtoCPC beyond the Supervised Knowledge Distillation? The authors claim that "Our method outperforms KD as well as CRD
in supervised model compression on CIFAR-100 and ImageNet". From the experiment results, it doesn't necessarily outperform CRD. 
3. In **Prior momentum** of 2.1, the authors state that "The prior momentum allows better estimation of prior term regardless of the size of negative sample". However, from the equations (6) (7) & (10), the sampled N is still required in the prior. What is the actual reason that we can ignore a large number of negative samples?
4. Furthermore, as the authors claim their ProtoCPC "does not require massive negative samples" when performing contrastive learning, will this method boost the training efficiency?

Minor comment:

- In Table 3 description, the references of SP and CC are missing.
summary_of_the_review: This paper proposes to combine knowledge distillation and contrastive learning for distillation tasks, which is also well-motivated. Extensive experiments on benchmarks validate its effectiveness. It's a nice paper with solid, but somewhat incremental, technical contributions.
review_title: null

Meta-review: This paper proposes a prototypical contrastive predictive coding by combining the prototypical method and contrastive learning, and presents its efficient implementation for three distillation tasks: supervised model compression, self-supervised model compression, and self-supervised learning via self-distillation. The paper is well-written, and the effectiveness of the proposed method is validated through extensive experiments.  Reviewers generally agree the paper has clear merits despite some weaknesses for improvement. Overall, I would like to recommend it for acceptance and encourage authors to incorporate all the review comments and suggestions in the final version.
Decision: Accept (Poster)