Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a method that learns deep prototype representations, by revisiting the nearest centroids clustering in the setting of deep network training. The authors construct a framework to combine the gradient descent kind of learning of neural networks with the distance-based learning of cluster centroids. In this paper, the clustering is done class-wise, defining a number of centroids K per class.
strength_and_weaknesses: __Strenghts__
- the idea of learning deep centroids is relevant
- interpretability of the classification with prototypes or a real sample closest to the prototype
- experiments with different backbones


__Weaknesses__
- _statements about limitations of existing methods are too stretched_, e.g. 
	1. lack of simplicity or explainability mentioned as weaknesses of existing approaches is not addressed clearly in the paper: the mention of lack of simplicity of existing methods makes one assume that the authors would propose a simpler method. Anyway, it is still absed on gradient descent optimization, with the addition of extra steps to ensure clustering after every update of the weights. Furthermore, the memory and algorithmic complexity of the proposed solution increased, making me wonder where the lack of simplicity is addressed. 
	2. learning without considering underlying data structures (i.e. by similarity) and on fixed dimensional spaces: the authors miss the relation of this paper with a body of work on similarity-based and distance learning, e.g. contrastive learning or arcloss, is not covered. In these works, learning and optimization of parameters is done on the basis of similarity/metric learning, and subsequently decisions are taken by looking at the closest samples in the latent space. Other methods based on prototype learning are also ignored, e.g. [1][2]
- _experiments should be extended_: methods based on latent similarity metric learning should be compared to, as the principle of classification/prediction for those methods is the same of closest-prototype picking.
- _result improvements are marginal, with no statistical analysis of differences or details about replication of experiments_: the results are only very marginally above the baselines, and statistical margins are not reported. Are the results from the best runs of training or e.g. median over 5 runs, average over N runs, etc.? It is difficult to evaluate the contribution of the clutering approach.
- _explainability/interpretability study needs extension and comparative analysis_: other approaches are used to do predictions based on distance/similarity in the latent space (e.g. supervised contrastive learning, circle loss, arcloss) that are not considered in the comparative analysis, also in terms of interpretability of the classifications. 

[1] nauta et al. Neural Prototype Trees for Interpretable Fine-grained Image Recognition
[2] Chen et al. This looks like that: Deep learning for interpretable image recognition
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, and the method design is well-constructed with proper motivations of the several choices. Novelty is related to the formalization of the method and is, in my opinion, acceptable for ICLR, but the placement of the work within the state of the art on prototype- and distance-based learning is falling short (see statements above). Some statements, especially in the introduction, are not backed with thorough experimental results or not addressed explicitly in the remainder of the paper. 

I trust that reproducibility can be ensured as long as the author would publish their code, as promised.
summary_of_the_review: The paper presents an interesting idea, well-constructed, and with a clear presentation of the devised methodology. The main concerns with the paper regard placement within the current literature, clarification of several statements made in the introduction and experimental analysis. 

This work seems to relate with metric learning, e.g. contrastive learning, approaches but the authors do not cover this part. Indeed, equation 7 reminds very clearly the 'contrastive loss function' (with the difference that only the min distance is considered in (7)), see [3] equation 1.

The reported results are only marginally 'better' than baseline networks, but the authors do not compare with other related approaches (mentioned few above), making the evaluation of the contribution of deep prototype learning difficult to appreciate. Also in the discussio nabout explainability, methods based on classification by nearest sample (arcloss/arcface-like methods, contrastive approaches) should have been considered. 

[3] https://openaccess.thecvf.com/content/CVPR2021/papers/Wang_Understanding_the_Behaviour_of_Contrastive_Loss_CVPR_2021_paper.pdf
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a deep nearest centroids (DNC) network for large-scale visual recognition. In contrast to the commonly employed parametric classifier (e.g., a parametric Softmax Classifier), the proposed DNC is claimed to be able to capture the underlying data distributions and is more interpretable. Experiments on a few benchmark visual recognition and semantic segmentation datasets demonstrate the superiority of DNC.

strength_and_weaknesses: Strength:
++ Nearest centroids based classifiers have been used in many classification problems and exhibit strong performance compared with its parametric counterparts. This work applies the non-parametric classifier in the basic image classification task and achieves competitive performance by utilising sub-centroids of training samples to describe class distributions.
++ Thorough experiments have been conducted to support the claims in the manuscript.

Weaknesses:
-- The number of centroids for each class K seems to be an important hyper-parameter. In the paper, K is set as a unique value for all classes; however, this may not be optimal since the intra-class variability can be very different across different classes. Is it possible to automatically determine the K value for different classes?
-- The nearest centroids classifier only considers the 1st-order statistics. Would the 2nd-order statistics of clusters contribute to the classification performance?
-- It would be interesting to compare the errors made by DNC and a parametric softmax classifier. Do they learn complementary knowledge?
clarity,_quality,_novelty_and_reproducibility: The paper is well written, easy to follow. Although the nearest centroid classifier itself is not a novel idea, the successful application of it to large-scale visual recognition is non-trivial. The method is clearly described and should be reproducible.
summary_of_the_review: To summarize, the paper investigates a classific non-parametric classifier and successfully integrates it into deep learning frameworks. Competitive performance have been achieved. The work can provide useful insights into representation learning and visual recognition for the community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers the use of nearest centroid classifiers in the context of deep neural networks, and in particular a setting where multiple centroids are used per class, obtained through clustering the feature vectors for each class.
The proposed method consists in concurrently learning the feature backbone network, and a per-class online clustering. For the clustering an optimal-transport based method is used to approximately equalise the amount of data assigned to each cluster.
Experiments on image classification (ImageNet, CIFAR-10) and semantic segmentation (ADE20K, Cityscapes) are conducted using different network architectures.
Results indicate improvements in prediction accuracy when comparing parametric linear classifier heads and nearest centroid classification heads. Moreover, networks pre-trained with the nearest centroid classifier, also transfer better to semantic segmentation tasks. 
In addition, when using data points close to the cluster centres for classification, the method allows for interpretable results via visualisation of the exemplar used for classification. 

strength_and_weaknesses: 
Strengths:

+ Two different tasks are considered to evaluate the effectiveness of deep nearest centroids classifiers: image classification and semantic segmentation (pixel classification). 

+ For each task two datasets are considered: ImageNet, CIFAR-10 for classification, and ADE20K, Cityscapes for semantic segmentation.

+ Multiple network architectures are considered for each experiment.

+ Exploration of the use of training samples rather than cluster centres as anchors for classification is interesting from an interpretability perspective.

+ The use of nearest centroid classification is shown the be beneficial for semantic segmentation: (i) using it to fine-tune the segmentation model, and (ii) using it to pre-train the feature backbone network. Using it for both yields best performance. 

 
Weaknesses:

- The claimed improved transferability is shown when pre-training on ImageNet classification, and then fine-tuning on ADE20K/Cityscapes semantic segmentation. There is no experimental evidence that transferability is improved towards other image classification tasks. Such results would be relatively easy to add, and would strengthen the support of the claim of improved transfer of features learned using the centroid-based classifier. 

- Experimental results do not include analysis of the stability of the results w.r.t. various sources of randomness in the experimental setup (initialisation of parameters, sampling of training batches). In particular for tables 1 and 2 this would help, as differences in accuracy are relatively small.  For ImageNet classification adding evaluation on the ImageNetv2 test set (https://github.com/modestyachts/ImageNetV2) would also strengthen the experimental results.

- It is not clear where the normalised exponential for of the solution presented in Eq 6 comes from. In my understanding it is a consequence of the KL-regularization used in [25], but which is not adopted here in Eq 5. I guess it is an omission.  

- The effect of the optimal-transport / Sinkhorn clustering approach as compared to a naive k-means approach is not ablated in the paper, while this is the main novelty of the paper over [52] besides using multiple centroids per class.

- The setting of the temperature parameter epsilon in Eq 6 is not discussed, nor is its impact experimentally analysed. 

- Relations of the current work to [23,52] are not discussed in sufficient detail, it should be clear from the paper how this work is different from these earlier works. For example, the related work section states that [52] showed weak results even on small datasets, yet it is not clear why the current paper would work better for those cases: the ablations show only limited improvements over the case with K=1 cluster center per class as used in [52]. It is not clear from this why for the current paper experimental comparison to [52] is not included. 


clarity,_quality,_novelty_and_reproducibility: # Clarity
The paper is clearly written and easy to follow, but the quality of writing can surely be improved. There are numerous typos, sentences that are unclear or poorly formed. 
Examples include
- page 1: "greatly boots pixel recognition"
- page 1: "... and ever popularity [20-23], Nearest Centroids, and particularly..."
- page 4: "... cannot providing..."
- Eq 5: the clustering objective that is stated suggests using dot-product as a score to determine cluster assignments and probably L2 normalised feature vectors. This is not explicitly stated, however, at this point in the paper. 

# Quality
Some claims in the paper seem too strong, and the contributions are not clearly identified.
For example, the introduction claims that 
-linear classifiers assume unimodality of the data "...bearing no intra-class variation.", which is not substantiated. 
- "linear classifiers are trained purely for classification accuracy":  this also holds for centroid based classifiers, their difference lies in the pararmeterization of the classifier.
- On page 2 there is a list of 5 attractive properties of centroid-based classification: these seem generic and not associate with the contributions made by the authors in the current paper. The list of experimental findings in the last paragraph of page 2 (running over to page 3) seem to be the claimed contributions of the paper.	

# Novelty
The paper evaluates nearest centroid classifiers in combination with deep feature learning networks. 
The nearest centroid classifier, as well as the optimal-transport based clustering approach are known in the literature. Their combination, however, appears novel to me.
I haven't seen nearest centroid classifiers for semantic segmentation, but I'm not actively working on this  application nor on centroid-based classifiers, so I could have missed something. 
In particular the transfer ability of the imageNet pertained feature backbone to the segmentation task is an interesting experiment in the paper, that contributes to the literature.

# Reproducibility
The work seems fairly easy to reproduce. The method is clearly described. Code is promised, but not provided. Experiments were conducted across an 8 V100 GPU hardware setup. 
It wasn't clear to me in Section 4.4 why using K>4 centroids per class was not possible for the imageNet classification experiments, but it was possible to go up to K=20 for the ADE20K semantic segmentation experiment. It would be useful to add a paragraph analysing the memory cost of the model wrt the number of centroids K and the "external memory" size used to store additional batches for clustering. 

summary_of_the_review: 
This paper considers nearest centroid classifiers in combination with deep feature learning networks. The authors train the feature network, and cluster the data of each class in parallel using an optimal-transport based technique: a novel combination of existing techniques. 
Experiments on image classification and semantic segmentation compare the centroid-based classifier with a linear classification head, and conclude improved accuracy using centroid-based classifiers.
Experiments also indicate that centroid-based pre-training for image classification yields better results when fine-tuning for semantic segmentation. 
Replacing centroids with the nearest datapoints (images) induces a small loss in predictive accuracy, but yields an interpretable exemplar-based classifier.
My ratings are based on a small perceived technical novelty, combined with some interesting experimental findings. The quality of the presentation can be substantially improved.

### Post-rebuttal comments ###
After reading author rebuttal and other reviews, I maintain my original recommendation of this paper as weak accept. 
The authors have addressed most of the comments in my review: a number of clarifications have been made, and additional results are added, in particular evaluation on the ImageNetv2 evaluation sets, which confirm the results obtained earlier on the imageNet validation set.  I do not see any major reasons to reject the paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes deep nearest centroids, a new non-parametric classification layer. For each class, a number of sub-centroids are learned during the course of training. At test-time, the prediction is the class corresponding to the closest sub-centroid. The sub-centroids are learned in an iterative fashion: 1) Network parameters are updated to minimize the distance between each sample and the closest sub centroid of the ground truth class 2)  The subcentroids are updated using “momentum updates” using the centroid assignments of the current minibatch + a memory bank.

Improvements are shown on CIFAR-10 (ResNet backbones) and ImageNet (ResNet + Swin backbones). Results on semantic segmentation are even stronger on 3 algo + backbone combinations. If the sub-centroids are restricted to training images, the algorithm shows out-of-the-box interpretability, in the sense that these sub-centroids can be leveraged to explain classification decisions by the model.

strength_and_weaknesses: **Strengths**

* The paper shows strong results on both classification and segmentation benchmarks. Ablations are shown to evaluate the impact of #subcentroids, showing that this design choice is important, i.e with no sub-centroid clustering, the algorithm performs worse than the classification layer.
* While a number of architectural changes have been proposed to neural networks, this paper proposes to modify just the last layer with performance improvements, which is of interest to the ICLR audience.
* The model provides out-of-the-box interpretability under some constraints. A few epochs before convergence, the authors propose to anchor each sub-centroid to the closest training image which can provide explainable predictions.

**Weaknesses**

See below for detailed feedback.
clarity,_quality,_novelty_and_reproducibility: **Novelty**

The proposed deep centroids layer can be considered as the application of the same algorithm for cluster assignments done in unsupervised learning [1, 2] to do within class cluster assignments. The authors should be upfront about this. Note that this does not mean, that the current work is not novel, but the writing in the introduction/related work should reflect this and the work should be placed in this context.

**Reproducibility**

* It would be nice to have a small Algorithm section showing the pseudcode of the algorithm. For example, the algorithm can show in what sequence the cluster assignments, (parameter, memory bank and cluster) updates are done. For reference, look at Algorithm 1 of the MoCo paper (https://arxiv.org/abs/1911.05371)
* The deep centroids layer has 2 additional hyperparameters (K and $\mu$). How was hyperparameter tuning done? Was there a separate validation set split from the train set? Please provide these details.

**Clarity and Quality**

I would appreciate it if the authors can clarify these details.

* **Section 3, transferrability**: The authors claim that for after pretraining, the softmax layer is thrown away, leading to "wastage" of parameters. This is not 100% convincing since the equivalence to the deep nearest centroids is the "sub-centroids" which are K times more than the paramters in the softmax layer which also have to be thrown away during finetuneing.
* How important is the Sinkhorn-Knapp algorithm to determine sub-centroid assignments? Can one just set the cluster assignment to be the nearest sub-centroid as done in K-Means?
* **Section 4.3**: For the "anchoring of sub-centroids", to obtain the images used to anchor the sub-centroids, do the authors find the closest image over the entire training set or is just the memory bank sufficient?
* **Parametric Softmax Classifier**: "and cannot directly supervise on the representation $x$". It is unclear what this means as $s$ is a a function of $x$
* **Memory Bank Size**: It might be more intutive to show this hyperparameter as a function of #examples rather than #batches.
* W and b are learnt without considering any underlying data distribution. It is unclear what this means as W and b are learnt by modeling the  conditional P(y|x) which is the data distribution.


The paper introduces some terms which in my opinion are a bit colloquial and diffficult to understand. The colloquail terms are bolded below

* Abstract: Case-based reasoning, it is unclear what "case-based" refers to
* Intro, para 1: "detached from the physical nature of the problem"
* Intro, para 2: it is fully aware of the limitations of parametric counterparts
* nonparametric manner and understandable from user’s view, in contrast to the parametric classifier that learns **non-transparent parameters** for each class. 
* Section DNC Classifier: finding **classification evidence** for a previously unseen sample by retrieving the most similar exemplar.
* Section Training of DNC: DNC directly optimizes the representation by **adjusting the arrangement** between sub-centroids and data samples.
* Section Versatility: **Reforms the training regime**
* Section Ad-hoc Explainability: explanations are **loyal to the internal decision mode**
* **stronger adhoc explainibility** - I think this sentence should be restructured.



**Typos**

This did not affect my rating,

* proximity of test data and the class sub-centroids in the feature space -> proximity of test data to the class sub-centroids in the feature space
* Parametrization nature -> parametric nature
* ImageNet [9] trained image classifiers initialize segmentation networks -> ImageNet-trained classifiers to initialize
* memory for online estimating the sub-centroids -> for estimating the sub-centroids online
* investigate post-hoc explains -> investigate classifier posthocs.
* even coming with better performance -> even with better performance
* The final output dimensionality is constrained as many as the classes. Constrained as -> constrained to be the number
* if using multiple sub-centroids (local means) per class instead of just on -> if multiple sub-centroids per class are used.
* coming with minimal architecture change -> with
* one single class weight/center is far enough -> Did you mean not enough, in the context of the results

[1] Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, NeuRIPS 2020
[2] Self-labelling via simultaneous clustering and representation learning, ICLR 2020
summary_of_the_review: I initally rate this paper below the border, even though the results are strong. See above for a list of my concrens.
I am happy to adjust my rating if the authors can address my concerns with respect to the novelty and clarity of the paper and update the draft as a result.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes deep nearest centroids, a new non-parametric classification layer. A number of sub-centroids are learned for each class, and the prediction is the class corresponding to the closest sub-centroid during testing. Good improvements are shown on CIFAR-10 (ResNet backbones) and ImageNet (ResNet + Swin backbones). Interesting results were also shown on semantic segmentation. In addition, the algorithm shows out-of-the-box interpretability, i.e. the sub-centroids can be used to explain classification decisions by the model.


justification_for_why_not_higher_score: combining clustering (centroids) in neural networks is not entirely new ideas, which can be traced back to early works, e.g. RBF 
justification_for_why_not_lower_score: The paper shows interesting results, e.g. semantic segmentation, and interpretability based on the sub-centroids. 
Hence this will be useful to be highlighted to the ICLR researchers
Decision: Accept: notable-top-25%