Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: SUMMARY

The paper proposes a multi-task loss for semi-supervised representation learning and category discovery (i.e. clustering unlabeled examples). At a high level, this paper adds a contrastive loss term to the approach from Han et al. 2020. The loss has five components: (i) an InfoNCE loss where positive pairs are transformations of the same image, (ii) an InfoNCE loss where positive pairs are images with the same class label, (iii) an MSE loss between the representations of transformations of the same image, (iv) a categorical cross-entropy loss for labeled data, and (v) a binary cross-entropy loss ("same class" vs. "different class") where class labels are based on thresholding a measure of similarity between representations (distance between Winner-Take-All hash codes). The loss supports multi-modal data in that (i) and (ii) may each be computed based on representations of any modality (intra-modal, e.g. anchor and positive are both audio) or any pair of modalities (inter-modal, e.g. anchor is audio and positive is video). The method is evaluated on category discovery benchmarks based on image and video datasets.


STRENGTHS

I think one can make a good argument that there has been too much focus on purely unsupervised methods lately - certainly the semi-supervised scenario this paper considers is far more realistic. 

The paper is generally readable and the figures and tables are clear. The proposed method is reasonable and the empirical performance is strong. Most of the reported numbers are based on multiple runs, and an ablation study is included.

The analysis of multi-modal configurations in Table 2 is interesting and makes a clear case for cross-modal representation learning. 

WEAKNESSES

It looks like the baseline methods were not re-implemented (the numbers were just copied over from previous work). Are all hyperparameters, models, etc. similar enough that these numbers can be fairly compared? Please clarify.

On the topic of fair comparisons, Han et al. 2020 seems to report stronger performance than is credited to them in Table 3. Also, the proposed method is very closely related to Han et al. 2020 (this paper adds a term to the loss proposed in that paper) but that is not clear from the text. Please make that more explicit. 

There doesn't seem to be any description of validation sets or hyperparameter tuning procedures. Without this information, it's difficult to evaluate the significance of the quantitative results.

The paper would benefit from a bit more editing - see "minor comments" section below. In addition, the general flow of the writing could be improved. For instance, phrases "the accuracy is increased to X/Y/Z on dataset A/B/C" are used often and are a bit hard to parse.

OVERALL

The paper addresses an interesting and important question and presents strong results. However, there are some concerns regarding (i) how hyperparameters were tuned and (ii) the difference between the contributions of this paper and others, notably Han et al. 2020. 

MINOR COMMENTS

**Abstract**

The terminology "We further introduce Winner-Take-All (WTA) hashing..." makes it sound a bit like WTA is something introduced in this paper.

**Section 3.1**

The notation $B \sim D^u \cup D^l$ is a bit unclear. Consider writing something like "$x_i \sim \mathrm{Unif}(D^u \cup D^l)$ for all elements $x_i$ in batch $B$."

Perhaps $Q(i)$ should be defined as $\{ q \in \mathcal{N} \backslash i : y_q = y_i\}$? Is $Q(i)$ meant to exclude only example $i$, or does it also exclude $z_{i'}$? Perhaps $p$ is meant to denote that?

The phrase "Without loss of generality" is a little out of place since no theoretical claim is being made.

**Section 3.2**

In the paragraph after Equation 4, "uisng" should be "using". 

It's not really clear what motivates the use of WTA here - one could imagine all sorts of schemes for deciding whether two feature vectors are "similar enough" and this one seems a bit unusual. While there are additional comparisons in the appendix, it's hard to evaluate those results without knowing more about the hyperparameter selection procedure.

It's also not clear why the binary cross-entropy loss is chosen over e.g. yet another contrastive loss, where the pseudo-labels are used to determine whether two examples are to be considered similar or different.

Presumably $\hat{z_i}$ should be $\bar{z_i}$ after Equation 5.

Why limit this step to just unlabeled examples in each batch? It doesn't seem like there's any reason it couldn't also be applied to the labeled examples.

**Section 3.3**

Please explicitly define $\mathcal{L}^\mathrm{MSE}$. Does "the predictions" mean "the representations of $x_i$ and $t(x_i)$" in this context?

It seems like there's some missing text in the phrase "We set $(1-\omega(r))$ for contrastive learning." 

**Section 4**

Is $e$ an arbitrary element of $\mathcal{P}(C^u)$ or is it the optimal permutation? The notation and text contradict each other. 

Both $M$ and $U$ have been used to denote "the number of unlabeled instances" - it might be helpful to choose just one. 

How is $C^u$ set? Is to chosen to be the correct number of classes? It seems like it would be appropriate to examine mis-specification of this parameter since it is generally not known in the "open world" setting the authors discuss. This is particularly important since $C^u$ is a hyperparameter of the proposed method (not just used for an evaluation metric).

**Table 1** 

"Kinetcis-400" should be "Kinetics-400"

**Conclusion**

I think "imperially" should be "empirically".

**Update: After considering the other reviews and subsequent discussion, I have decided to maintain my score. As covered elsewhere in the discussion, the key shortcoming of the paper in its current form is that it provides little insight into why the proposed method outperforms the highly similar method in Han et al. For instance, the fact that WTA is crucial for the method proposed in the paper but does nothing for Han et al. is rather mysterious. I think it is necessary to provide more insight into these differences before readers can have full confidence in the proposed method.**
review_title: Reasonable approach but needs clarification

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes an end-to-end framework for new class discovery. There are several key steps in the framework. (1)First is that they use separate losses, for instance, and class discrimination, so that even the unlabeled data can be trained jointly. (2)The second one is they use WTA hashing other than clustering algorithms to assign pseudo labels for unlabelled data. (3)The last one is they extend the framework to multi-modality settings. 

Pros:
+ The paper is easy to follow. The authors have a very clear structure for the motivations, problems with the current approaches, and contributions.
+ The idea to generalize/unified instance and class discrimination in this semi-supervised setting are pretty simple and effective. 
+ The final experimental results are impressive compared to previous approaches. Especially adding multi-modality and video in evaluation is appreciated.

Cons:
- The presentation of contribution two is not convincing in two ways. First, at the algorithm level, if WTA is not the paper’s contribution, I recommend authors either weaken the way they describe it in abstract/intro and shorten sec 3.2; or explain in details why not other LSH variants or even learning to hash variants for unsupervised/semi-supervised setting? The reasoning in 3.2 is mostly adapted from the original WTA paper and also the follow-up best paper in CVPR[1]. Second, in experiments, I can see when changing from WTA to Simhash, the performance drops sharply (even worse than the baseline). Does it mean that WTA is the real component that makes the framework working? I can also infer it from table 1 that when BCE is omitted, the performance is affected most.
- The framework is ad hoc. There are many losses and design options making the practicality of the framework doubtful.

Minor comments that will not affect scores:
- In the experiments section, I recommend first presenting the final results first and then ablations. 

[1] Dean, Thomas, et al. "Fast, accurate detection of 100,000 object classes on a single machine."  CVPR2013

===============================================================================

After reading the rebuttals and comments from other reviewers, I decided to keep my original score.

My biggest concern hasn't been addressed. I understand there is a comparison between different alternatives in appendix A, but only WTA is outperforming RS in the existing work by a large margin. All the other reviewers and I are curious why this is the case.   
Performing an A/B testing type of analysis can only tell us WTA is the best in some limited benchmarks. However, what would really make this framework principle and shine are the insights and messages we can learn from your solution. 

I further thought about the phenomenon and came up with some hypotheses. For example, WTA is an LSH that the collision probability of h(x) and h(y) corresponds to the ranking/ order alignment of $x_0...x_{d-1}$ and $y_0...y_{d-1}$. Because the choice of K is usually 2,4,8,16, it is more like local ranking information within a few random dimensions, while RS in Han et al is comparing the ranking of x and y globally (across d dimensions). Also, if you see your table 7,  when k goes larger, your performance saturates and degrades. If you start by analyzing the fundamental difference between RS and WTA along with your appendix B and C, you can maybe hypothesize that local ranking (among random dimensions) is more effective than global ranking and verify it with further experiments.  E.g. Intuitively with simple calculation, the half of the pair-wise order information among 4 elements (K=4) could be preserved with WTA hash code ( it is the index of the max so if the hash code is 0, it records 3 out of 6 relations: element 0 > elem1, >elem 2, >elem 3). 
review_title: The framework shows impressive empirical results but the design is ad hoc and has limited novelty in some components.

final_score: rating: 4: Ok but not good enough - rejection
review_content: [Summary]
This paper addresses the problem of clustering unseen classes with the help of labeled data from an exclusive set of classes. It proposes an end-to-end learning strategy that combines a set of optimization objectives for training a deep neural network that can output a cluster index for the unlabeled data. The objectives include modified BCE for clustering, instance-wise contrastive loss, class-wise contrastive loss, consistency loss for augmented data pairs, and standard cross-entropy for the labeled data. The objectives utilize labeled and unlabeled data jointly. Extensive experiments on multiple image datasets show its advantage over SOTA in the class-discovery benchmark. The extended setup with multi-modality data demonstrates the capability of gaining additional performance when richer information is available.

[Strengths]
The experiments have good coverage on four popular datasets. The ablation of each component clearly points out the import factors to achieve good performance. The results show a significant advantage over the SOTA method.

[Weaknesses]
It is unclear what the novelty is in the proposed method compared to (Han et al. 2020). The primary difference is replacing the self-supervised learning method (rotation) with contrastive learning. The remaining learning objectives are highly similar to (Han et al. 2020). Although this paper optimizes all objectives jointly, it is unclear what the necessary change is to make this possible (or not possible in (Han et al. 2020)). Lastly, the winner-take-all-hash in the L_BCE is highly similar to the rank statistics (Han et al. 2020). The author should give an in-depth comparison against  (Han et al. 2020) and discuss what issues are mitigated by the proposed changes.

[Questions]
1. The performance of DTC and RS in Table 3 is significantly lower than the original paper. Why?
2. How will the choice of mu in eq-4 influence the performance?

-------------------------------------------
POST REBUTTAL

Thanks for the author's swift responses. I appreciate that. However, the paper still needs to provide more evidence and precise comparisons to clarify the mentioned concerns. Three major concerns are listed below:
1. Why does the proposed pipeline perform better than RS? The paper should add the proposed modifications one-by-one into the RS method and show/discuss how these individual design choices affect the performance.
2. WTA is worse than rank statistics in Appendix Table 5. Is it a counterexample of WTA? It needs more experimental supports to show WTA is a better method.  The hyper-parameters used in all experiments should be provided in a table and explain how the parameters are selected. Please make sure the same tuning budget is given to all the methods for a fair comparison.
3. What are the necessary changes to make end-to-end training possible (or not possible in RS)? The paper should summarize RS's design and explain why it can not do end-to-end training, then explain what modifications are made to make this possible. This comparison is better to be illustrated in figures.

Overall, this is an interesting paper, and the community will be benefit from it if well presented. Please consider revising and submitting to another venue.

review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper proposes a contrastive learning method for novel category discovery. The authors propose loss functions that can utilize labeled data of known categories and unlabelled data like semi-supervised learning, whereby they leverage both category discrimination and instance discrimination. They also extend cross-modal discrimination for multi-modal data. But these are the side loss functions. The key component to discover novel classes is a clustering loss, for which the authors propose to use Winner-Take-All hashing to generate pseudo labels of cluster partitions.

The proposed method is evaluated on CIFAR-10/-100 and ImageNet for single modality, and Kinetics-400 and VGG-Sound for audiovisual modality, and shows notable performance improvement over the state-of-the-art.


Reasons for score: 

The proposed method shows significant performance improvement for the target task, but the manuscript seems not ready for publication to this reviewer in that: 1) it is unclear specifically what makes the proposed method better than the competing method [Han et al. 2020], 2) the motivation of the proposed method sounds hand-wavy (in particular, why should the multi-modal extension be considered?), and 3) there is room to improve the paper organization further (it comes and goes).
This reviewer would like to ask for clear and logical responses to the comments. 

Pros:
- Noticeable performance gain over the state-of-the-art
- The authors demonstrate that WTA hashing based clustering is effective for novel category discovery.
- Reasonable ablation study

Cons:
- Lack of motivation of the designs (in particular, multi-modal)
- The paper is written in a confusing way to identify the most significant contribution (I deem it is WTA hashing based clustering).
- The experiments are not designed to clearly reveal a key contribution.
- There is room to improve the paper organization further.


Detail comments and questions:
- After going over all the experiment including the experiments in the supplmentary, this reviewer thinks that WTA hashing based clustering is the source of the most significant gain the authors obtain. Is this a correct parse? If yes, then both the description and experiments are not presented to effectively emphasize this part. In particular, the experiment could be designed to show more clearly about this point.
	- Specifically, Tables 3 and 4 show that, even with contrastive learning (CL), the competing methods do not show notable performance gain. Then, the key difference that leads to the performance gain may come from the WTA hashing based supervision. It would have been effective to present the combination of RS [Han et al., 2020] + WTA hashing based clustering, whereby the source of the performance gain becomes clearer.
	- The ablation study in Table 1 does not replace the above suggestion, because it is obvious to have detrimental performance when omitting the closest loss to the target task (the novel class discovery) which is the WTA hashing based clustering loss. Thus, to show the effectiveness of WTA hashing based clustering in the target task regime, it should have been compared with its counterparts from the state-of-the-arts.

- Why is multi-modal important or effective for novel category discovery? It is barely discussed.

- The paper structure and organization should be improved.
	- It would have been clearer to put a separate subsection for describing the multi-modal extension case from Sec. 3.1, because the current description is written in a back and forth way. The current description would make the readers distracted to concentrate on the cores.
	- The main and key experiments should come first, and the other side experiments including the ablation study go later. Thereby, the reader can get a clear goal and achievement of the development first before going to the details.
	- The current two paragraphs right above Eq. (4) are wordy and hard to take the true intention of the context. For example, in the second paragraph, $k$ is mentioned but the discussion is already made before introducing the algorithm in the previous paragraph; thus, not smoothly linked well what the authors want to convey in the first paragraph. It would be improved by rewriting those paragraphs to clearly link between the algorithm and the design principles.

review_title: Good performance, but many rooms to be improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes a new framework for novel category discovery, i.e., assigning novel class labels of an unlabeled dataset from a given (different but relevant) labeled dataset. Specifically, the framework (a) combines SimCLR, supervised contrastive learning, supervised learning, and consistency loss to learn a better representation from a mixed dataset of labeled and unlabeled samples, and (b) proposes a loss for learning a cluster assignment based on the winner-take-all coding of the learned representation from (a). Experimental results on image and video datasets confirm the effectiveness of the proposed framework compared to prior works, in terms of the average clustering accuracy on samples with unseen classes.  

Overall, the paper is well-written. The method is clearly explained given the complexity of the method itself. I liked their strong experimental results, and a thorough ablation study performed to verify the individual effects of each component of the method. It is also inspiring to see their another ablation study to explore different modality pairs possible in multi-modal contrastive learning, which concludes that contrasting against different modalities leads a better representation (at least) for the target task.
- p5, "Concretely, we randomly generate a set of H permutations ...": It would be nice if the paper could clarify what each permutation actually permutes, i.e., the domain. Also, are those permutations kept unchanged for both training and inference, or else?
- Eq. 6: I feel L^CE is somewhat less motivated than the others - I found (possibly) L^NCE-C already forces the representation to be class-discriminative, then why should L^CE be crucial for the method to work, as shown in the ablation study? I think more rationale behind this could be provided in the paper.
- Eq. 6: Should L^MSE and L^CL be always jointly ramped up and down? What happens if L^MSE is not ramped-up, i.e., not weighted by w(r)?
- Could the proposed framework still work when there is no available labeled sample, i.e., for pure unsupervised clustering?
- Table 3: Why DTC+CL and RS+CL are not considered?

---
Post-rebuttal update 

I have read other reviews and responses, and I am willing to decrease my score to 5.
My initial review was based on assumption that RS (Hen et al., 2020) is significantly different to this paper, but after reading other reviews and responses now I can see that it is indeed questionable. Also, It was somewhat surprising to me that the paper copied wrong values from the RS paper, as pointed out by R5, considering that RS is the closest related work of the paper.

Although I still think the proposed method is reasonable and the empirical results are impressive, I agree with other reviewers that the paper should have done more analysis on why the proposed method works to convince the reviewers (and the future readers), and to give more insights on why the combination of losses (and the use of WTA) is a right way to go.

review_title: Official Blind Review #1

Meta-review: The reviewers unanimously raised concerns on the lack of insights on why the proposed method works better than Han et al., 2020, and why WTA brings significant gains only to the proposed method and not to Han et al. I think the paper is promising but providing these insights are critical to making the work convincing to the readers. The reviewers have made excellent points to improve the paper; I'd recommend the authors to incorporate them in their future submission.
Decision: Reject