Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors introduce a method which is tailored to unsupervised representation learning via minimizing mutual information. Authors support their method with an experimental study. The proposed method has some merits like finding a representation that has higher mutual information compared to baselines.  
I have several concerns about the paper:
a)	Authors state that some greedy optimization may end up lower MI optima. Since deep neural networks are not convex, is this a surprising point or is this expected one?
b)	I think empirical comparison needs significant improvement. Authors mention the proposed method was outperforming the state of the art “at the time of writing”. However, currently I believe this is not the case. For example, GATCluster achieves 28.1% on Cifar100-20. 
c)	I see the proposed method as shown in Fig 1, is novel. As far as I understand it is adding some extra blocks to already existing method and calculates mutual information between heads. I would like to point out that hierarchical ordering is an interesting idea however the impact of hierarchical ordering is not very clear. Why does architecture need extra heads i.e. h_4 to h_8.  Would you please emphasize the novelty of the idea bit more?
Although I have some concerns about the paper, I would like to be extremely clear that I am open to change my view if more explanation and/or evidence supplied.

review_title: MI based unsupervised representation learning

final_score: rating: 3: Clear rejection
review_content: <Summary>

This paper addresses the problem of unsupervised learning of class representation using data augmentation. Its key idea is to encourage the learned representations to have low MI while maximizing the original augmentation-driven MI objective. It reports the improved performance for the benchmarks of Ji et al. 2019 – classification on some easy datasets (e.g. CIFAR-10, CINIC-10, SVHN and STL-10).
 
<Strengths> 

1. The proposed approach proposes an MI minimization strategy for unsupervised clustering that can obtain more informative representation than previous methods.

2. The proposed DHOG method shows better performance than some baselines reported in  Ji et al. 2019 for some easy classification tasks on  CIFAR-10, CINIC-10, SVHN and STL-10.

<Weakness>

1. This work summarizes the two contributions in sec 1.2. However, both of them are not fully supported. 

(1) This work claims that “the current methods do not effectively maximize the MI objective because greedy SGD typically results in suboptimal local optima.” However, there is neither theoretical justification nor empirical results that show why the proposed approach does not suffer from this issue and why it is subsequently better than other works. 

(2) This argument may be based on that the proposed DHOG adopts distinct head that encourages different solutions between heads one another and a simple-to-complex hierarchy structure to the heads. Conceptually, they could be reasonable solutions to the issue, but there is no theoretic and empirical evidence that supports this argument. 

(3) The other contribution that this work claims is strong performance of unsupervised clustering on benchmark datasets. However, the empirical comparison is only carried out with baselines in Ji et al., 2019. 

(4) More critically, this work fails to cite the following recently published papers that reported much stronger performance on the same datasets. 

a. W. Van Gansbek et al, SCAN: Learning to Classify Images without Labels, ECCV 2020.
b. G. Shiran & D. Weinshall  Multi-Modal Deep Clustering: Unsupervised Partitioning of Image, ICLR 2021.
c. N. Astorga et al. MPCC: Matching Priors and Conditionals for Clustering, ECCV 2020.

<Conclusion>

My initial decision toward this submission is rejection because the main contributions of this work are not fully convinced. Moreover, this work ignores key related papers that show better performance on the same benchmarks.

review_title: Review #1

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an extension to the Invariant Information Clustering (IIC) methodology for image clustering. 
IIC is an end-to-end deep clustering methodology that directly provides the cluster label of an input image. 
Training is based on maximizing Mutual Information (MI) between pairs of related images (typically generated through augmentations).
In this work the authors argue that IIC performs suboptimal maximization of MI, thus the obtained solutions focus mainly on low-level representations and do not distill higher level semantic similarity of images.
To solve this problem, they propose an IIC extension (called DHOG) where multipe clustering solutions are obtained through MI maximization, under the constraint that the solutions found are sufficiently different (ie. minimum MI between the clustering solutions is seeked).

The proposed objective function is novel, however it involves a regularization constant \alpha to be set by the user. 

The idea of stopping gradients to impose a hierarchy of solutions is interesting.

The experimental results indicate improved clustering performance.
 
Comments to be addressed:
1) Application of the method requires the specification of the number of K heads. It in this work K=8 is set. What is the reasoning for such selection and how does the value of K affect the results?
2) DHOG is computatonally more complex compared to IIC. What is the execution time of DHOG compared to IIC?
3) What happens for values of \alpha greater that 0.05? How sensitive is the method on the value of \alpha?
4) It is strongly suggested to include a visual example illustrating the difference among the obtained clustering solutions (e.g. on CIFAR-10) to provide visual evidence that improved clustering solutions are progressively found.  
5) It is not clear whether IIC results are obtained using Sobel preprocssing or not. What if Sobel preprocessing is used in DHOG?   
6) The presentation of the paper could be improved, by first briefly describing the IIC approach and then presenting the proposed extension (DHOG).
review_title: An extension to Invariant Information Clustering (IIC) aiming to provide more informative solutions

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors study the problem of unsupervised representation learning from data augmentations. Specifically, the authors claim that existing methods are prone to getting stuck at local minima owing to easy-to-learn local representations that optimise the commonly used MI objectives, and then propose a hierarchical method that tackles optimisation at multiple layers of the feature hierarchy.

Strengths:
- A highly novel solution to an important problem in representation learning.
- Strong results obtained with respect to the state of the art.
- Extensive analysis and comparisons.

Weaknesses:
- "We demonstrate that current methods do not effectively maximise the MI objective" + footnote: "We show this by finding higher mutual information solutions using DHOG, rather than by any analysis of the solutions themselves." => This claim calls for an analysis of the solutions and since this is missing in the paper, I would rephrase the claim.

- It would have been nicer to have experimental analysis on different features (e.g. colour) being more prone to local optima, though this is intuitive. This could have significantly increased the impact of the paper.

Minor comments:
- "a reasonable mapping need only compute colour information" => "a reasonable mapping needs only compute colour information".
- "Learning a set of representations by encouraging them to have low MI," => This should be high MI?
- "CIFAR-10, CIFAR-100-20 (a 20-way" => The parenthesis is not closed.
- "A network was learned to associate" => "A network was trained to associate".

**AFTER AUTHOR RESPONSE**

I have read the other reviewers carefully and the feedback provided by the authors. The reviewers had two major concerns: (i) Theoretical or empirical justification/proof for the following claim (and the motivation) of the paper: “the current methods do not effectively maximize the MI objective because greedy SGD typically results in suboptimal local optima”. (ii) Lack of comparisons with newer methods from e.g. ECCV2020 etc.

For the second, I feel empathy with the authors: In such a rapidly progressing field, it is difficult to integrate comparisons with new methods that are published while writing/submitting a paper. I am sure all of us have experienced similar problems.

For the first issue, I disagree with the authors and agree with the other reviewers: This is an important claim that needs to be justified and I disagree with the authors's comment on the current results of the paper being a sufficient empirical evidence for the claim. An ICLR paper should have provided the necessary evidences and justifications.
review_title: Novel paper, nice results

Meta-review: During the discussion phase, although the reviewers acknowledge superior empirical performance of the proposed method, they shared the two major concerns:
1. Lack of theoretical or empirical justification/proof for the key statement: "the current methods do not effectively maximize the MI objective because greedy SGD typically results in suboptimal local optima".
1. Lack of comparisons with newer methods from e.g. ECCV2020 etc.

In particular, the first point is crucial. As the reviewers pointed out, since it is the main contribution and the key message of this paper, it should be carefully examined theoretically and/or empirically. However, in its current state, there is no theoretical analysis, and empirical evaluation is not convincing. 

About the second point, although I think it cannot be a solo reason for rejection, at least it is better to cite and discuss it fo the completeness.

Overall, the contribution of this paper it not significant enough for publication. Hence I will reject the paper.
Decision: Reject