Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes an unsupervised extension of the closed-loop transcription, named as U-CTRL, to learn the better representation for discriminative and generative learning tasks. The authors show that the learned representation is more structured, i.e. having the cluster shapes in semantically-meaningful way, improving the performance of linear evaluation and conditional generative tasks. 
strength_and_weaknesses: **Strengths**:

* This manuscript is basically very well-written, and clearly summarizes the necessary previous works to understand the main argument. It's easy to follow most of the parts in the manuscript. 
* It seems that the approach based on closed-loop transcription and rate-distortion theory for learning better representation is quite principal. 
* This manuscript includes all the details in the experiments, including detailed setup, hyperparameters, and network configuration. It seems that all the results are reproducible. 


**Weaknesses**: 

My biggest concern on this work is the low performance of discriminative learning shown in Table 2. Compared to the similar baselines, Table 2 is not enough to argue that the learned representation by U-CTRL is highly discriminative. I extracted the linear evaluation experiments in GCRL and W-MSE, as shown in below:


|Method| CIFAR-10 | CIFAR-100 | Tiny ImageNet|
|----------|----------|----------|----------|
|U-CTRL| 87.4 | 55.2 | 36.0 | 
|GCRL | 83.9-95.1 | 58.7 - 76.0 | - | 
|W-MSE | 91.6 - 92.0 | 66.1 - 67.6 | 48.2 - 49.2 |

Besides the comparison to GCRL and W-MSE, I’ve found that the performance of BYOL in Table 2 is much worse than the reported numbers in Table 1 in W-MSE paper. Could you explain a little bit more why the performance gap is observed? 

I’m also still not convinced that the learned representation based on U-CTRL is more structured than SSL methods, especially the methods based on contrastive losses. Therefore, it could be necessary to measure the cluster quality of representation learned by contrastive losses, since the contrastive loss enforces the semantically-meaningful clusters in the representation space. In this work, U-CTRL was only compared to the learned representation based on generative learning, which is not sufficient to argue that the representation from U-CTRL is more structured than the ones from existing approaches. Figure 5 seems to support the argument, but it could be better to include the results of SimCLR and BYOL. 

This work also argues that the structured representation improves the quality of generated samples a lot. However, as shown in unCLIP (a.k.a. DALL-E 2), the representation based on a simple contrastive loss definitely helps to train a decoder. It means that the learned representation based on contrastive losses is already structured enough to train a generative model.

References:
* [W-MSE] Whitening for Self-Supervised Representation Learning, ICML’21. 
* [GCRL] Hybrid Generative-Contrastive Representation LEarning, arXiv’21.
* [unCLIP] Hierarchical Text-Conditional Image Generation with CLIP Latents, arXiv’22.
clarity,_quality,_novelty_and_reproducibility: In terms of clarity, I really enjoyed reading the manuscript. Most parts of this manuscript is clearly well-written. And, in terms of reproducibility, many details are well described, so I believe that all the results could be reproducible. 

In terms of quality and novelty, as described in the section above, I’ve raised some issues on the experiments and motivation. 
summary_of_the_review: My initial evaluation is borderline, slightly leaning towards rejection. My major concern on this work is the insufficient empirical justification. In specific, I’m not sure that the learned representation based on U-CTRL is discriminative enough compared to the similar recent works. In addition, I failed to figure out that the representation learned by U-CTRL is more structured than the ones by contrastive losses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper claims to extend the previously proposed CTRL framework for representation learning to unsupervised setting. Paper proposes self-consistency and augmented sample compression based constraints to define an objective function for representation learning. Several experiments evaluate the representations learned by the proposed method by comparing against both discriminative and generative prior art.
strength_and_weaknesses: Strengths
- Strong performance against generative baselines, although on smaller datasets.

Weaknesses
- Unclear contribution: While the paper explicitly claims that it extends the CTRL framework to unsupervised setting, eq 4 already appears in Dai et al., 2022. Unless I am mistaken, CTRL framework as in Dai et al, 2022 is not limited to supervised setting only. This fact dilutes the claimed contribution. However, the incorporation of constraints (eq 5, 6) appears to be new.
- Limited Novelty: It seems that incorporation of self-consistency (eq 5) and augmentation (eq 6) are the main contributions of the paper. These on their own seem to be marginal at best. While experiments show that these do lead to superior results, some crucial ablations are missing (see next pt.)
- Missing ablations: Given that constraints listed in eq 5 and eq6 are the main contribution, these choices need to be studied. More specifically, how does the performance change from no constraint, to only first constraints, to only second constraint, to both of them?
- Inferiority to "discriminative" methods: The methods categorized under discriminative do not use labels (just like the proposed method), and appear to outperform it on the evaluated datasets. Further, more recent methods e.g. DINO [1], are superior to these baselines, while using a "simpler" training objective in comparison to the proposed method. This further reduces the appeal of the proposed method.
- Missing larger scale experiments: Most recent representation learning methods report their performance on the larger ImageNet dataset, to more closely mimic the practical setting. It would be beneficial to have that comparison to place the proposed work w.r.t. to these methods.



[1} Emerging Properties in Self-Supervised Vision Transformers, Caron et al.




clarity,_quality,_novelty_and_reproducibility: - Clarity: Paper is clearly written
- Quality: Par with an average submission with promising early results.
- Novelty: Marginal at best, as the main contribution seems to be the two constraints described in eq 5 and eq 6., impact of which is not ablated in experiments.
- Reproducibility: Training details etc. are provided in the appendix, however the proposed method seems non-trivial in terms of training. Sharing code would be quite useful.
summary_of_the_review: I am leaning reject as the contributions seem marginal at best. Further, method is behind in terms of learned representation quality in comparison to discriminative methods. In addition, there are several other issues as detailed in the weaknesses section.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper illustrates a learning framework toward a unified representation which is effective for both generative and discriminative purposes.
- To do so, this paper introduce the idea of recent work, "Closed-Loop Transcription (CTRL) via a Constrained Maximin Game" in an unsupervised fashion.
- Specifically, the proposed method minimize the rate reduction between $Z$ and $\hat{Z}$, where $X \xrightarrow{f(x,\theta)} Z \xrightarrow{g(z,\eta)} \hat{X}  \xrightarrow{f(x,\theta)} \hat{Z}$, $f(x,\theta)$ is encoder and $g(z,\eta)$ is decoder.
- Then, the rate reduction between two code are defined as: $\Delta R(Z, \hat{Z}) \coloneqq R(Z \cup \hat{Z}) - \frac{1}{2}(R(Z) + R(\hat{Z}))$, where $R(Z)\coloneqq\frac{1}{2}\log\det(I+\frac{d}{N\epsilon^2}ZZ^\top) $.
- This paper also propose auxiliary loss term to 1) improve sample-wise self-consistency: $\sum\limits_{i\in N} \Delta R(z^i, \hat{z^i})$, and 2) self-supervision:  $\sum\limits_{i\in N} \Delta R(z^i, \hat{z}_a^i)$.
- Putting all this together, the unsupervised CTRL objective is defined as follows: 
$\max\limits_\theta R(Z) + \Delta R(Z, \hat{Z}) - \lambda_1 \sum\limits_{i\in N} \Delta R(z^i, \hat{z^i}) - \lambda_2 \sum\limits_{i\in N} \Delta R(z^i, \hat{z}_a^i)$
- $\min\limits_\eta R(Z) + \Delta R(Z, \hat{Z}) + \lambda_1 \sum\limits_{i\in N} \Delta R(z^i, \hat{z^i}) + \lambda_2 \sum\limits_{i\in N} \Delta R(z^i, \hat{z}_a^i)$
- The experiments are conducted on three datasets: CIFAR10, CIFAR100, Tiny-ImageNet.
- The proposed U-CTRL shows clear performance gains over generative self-supervised learning methods.
- Its performance is also comparable to the discriminative self-supervised learning such as SimCLR, MOCO, BYOL.
- Its generative performance (UGIG task) is better than other generative models.
- By leveraging its generative objective, domain transfer performance is also better than other self-supervised methods.
strength_and_weaknesses: Strength:
1. The motivation is clear: one unified representation for both generative and discriminative purposes.
2. This paper is written clearly.
3. The empirical results are sufficiently strong enough to understand the effectiveness of the proposed method.

Weakness:
1. The proposed method is built upon the idea of CRTL-binary (i.e., $\max\limits_\theta \min\limits_\eta \Delta R(Z, \hat{Z})$). 
To understand the contribution of this paper, ablation study of the additional loss terms (e.g., $R(Z), \sum\limits_{i\in N} \Delta R(z^i, \hat{z^i}), \sum\limits_{i\in N} \Delta R(z^i, \hat{z}_a^i) $  ) is highly recommended.
2. As noticed in the footnote of page 6, $ \Delta R(Z, \hat{Z})$ can be replaced with $\ell_2$ or cosine distance in practice and this trick cannot be applied to $R(Z)$ term. If rate reduction term is replaced with any simplified distances, it should be discussed in more details in the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: This paper is clearly written (see strength 2).

Quality: Except for that this paper does not include ablation study (weakness 1), the empirical evaluation is fairly conducted and the results are convincing (strength 3).

Novelty: The novelty of this paper is from the auxiliary loss terms of $R(Z), \sum\limits_{i\in N} \Delta R(z^i, \hat{z^i}), \sum\limits_{i\in N} \Delta R(z^i, \hat{z}_a^i) $ (see weakness 1). This is another reason to conduct an ablation study on these terms.

Reproducibility: This paper provide all the experimental details to reproduce the results.
summary_of_the_review: This paper propose U-CTRL for learning an unified representation (generative and discriminative). The empirical results are very convincing, however, an ablation study is highly recommended to find their novelty over previous work.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: 
The authors present an unsupervised approach, based on recent ideas
called 'closed-loop transcription', to extract feature encodings that
perform well on both discriminative and generative tasks, on CIFAR-10,
CIFAR-100 and Tiny Imagenet data sets.  Often there is a tradeoff
between generative vs discriminative goals, and their approach points
to the possibility of methods excelling at both (empirically).


strength_and_weaknesses: Strengths:

-- A variety of experiments, on several tasks, show that the technique
   is superior than existing unsupervised generative or discriminative
   techniques.


Weaknesses:

-- The paper is dense, and I didn't come out with an understanding of
   why the method may work better.

-- The cost of optimization is not clear: how many rounds to
   convergence? (for 8 and 9)


Rewordings, etc:

-> (page 6, 'study how the..') Finally, the third set (Section 4.3)
study how the advantages that generative representations have over
discriminative ones.

-> (properties a whole greater) In addition, these results lead us to
 ask a fundamental question: when is incorporating both discriminative
 and generative properties a whole greater than the sum of its parts,
 particularly outside of the context of computational efficiency?

-> Why does superior clustering performance reflect the generative
   quality of the features: it seems to be that's another way of
   assessing the informativeness or the discriminative power of the
   features learned.

--> page 6: You mention 'structure' here and elsewhere: "In order to
facilitate discriminative or generative tasks, it must be highly
structured."  Do you just mean that the features come together to
reflect the underlying (hidden) class structure (rather than having structure
in the sense of having sub-parts?)


clarity,_quality,_novelty_and_reproducibility: 
The paper is well written overall, and develops the approach well, and
I believe the results are reproducible.  There are several clarity
issues that I raised above.


summary_of_the_review: This is a new idea, and shows good promise in a number of directions.
Many experiments and comparison are performed to showcase the
potential.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper introduces a new approach for unsupervised feature learning, which is motivated by the "closed-loop transcription" framework, CTRL. Overall it is quite different from many other current popular self-supervised or autoencoder-based methods for feature pretraining, and as such I think it is liable to be of general interest. Concerns from the reviewers focused on novelty (relative to the original supervised CTRL paper), and on the importance of the empirical results (particularly a lack of ablations of the two Lagrangian terms, other potential baselines, and worse-than-expected discriminative performance). The author response went some ways towards addressing these concerns; at least one reviewer raised their score. However, the overall set of recommendations is still mixed. 
justification_for_why_not_higher_score: The authors did a good job of updating this paper, but it is still a bit borderline, and I think that some of the additional comparisons and discussions requested by the reviewers are fair.

However, I would have no objection to seeing this accepted.
Decision: Reject