Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed to condition the final representation on an invariance descriptor. By doing so, the representation can be adaptively adjusted for target tasks that may require different kinds of invariance. 
strength_and_weaknesses: ## Strength
1. Thorough analysis to support the claims made in the paper. For example, Fig 1 shows that the final representation indeed adjust adaptively according to the invariance descriptor. Fig 2 also demonstrates that the model can identify the invariance required intrinsically for each downstream task.

1. Strong transfer results to various downstream tasks. This also implicitly confirms the (somewhat intuitive) hypothesis that each downstream task requires different invariance.

1. The authors demonstrate the generalizability of the proposed method on different (1) archs (ResNet and ViT), (2) SSL methods (MoCo and SimCLR), (3) downstream tasks (10 of them spanning across regression and classification tasks), and (4) modalities (vision and audio)

## Weakness

1. It is unclear why training the model with the vanilla contrastive loss in Eq 2 and 3 can encourage the model to produce the final representation conditioned on the invariance descriptor. Without any additional loss or special training techniques, the model can simply ignore the invariance descriptor and degenerate to vanilla SSL models like SimCLR or MoCo. The authors should highlight this part in the Method section.

1. Poor figure quality. For example, they are not vectorized figures and the resolution is not good. The fonts are too small and the shape is distorted (eg. Fig 2)
clarity,_quality,_novelty_and_reproducibility: Overall a well-written paper. Most of the claims are well supported by the analyses and experiments. The quality of the figures (resolution, font, non-vector, etc) needs some improvements.
summary_of_the_review: Overall a good and novel paper. Thorough analyses are conducted to support the claims made in the paper and the final results indeed outperform vanilla SSL methods on various downstream tasks that may potentially require different kinds of invariance.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper aims to learn more generalizable representations. To this end, this paper proposes amortized invariance learning, which learns augmentation-specific invariance by parameterizing the feature extractor by invariance hyperparameters. This paper shows the superiority of amortized invariance learning under various transfer learning benchmarks with both CNNs and ViTs.

strength_and_weaknesses: Strengths
- The problem of interest, learning augmentation-relevant information, is an important problem in self-supervised learning literature.
- The proposed idea, learning a hyper-network to learn augmentation-specific invariance, is interesting.
- When using CNNs, the proposed method outperforms existing SSL methods that use linear evaluation for downstream tasks.

Weaknesses
- This paper should be compared with fine-tuning, not linear evaluation. This is because the full propagation for whole parameters is necessary to optimize an invariance descriptor. Therefore, its time requirement is the same as fine-tuning. I think fine-tuning can perform better than the proposed method.
- In ViT experiments, the gain over the baseline (MoCo-v3) is marginal.
  - Prompt-ViT outperforms MoCo-v3 on 6 of 10 downstream classification tasks and 4 of 8 few-shot tasks (10 of 18, i.e., 55% in total). This means Prompt-ViT achieves a similar performance to the baseline.
- Theorem 1 is hard to understand and seems to be meaningless.
  - What is $\hat{y}$? What is the definition of the prediction $\hat{y}$?
  - The authors should describe details in the proof of Theorem 1. The sentence "Our result follows from using the union bound to optimise over the choice of feature extractor." is not enough for formal proof.
  - I cannot see the meaning of Theorem 1. The theorem does not tell the benefit of the proposed method. I think the theorem is just a well-known generalization bound (except the third term). The bound can be obtained in any case: fine-tuning, full-invariance learning (e.g., MoCo), etc. IMO, the authors should (theoretically) discuss an advantage over the baseline (linear evaluation using representations obtained from full-invariance learning).
- The writing of this paper can be improved.
  - Confused notation $i$: an invariance descriptor or an instance index (e.g,. see Eq (2)).
  - Confused notation $\mathcal{K}$. In the denominator of Eq (2), $k_{A_i}^i$ is not defined. I suggest using $\sum_{k\in\mathcal{K}}\exp(q_{A_i}\cdot k/\tau)$.
  - Confused notation $\mathcal{D}^t$: pretraining or downstream task dataset.
  - When using MoCo, how to compute the key representations $\mathcal{K}$ for an invariance descriptor $i$?
  - I suggest mentioning that $i$ is a continuous vector when defining $i$.
  - The authors should describe formal details of Hyper-ResNet in the main manuscript.
  - Figure 1 is too blurred. Suggest using high-resolution images.

clarity,_quality,_novelty_and_reproducibility: The details are described in the previous section. In summary,
- Clarity :: The used notations and some explanations are unclear.
- Quality :: Empirical results are convincing in CNN experiments, but not in ViT ones. Also, this paper does not compare the proposed method with the important baseline, fine-tuning. Theoretical results are not interesting.
- Novelty :: I think The proposed idea is novel.
- Reproducibility :: This paper describes implementation details, but code is not available.

summary_of_the_review: Although I feel the methodological novelty, both the empirical and theoretical results are not convincing. Hence, I vote for rejection.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Current SSL representations are pretrained to be invariant to a prespecified fixed set of augmentations. This paper provides a simple way of pretraining representations that can be modified to be invariant to any subset of the pretraining augmentations. They essentially learn a function(al) that maps a set of augmentations to a pretrained encoder invariant to those augmentations. For CNNs this function maps augmentations to weights of the network, while for transformers this simply corresponds to appending the index of augmentations to the input sequence. 

They evaluate their representation on transfer learning benchmarks by training both the linear probe and the set of augmentations to be invariant to.  They show significant gains compared to linear probing of standard SSL and  previous baselines that aim to learn representations that are not fully invariant.
strength_and_weaknesses: **Update**: the authors updated the manuscript to address many of my concerns. In particular, improving the baselines and discussing the computational complexity of their method. I am thus increasing my score 5->6.

-------
**Strenghts**
- **really cool idea** the idea of learning a functional from desired invariances to representations is really nice and I think will be valuable for the community. I actually think that the authors undersell it's usefulness. Eg even when downstream practitioners do know what invariances they want in their downstream tasks, they can now get the desired representations without any retraining. The current paper puts all the emphasis on learning task-specific invariances, but this is only one specific application (and arguably not the best as I say below).
- **simple method** the proposed method is very simple for ViTs, it could thus be easily incorporated in SSL and has the potential of being impactful assuming that the method is indeed useful (see weakness about baselines below). Note: for CNNs it's more complex which will likely hinder it's usefulness.

**Weaknesses**
- (addressed in rebuttal) **Unsuitable baselines: what are the advantages compared to fine-tuning?** my main issue with all the experiments is that I don't think that any of the selected baselines are fair comparisons because those only train the linear probe. Take for example the case of ResNet50, by tuning over the augmentation index i (which directly impacts the weights of your model) you are essentially finetuning all the parameters of your model but comparing to baselines that only train the linear probe. Looking at the Mocov3 results (table 4 vs original paper) it seems that your method performs much worst than full fine-tuning. So what are the advantages of your method compared to full fine-tuning? Is it computational or sample efficiency? If so you should compare to fine-tuning methods under the same computational or sample budget (ie only doing a few passes or doing few shot fine-tuning) [^1]. Your generalization bounds try to give a potential explanation for the gains in terms of better sample efficiency, but this has to be empirically validated given that those bounds are well known to be loose. 
- (addressed in rebuttal)  **lack of discussion on scalability** specifically :
    - how efficient are the hypernetworks? what is the increase in the number of parameters and training time?
    - it seems that you restrict yourselves to very few augmentations. Does your method easily scale to many more augmentations?  eg the 14 augmentations set used in autoaugment? Pretraining the model for essentially all typical image transformations would be much more useful as it would allow practitioners to have a fine-grained choice on the desired augmentations.


[^1] Note that the advantage of linear probing vs finetuning is very large in terms of computational gains: because you can prefeaturize your dataset once before training the linear probe, making it orders of magnitude more efficient. In your case, you still have to encode every example at every step. I thus doubt that your hypernetwork is more computationally efficient that fine-tuning a RN50. Even for ViT the gains will likely be minor given that the bottleneck is usually encoding.

clarity,_quality,_novelty_and_reproducibility: **Clarity**: the paper is generally pretty clear, although it could definitely be better. My main two issues in that respect are: 
    - the explanation of the method (section 3.1) is relatively unclear despite its simplicity. I would highly suggest that the authors add a figure to simply and succinctly show the idea of their method.
    - the authors spend a significant amount of space and experiments on dorsal and ventral invariances, which I find more confusing than useful. First, the names will be confusing for most readers, why not use the standard and self-explanatory "color" vs "geometric"? Second, I don't quite understand the purpose of all those experiments and results. If the point was to have more interpretable results by aggregating over similar augmentations, why not train on all augmentations independently and then simply show the aggregated invariance hyperparameter (eg avg overall dorsal invariance hyperparameter rather than train another model)?

**Quality** my main issue about the paper's quality concerns:
- lack of comparison to the main alternative: full fine-tuning
- lack of discussion about computational efficiency
- figures/tables are generally of poor quality. Eg:
    - text in figures is unreadably small and blurry
    - tables do not follow ICLR requirement of having legends above  the table
    - axes are not always explained (eg figure 1 left) or not self-explanatory without reading the paper (eg the use of "dorsal" and "ventral" instead of standard "geometric" vs "color" augmentations)

**Novelty**: the idea (and instantiation of it) is novel to my knowledge.

**Reproducibility**: seems ok, code snippets and many hyperparameters are provided.
summary_of_the_review: The overall idea of the paper is very nice and simple. I believe that it could be useful to practitioners and the research community. My main concern is that the authors only consider a single application for their method (using the data from downstream task to update the parameters) but do not compare or discuss the obvious baseline for this setting: fine-tuning. I thus do not think that it is currently ready for publications. I am happy to improve my score if the authors address this concern, for example by doing one of the following:
- empirically showing advantages compared to fine-tuning (better accuracy, computational efficiency ...)
- explaining why fine-tuning is not the right baseline
- talking and ideally showcasing about other potential applications
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, authors propose a method to dynamically learn/adjust what invariances a self-supervised representation should be robust to such that it can cater to various types of downstream tasks which may have conflicting invariances. Towards this end, they propose a hyper-network based learning method which can learn the proper invariances for a given target task. Experiments show that learning task-specific invariances (via various stochastic augmentation policies) works better compared to the original SSL models (SimCLR and MoCo) which uses a fixed augmentation policy for all tasks. The proposed method also outperforms existing works from the literature which tries to solve the same problem (e.g. LooC). Authors  perform experiments with two family of architectures - ResNets and ViT and also two task modalities - vision and audio and show improvements on most, if not all cases.
strength_and_weaknesses: Strengths
========

Narrative
-----------
The paper is written in a clear way - starting from the problem formulation and why it is important, followed by clearly motivating their proposed approach and positioning it against the baselines and other relevant methods from the literature and finally showing experimental results with different architectures and task modalities. 

Practicality
-------------
Apart from only considering the formulation or theoretical properties, authors also paid attention to make sure their proposed algorithm can be trained without a lot of tweaking and training can scale to standard SSL benchmarks like ImageNet-1K. Towards this, authors modified the hyper-network formulation for ResNets. For ViTs, they proposed a different prompt tuning based formulation as they found hyper-networks + ViTs are harder to train. There are also additional tricks mentioned in the paper for ViTs which are useful for reproduction. I appreciate the effort to make sure the proposed algorithm can be used by practitioners.

Coverage
------------
Authors perform experiments using multiple classification and regression tasks for visual recognition, few-shot classification and also performed experiments in the audio domain. As mentioned before, they experimented with two different family of architectures - ViT and CNN. I believe the empirical setup is strong and helps to properly evaluate the contributions of this work. 

Weakness
================
Additional details on test-time adaptation will be helpful
-----------------------------------------------------------------
More details on the test-time adaptation procedure will be helpful as that is the core contribution of this work. Along with that, some explanation of the basic hyper-network formulation will be useful as well. I have gone through the code provided in the appendix (for BottleNeck layer of ResNet) but I am unclear as to how this is plugged into the end-to-end optimization procedure. For example, I am not sure what is the shape of the invariance hyperparameters for ResNet for every block and how are those parameters getting updated. 

Connecting to other types of work
----------------------------------------- 
Although there has not been any work in terms of learning proper augmentations or learning a differentiable masking in the particular context of SSL, there has been prior works independently. For example, [this paper](https://arxiv.org/abs/2104.04282) learns the optimal data augmentation policy by differentiable search. [AutoAugment](https://arxiv.org/abs/1805.09501) does a similar thing using reinforcement learning. How does the proposed hyper-networks based method compare to such methods to find the right data augmentation for the given task? Can these algorithms be used to search for the augmentation strategy instead which will be in sync with the optimal invariances to learn each task?

Additional Questions
=================

* While trying to learn the right augmentation strategy for a given task, I do not see a justification why a hyper-network based strategy was used as opposed to using other ideas from the literature like straight-through estimator or using a standard meta-learning algorithm. 
* While adapting to the downstream task, in Eq. 4, what will happen if we rather minimize the loss on a validation set like meta-learning? Why did you choose to minimize it on a train set instead of validation?
* Although there has been considerable effort made to reduce the computation time, still, adding hyper-networks require making three additional forward pass calls for each block in a residual network. How much additional time does these calls cause? Assuming they cause 3x, that is equivalent to performing downstream adaptation with three different augmentation strategies as hyperparameters. In that case, there is no practical advantage of this algorithm as it also tries to choose between three combinations (dorsal, ventral and both).
* What is the time difference between ResNet + hyper-network and ViT with prompt tuning? Also, my understanding is that these additional calls for hyper-networks are only needed during downstream adaptation and not pre-training - is that correct? 
* When you mention ViTs are harder to train with the hyper-networks formulation, does it mean it needs more hyperparameters tuning or is the training itself unstable and leads to NaN and other problems?

clarity,_quality,_novelty_and_reproducibility: Clarity
=====
The narrative is clear, easy to follow and enjoyable to read. The paper motivates the problem statement well, clearly explains the high-level approach and provides empirical justification. However, as mentioned above, more clarity on the downstream adaptation algorithm in the main manuscript will be beneficial for the readers.

Quality
=====
In my opinion, this works meets the bar for quality for a top-tier ML conference. The presentation is crisp, amount of contribution is non-trivial and experimental results are strong. 

Novelty
======
The idea of using hyper-networks to learn the proper augmentation strategy such that right kind of invariances are preserved for a given downstream task adaptation is novel. However, I'd like the authors to clarify my comments regarding how this is different from other algorithms that also learn the optimal data augmentations for a given task before I finalize my thoughts on novelty.

Reproducibility
============
The authors provide most of the details and implementation notes that will be required to reimplement the algorithms. However, as I pointed out above, additional details on the end-to-end optimization process with hyper-networks and ViT + prompt-tuning will be helpful. 

summary_of_the_review: Overall, I think the paper takes a new approach to solve a well-known and important problem for self-supervised representation learning. The proposed method intuitively makes sense at a high-level and shows promise in empirical evaluations. However, I'd like for more clarifications regarding the details of the algorithm, the computational cost as well as some explanations regarding its difference with respect to a few other works from the literature. At this point, I am leaning towards marginally below acceptance threshold but I will be willing to increase my score if authors provide satisfactory responses to my comments above. 

Updates after Rebuttal
---------------------------
Updated score by one point based on all the discussions between the authors and myself.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes amortized invariance learning for achieving generalizable representations. Most reviewers agree the idea is interesting and simple at the same time. The results are also demonstrated with different methods and network architectures in SSL. However, there are some concerns of the novelty and the correctness / clarifty of the paper. These have been mostly resolved after the rebuttal. The majority of the reviewers recommended accepting the paper and the AC agrees.
justification_for_why_not_higher_score: The authors are encouraged to clairfy the details in the paper as suggested by the reviewers.
Decision: Accept: poster