Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work presents an augmentation strategy leveraging the latent space of VAE and adversarial samples, named Identity-Disentangled Adversarial Augmentation (IDAA). IDAA perturbs the latent space of a pretrained variational auto-encoder to produce adversarial samples that preserve the identity attributes. Experiments show that IDAA improves the performance of existing SimCLR and SimSiam baselines. 
main_review: The idea to disentangle the identity with VAE is technically correct and the analysis in section 3 with the information bounds and ELBO provides a clear theoretical support method. The proposed IDAA seems to be able to improve the current augmentation strategy by removing the identity attributes when generating augmentations. A weakness here is IDAA seems to rely upon existing pre-defined augmentation combinations. The authors better tune down their claim or provide more support that IDAA can achieve on par performance even with random augmentation or without any augmentation. 

Although I like the analysis in section 3, I still have some concerns about the claim that IDAA can improve different SSL methods. As known recent SSL methods can be generally classified as contrastive and non-contrastive ones. The contrastive methods, such as SimCLR, mainly use InfoNCE loss, and the analysis in section 1 well supports the relation of IDAA to these methods. However, the relation to non-contrastive methods like SimSiam, SWAV, etc. is not clear since they do not use InfoNCE loss. Thus, I am curious whether IDAA really improves the performance of these methods. If the authors would like to claim IDAA generally improves existing SSL methods, I think it is also necessary to show how IDAA works with recent SSL methods such as MoCo, SWAV, BYOL, DINO, etc. 

I also have two main questions about the experiment part:

- The results reported in the paper seem not consistent with existing papers in SSL area. For example, SimCLR and SimSiam are known to achieve over 90% accuracy in linear classification when ResNet18 backbone with 800 epochs, and the results in Table 1 seem to have significant differences. In Table 1 CLAE seems to harm the performance of SimCLR and SimSiam, which is not consistent with the conclusion in their paper either. Based on these results, it is hard to judge whether IDAA improves existing SSL methods' performance and outperforms other augmentation strategies.

- According to $\beta$-VAE, the disentanglement is affected by choice of $\beta$, and they have better disentanglement in the latent space when $\beta$ is bigger than 1. According to Eqn (5) in the paper, it also seems the identity is better preserved from wrong augmentation when $\beta$ is bigger (please correct me if I am wrong). The result in Fig. 7 seems to suggest the best choice is $\beta=0.1$, which is a little bit out of my expectation and I am worried this could violate the lower bound shown in Eqn (8). Could the authors provide additional justifications with some analytical examples or provide some possible explanation?
summary_of_the_review: This paper provides a novel augmentation strategy that leverages the latent space of pretrained VAEs to produce adversarial samples. The technique part is generally correct but is limited to InfoNCE loss, and the experiment results are not completely consistent with existing papers. Based on these concerns, I had difficulty in judging the value of the proposed method. I am currently inclined to recommend a reject and I suggest the authors keep the same configuration of the baselines to justify the improvement of their method.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a new technique for disentanglement based inspired by the premise of an adversary that generate diverse samples that preserve identities via augmentation. The main motivation of the paper is for self-supervised learning which is an important area that benefits a lot from data augmentation. In particular, they use the reconstructed sample of a VAE and its distorted difference (G(x) - x) as an adversary to perform data augmentation and show its effectiveness via success on a number of different self-supervised learning tasks.
main_review: This work studies an important area of machine learning that is important to a vast literature in self-supervised learning. In particular, they propose a method that utilizes the strengths of deep variational autoencoders and their information theoretic properties to inherit identity disentanglement. 

The main strengths of the paper is the utility and simplicity of such a method. As illustrated on the extensive applications, the method is readily applicable to schemes and I see the work presented in this paper to be extremely useful for the machine learning community at large for applications in data augmentation beyond those described in this work. In particular, this work may be useful for adversarial training via augmentation, which is a closely related area and given the 'adversarial' motivation hinted in this paper, can be developed into such a scheme.

The simplicity of the paper also serves to some extent as a weakness. This is due to the fact that there are several questions that may be asked albeit not taking too much merit away from the idea. The first is how specific is this method to the VAE? While the main reason for using VAEs is the link to information theory (the appearance of mutual information), which is the reason the main theorems work, I wonder how applicable this result would be at least heuristically with other deep generative methods such as the Wasserstein Autoencoder for example. I think such avenues can be deferred to future work however I would be interested to know the author's position on this.

The paper is very well presented and I found the level of exposition very high with little or no issues understanding the contribution of this work. I am a non-expert in this area however one small question that does emerge is the motivation of using an adversary for this task. While I understand the adversary takes the role to strengthen the identity preservation, I do feel that this paper may be conflated with adversarial training, which as mentioned earlier is a potential application yet not explored. This is however a minor point as a non-expert, it may be a common phrase used for disentanglement.
summary_of_the_review: I believe the paper presents a new method and serves to a useful problem of machine learning for a number of tasks. Due to the simplicity and utility of the method, I lean towards acceptance of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel augmentation method for contrastive SSL. To this end, the authors leverage VAEs—generating an augmentation for an image x by adversarially perturbing it's encoding (z = E(x)) to generate a perturbed reconstruction (G(z')), which is then added back to the original residual (R(x) = x - G(x)). The motivation for this approach is to maintain the identity preserving information via R(x), while creating challenging augmentations. The authors empirically evaluate their approach for downstream classification, transfer learning and semi-supervised learning.
main_review: Finding better augmentation schemes for self-supervised learning is undoubtedly and important and challenging problem. While the proposed method does seem to offer empirical gains with respect to baselines, I am not completely convinced about the motivation and some of the theoretical claims.

In particular, in Theorem 1, the authors claim that they can perturb z to produce x' without hurting the lower bound. However, it seems odd that this theorem does not have a dependency on $\delta$ (that is used to perturb z). After all, one could construct a z' that is arbitrarily different from z (using a large delta). In this case, x' could be very different from x and I(x', y) would be very small (possibly violating the lower bound). The proof utilizes I(R(x) + G'(x); y) ≥ I(R(x); y), without explaining why this is the case. Overall, I do not see how this is true for arbitrary $\delta$.

Other comments:

* The claims made in Section 4.1 regarding the qualitative differences between augmentation samples (in Figure 4) are subjective and hand-wavy. Based on visual inspection alone, the difference in quality between these augmentations is not apparent and thus should be removed.
* Baselines:
    - Why do the authors only consider SimCLR in Table 2?
    - In Table 3, why are CLAE and random augmentations missing?
    - I am curious whether simply attacking the z of a standard VAE (with beta=1) to produce an x' = G(z') would lead to vastly different performance. In general, I am not convinced that the decomposition into the generated and residual components is necessary. This seems like an essential baselines to compare to.
* All the plots in the paper need to be made larger and more legible.
summary_of_the_review: While the empirical results of this paper are interesting, I still have concerns about the theoretical claims and the comparison to baselines. I would be happy to increase my score if the authors addressed these.


#### Post-rebuttal Update ####

I thank the authors for their detailed response. Based on the additional experimental results that the authors presented to all the reviewers, I have decided to increase my score to a 6. I still have two comments:

- The authors' response regarding theorem 1 does not completely address my concern. Theorem 1 as stated right now allows z to be perturbed by an arbitrary $\delta$, regardless of x. (In contrast, in adversarial training x itself is perturbed by $\delta$ which is constrained by some $\epsilon$.) Thus, it is still not clear how the bound in Theorem 1 can hold for some arbitrary $\delta$. For instance, imagine we have two images $x_1$ and $x_2$ and their corresponding latent $z_1$ and $z_2$. I could simply define $z_1' = z_2$ by selecting $\delta = z_2 - z_1$. In the case, I would expect that the bound is violated. I believe that there should either be an assumption on $\delta$, or it should feature in the resulting bound.

- In the caption for Table 8, there is a typo: `copmaring`.

- The authors should incorporate all the additional results they reported during the rebuttal into the paper (or appendix).


#### Post-discussion Update ####

Over the course of the discussion between the authors and the reviewers, it has become apparent that several of the experimental settings chosen in the paper are non-standard, which make it hard to verify the validity of the results. I do share the concerns of my fellow reviewers in this regard, and thus will downgrade my score to a 5.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work introduces a novel adversarial augmentation strategy. This augmentation, nammed Identity-Disentangled Adversarial Augmentation (IDAA) aims at producing better hard positives/negatives while preserving the identity of the sample. This is achieved by perturbing the latent space of a variational auto-encoder while preserving the identity relevant features through the residual term. This allows for the training of better and more sample efficient self/semi-supervised.
main_review: Strengths:

- The idea is novel;
- The main hypothesis that other augmentation strategies result in pair of samples that are either too easy or run the risk of corrupting the instance-level information is backed up by empirical evidence;
- This work provides a information theory based interpretation of the IDAA strategy with sufficient proofs;
- The claim that better hard positives/negatives result in more sample-efficient models w.r.t the batch size and the number of epochs is highlighted through various ablations;
- IDAA provides significant performance improvements on all reported benchmarks;

Weaknesses:

- Skipping the conclusion is not an acceptable practice to gain more space. One could move some of the training hyperparameters to the appendix to gain some extra space in the main paper for that;
- The ablation on the batch size raises the concern that IDAA does not scale up too while. It is unclear whether SimCLR will surpass SimCLR+IDAA with a sufficiently large minibatch. Indeed, we observe that increasing the batch size is beneficial for SimCLR only on Figure 6-a;
- The datasets explored in the experiments of this work are somewhat limited in terms of diversity. Training a contrastive self-supervised model on the full imagenet dataset can be resource intensive, however the authors could have explored the use of lower resolution versions of it while keeping all the classes (e.g. ImageNet 64x64);

Minor improvements:
- Typo in the introduction: 'ullustrated in Fig. 2'
summary_of_the_review: This work introduces a novelty augmentation strategy with sufficient empirical and theoretical arguments. Although some concerns remain about its scalability to larger batch sizes and more challenging datasets, the authors clearly show significant performance improvements when using IDAA across all experimental settings and empirically justify all claims/hypotheses made.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies contrastive self-supervised learning. In particular, it proposes an adversarial augmentation method for different view generation in contrastive learning. It claims that using such adversarial mechanism, the generated views are identity-preserving while being hard positives/negatives. Empirical experiments on several benckmarks validate the improved performance and efficiency.
main_review: ## Strengths
+ The idea to decouple the identity from view generation seems to be novel and not explored in the CL literature before.
+ The presented method is simple and straightforward. I can see the potential of extending such formulation to a broader range of tasks and data domains.
+ The evaluation is relatively thorough, spanning across self-supervised & semi-supervised settings, and demonstrates the improvement over the vanilla and the CLAE method.

## Main Weaknesses
1. __Lack of comparisons to methods within the same category__ 
- The comparison is rather limited. The authors only compare to the vanilla (no adversarial augmentation introduced) and one method based on adversarial augmentation (CLAE). However, the basic idea is still _how to generate effective views / hard negative or positive samples_. Such problems have been extensively studied in the literature for CL [1-5]. How does IDAA perform against these methods? If not thoroughly evaluated, the performance gains are not justified and hence not interpretable.

2. __The definition of "identity-preserving"__ 
- In Fig.1, how is the distance measured? Is it measured in the feature space? How is the "Identity preserving boundary" defined? For CLAE, if the noise added to the input is bounded by a l_inf purturbation, then by definition the input should preserve the identity of the image. The information that this figure delivers is unclear.
- Continue on this point, to me it is very confusing on how you define identity-preserving. In fact, from Figure 4 in the paper, all augmentation methods actually preserve the identity. This also holds for CLAE. The only difference is that the perturbation generated by IDAA is more interpretable, but this has nothing to do with "identity-preserving". The augmented images still preserve their identities.
- Further, the motivation is not well explained. Given that the visualization does not directly reflect the point of "identity-preserving", the claims that current methods/augmentations "could change sample-identity and result in poor representations" is not justified.

3. __The IDAA is not self-contained as universal augmentation, but still needs a base set of augmentations__
- This might be a smaller issue, but IDAA still requires base augmentations (i.e., color jittering or random cropping/resizing) to work. In the experiments, all results of IDAA are based on some exsiting augmentations (e.g., those from SimCLR). The description of IDAA tends to sell it as a "universal" augmentation methods regradless of inputs, but domain knowledge (e.g., how you augment image data) is still needed from experiements. What is the actual performance of IDAA without using existing augmentations (those in SimCLR)?

## Other issues / questions
- Since the overall objective (i.e., generating better views for contrastive learning) is agnostic to the data modality, is the proposed scheme also be extended to other data domains beyond images, like time series / texts? What would be the most challenging parts when extending to other modalities?
- What is the actual computational cost (training time) compared to the vanilla SimCLR?

## References
[1] Hard Negative Mixing for Contrastive Learning.

[2] Adco: Adversarial contrast for efficient learning of unsupervised representations from self-trained negative adversaries.

[3] What makes for good views for contrastive learning.

[4] Debiased contrastive learning.

[5] Contrastive learning with hard negative samples.
summary_of_the_review: Overall, the idea is interesting, however the current draft is not well-justified on the actual algorithm design, the motivation, and (perhaps the most important) empirical comparisons to existing CL methods that considers hard view mining. Further comments/questions are listed in the weaknesses / questions part.

The paper has its potential to the field, but issues need to be addressed first. I'm happy to change my score if the feedback addresses my concerns. Please refer to the points in the weaknesses / questions part. I would like to see feedbacks on these comments/questions.
review_title: null

Meta-review: To improve the data augmentation for an existing self-supervised learning framework, the paper presents identity-disentangled adversarial augmentation (IDAA) that utilizes a pretrained VAE and adversarial perturbation in the VAE latent space to generate identity-preserving hard negative/positive samples. The proposed method has been clearly described and is of interest to the community. A wide variety of experiments have been done to illustrate the effectiveness of IDAA. Given the concerns on the validity and generalizability of the experimental results presented in the paper, the AC does not consider the paper of its current form to be ready for publication, as discussed below.

The AC appreciates the amount of effort that the authors have spent responding to the reviewers. However, thees very detailed rebuttals do not appear to be that effective in directly answering several key concerns of the reviewers.

1. While many experiments have been added to improve the paper, most of the settings are still considered by the reviewers to be unconvincing due to the use of a small number of epochs, unusual network backbones, and/or non-standard datasets (e.g., downsampled 64*64 ImageNet with about 30% accuracy).

2. The comparison to the literature is considered to be insufficient. While the basic idea is still how to generate effective views or hard negative/positive views, the paper chooses to narrowly focus its comparison with CLAE that relies on adversarial perturbation in the pixel space, ignoring a careful comparison with many existing methods that boost the performance by mining hard negatives/positives and/or using a memory bank to accommodate a large number of negatives. 

3. Related to point 2, a main justification of IDAA is that it outperforms CLAE, which, however, is shown by the authors to be a weak baseline that barely improves an existing self-supervised learning (SSL) framework in a variety of different settings. For example, the results in Table 1 show CLAE hurts SimCLR's performance. The weak performance of CLAE makes it even more important to include stronger baselines.

4. A reviewer also pointed out that the use of a pretrained VAE introduces additional model parameters and increases the computation cost, and hence a careful discussion on how to ensure a fair comparison with the baselines is desired. 

5. The presentation could be somewhat misleading in giving the impression that IDAA is a stand-alone domain-agnostic data augmentation technique. While it is totally fine that IDAA is a complementary data augmentation technique, the authors need to carefully revise the presentation to minimize the risk of giving the reader a wrong impression.  

6. The authors dismissed some of the negative reviews by arguing them to be “mostly incorrect, wrong, makes no sense”. These strong arguments, if not supported with clear evidence asked by the reviewers, may convince neither the reviewers nor the AC and lead to unnecessarily tedious lengthy discussions. From the AC's reading of the paper and reviews, it appears that there are several cases where a reviewer was asking the answer to Question A, but the authors were providing an answer to Question B and pushing the reviewer to accept that as a satisfactory answer to Question A. For example, while many experiments have been done on several different datasets, there is a legitimate concern on why these computing resource has not been spent on getting results on ImageNet with ResNet 50. When the experiments are only being done on ImageNet-100 or the downsampled full ImageNet with a small network, there could be clear concerns on whether the observed performance gains are only applicable to relatively small/low-resolution datasets or small networks. For example, the VAE and/or the adversarial attack may not work that well on large-scale/high-resolution images and hence IDAA may break down when scaling it to a larger dataset/model. That type of concern can only be addressed if the authors have taken the time to follow a standard setting, such as ResNet50 on ImageNet, and make comparisons with a wide variety of baselines whose results on these standard settings are readily available.
Decision: Reject