Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper first investigates the problem of dimensional confounder in contrastive learning, which refers to a subset of dimensions learning only task-irrelevant background information. The author shows the existence of the dimension confounder, where the performance of Barlow Twin starts to drop and eventually collapses after the dimension of the projection head reaches a certain point. The author then proposes to learn a dimensional mask approach, MetaMask, consisting of learnable weights to reweight each dimension of the encoder output in contrastive learning, so that the dimensional confounder can be assigned lower weights. The authors show theoretically that the proposed MetaMask achieves tighter risk bounds (lower and upper) for downstream classification tasks, and empirically improves the performances on downstream tasks and robustness towards dimension sizes by applying MetaMask to the prior models. 
strengths_and_weaknesses: Originality: 

Strengths: the idea is novel, addressing the issue of dimensional confounders by reweighting encoder outputs using meta-learning techniques.  The idea is related to MAXL[1], but the two approaches are different. MAXL applies a mask with weights on the softmax logits in a hierarchical binary prediction setup, where the logits are single scalars. On the other hand, this work applies a mask with weights on the encoder output for contrastive learning, where the output has a large dimension.

The reviewer is not familiar with meta-learning literature and only evaluates the originality from a contrastive learning perspective.

Quality: 

Strengths: the idea overall has technical soundness, with theoretical justification on the proposed MetaMask creating tighter lower and upper bounds of risk, and shows reducing the risk can improve downstream performances. 

Weaknesses: The reviewers have some additional clarification questions listed in the next section.

The reviewer cannot verify the correctness of Equation 8. 

Clarity: 

Strengths: the paper is very well written, with clear motivation, thorough related work, a succinct method section, and well-organized theoretical analysis and experimental results.

Significance: 

Strengths: the community can adopt the method to reduce the impact of the dimensional confounder, especially in large models. The performance improvements are significant in some cases (Table 2, SimCLR on STL-10 and NNCLR on CIFAR-10).

[1] Liu, Shikun, Andrew Davison, and Edward Johns. "Self-supervised generalisation with meta auxiliary learning." Advances in Neural Information Processing Systems 32 (2019).

questions: Q1: Figure 1(a) is interesting -- the proposed MetaMask seems to have much higher contributions on most dimensions than SimCLR -- why is this and is this desirable?

Q2: How the experiments for Figure 1(b) are a little unclear. How many layers were you using? Were you using the same dimension for all the layers of the projection head, or did you shrink the dimension at the last layer to serve as a dimension bottleneck? These are important because your results sharply contrast with Barlow Twins'. 

Q3: Could you provide the dimensional mask rate that improves the performance on both ImageNet and CIFAR-10? Although you claim the ImageNet rate is higher, the difference seems indistinguishable from Figure 2.
 
Q4: The claim you made, that (Line 50) "the dimensionality of the projection head ... acts as a dimensionality bottleneck" is not the same as the claim from Barlow Twin, "the output of the ResNet is kept fixed to 2048, which acts as a dimensionality bottleneck". The bottleneck, according to Barlow Twin, seems to be the ResNet output dimension, rather than the projection head dimension. Could you clarify this?
limitations: The authors discussed the limitations of the work. The limitation states that the proposed MetaMask is specific towards improving contrastive learning but could not yet show the theoretical guarantees of MetaMask on other self-supervised learning methods, such as masked image modeling. This is a correct assertion and the discussions of limitations are helpful for the community.

The authors have not shed light on the negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper handles the problems which existing self-supervised learning algorithms suffer. Authors demonstrate that existing self-supervised learning algorithms suffer dimensional redundancy and dimensional confounder. They propose MetaMask which learns a dimensional mask through meta learning to decrease the learning signals of features having confounders. For the rationales of MetaMask, they provide theoretical support where MetaMask has tighter risk bounds in downstream tasks compared to baselines. They demonstrate that MetaMask combined with competitive self-supervised algorithms has superior performance to baselines on multiple benchmark datasets under two architectures. 
strengths_and_weaknesses: The paper has several strong and weak points.

Strengths:
1. This paper shows that the existing self-supervised learning algorithms suffer dimensional redundancy and dimensional confounder. The problem is also quite intuitive.
2. Theoretical support for the reasons why MetaMask works is provided. (I have not carefully checked the proofs)
3. Authors conduct extensive experiments and demonstrate that MetaMask combined with existing methods improves performance in most cases.

Weaknesses:
1. MetaMask adopts bi-level optimization, so it would necessarily require significant amounts of additional computational cost during computing second-derivatives. However, authors do not provide any analysis of computational costs.
2. Authors maintain that MetaMask reduces the gradient effect of the dimension containing the confounder (in lines 76-80), but they do not explicitly demonstrate that masked dimensions (having low weights in MetaMask) have confounder (non-discriminative features such as backgrounds). It would be great if the authors suggest experiments about it.
3. The improvement of MetaMask seems marginal in important cases although MetaMask improves baseline performance by combining with them in most cases. Specifically, in Table 2, MetaMask combined with existing algorithms in the modern architecture (ResNet18) shows comparable performance to the most competitive baselines (such as BYOL in CIFAR-10, NNCLR in CIFAR-100, and NNCLR in IN-200).
4. (simple baseline) Applying dropout on $h_i$ in Barlow Twins might be the simple baseline for MetaMask and it would be effective in that BarlowTwins with randomly masked dimensions outperform naive BarlowTwins in Figure 2. Would authors provide the performance of Barlow Twins + dropout?

Miscellaneous minor issues:
1. In Figure 1, (a) and (b) are missing in the figure (only mentioned in the caption). 

---

[After reading authors' answers] I appreciate the detailed responses and authors address several of my concerns. Accordingly, I change my rating to weak accept.
questions: 1. (ablation study) Barlow Twins handles dimensional redundancy but suffers dimensional confounder. MetaMask mitigates dimensional confounder by learning and applying a dimensional mask. There would be performance gain only from alleviating dimensional confounder. Can authors show the performance of MetaMask without redundancy-reduction objective function? (I understand that Barlow Twins could exacerbate dimensional confounder.)
limitations: It would be great if the authors suggest possible issues from the adoption of bi-level optimization such as computational costs.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a self-supervised learning scheme with a mask on top of the contrastive learning and the Barlow-twin method to perform dimensional reduction and confounder elimination. The concept is to use a meta-learning approach to seek a mask deleting redundant features while optimizing the feature representation and model parameters on the fly. The training loss is based on a combination of contrastive and Barlow-twin.
strengths_and_weaknesses: Strengths:

The proposed method seems easy to implement based on existing works and achieve good performance improvement. The method has been validated with theory.

Weakness:

The ablation study is insufficient, as the method uses both Barlow-twin and contrastive loss. Still, it's unclear that this meta-learning alike optimization scheme is essential compared with training on a loss combination of Barlow-twin and contrastive loss. The comparisons have only been made against either one of them. Also, in Figure 1, the performance improvement by the random mask is not significant enough to persuade me that the feature redundancy is sufficiently essential (only within 0.1% improvement on ImageNet). Finally, why is the performance not tested on full ImageNet while the motivation figure is?

Theoretically, the authors show that the conditional variance can be reduced with the optimal mask. Still, since the identity matrix (no mask) is also a special case of masks, this is a seemingly trivial statement.

questions: I think the improvement is good, but I want to know that this meta-learning type is more essential than just optimizing a loss of Barlow-twin and contrastive loss. Also, the theoretical point that mask includes identity seemingly makes the theorem vacuous, and I wonder if the authors can respond to my concern.
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposed a hybrid joint-embedding framework, which adds learnable masks on representations and applies both contrastive loss and redundancy reduction loss. This mask is updated when fixing the rest of the network. 
The experiments show that the proposed method improves most of the recent frameworks (BarlowTwins, SimCLR, SwAV, BYOL, etc.) on various datasets (CIFAR-10, CIFAR-100,  STL-10, IN-200)
strengths_and_weaknesses: Strengths
1. The authors point out that joint-embedding methods will learn harmful features. This is a widely ignored problem in the self-supervised learning method.
2. Experiment results show consistent improvement of the proposed method overall frameworks on various datasets. 
3. The second-order optimization on the mask is supported by theoretical proof. 


Weakness
1. The masks are trainable means that in the end, they are fixed and will ignore several features known as confounders. However, this needs to be verified via ablation experiments. For example, a linear probe on the whole unmasked features should give lower accuracy.  
2. The experiments are not convincing due to a lack of fair comparison. The authors use a very customized setting, e.g., AlexNet on various joint-embedding frameworks. No hyperparameter tuning is conducted on these models. 
3. Second-order optimization creates significant computational overhead. The authors need to show how much more training time is needed for each model.
4. No experiment details. 


====== post rebuttal comments =====

I've carefully read the authors' responses. All my major concerns are addressed. Though the AlexNet setting on 1 GPU is not convincing enough, the fair amount of controlled experiments in the paper convinces me of the effectiveness of the proposed idea.

I've increased my score.

questions: 1. What's the reason behind MetaMask's collapse for large embedding dimensions? This is not observed in standard BarlowTwins.

2. Why is this considered "meta"? It's basically second-order optimization.

limitations: N/A
review_title: null

Meta-review: metareview: This paper starts with the experimentally findings that the interference of task irrelevant information and the disadvantages of sample inefficiency in contrastive learning appear due to dimensional redundancy and dimensional confounder. Based on these experimental findings, the authors propose a dimensional mask learning method based on bi-level optimization. Finally, the theoretical basis for learning through meta mask is also presented. All reviewers agreed with the strength of the proposed method. There were some concerns such as insufficient ablation study, but they were all resolved through the authors' rebuttal and subsequent discussion. I hope the authors make it sure that the concerns raised in the review process can be resolved in the final version.
confidence: Certain
Decision: Accept