Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposed a blockwise learning strategy based on the Barlow Twins framework as an alternative to replacing standard backpropagation that is commonly used in existing SSL algorithms. Compared to related works, their work addresses the challenge of large-scale datasets. In addition, they compared several spatial and feature pooling strategies for building intermediate features and evaluated a number of strategies to customize the training procedure for blockwise training. Through their numerical studies, they demonstrate that blockwise training can achieve competitive performance compared to the original framework.
strength_and_weaknesses: Strength 
- This paper proposed an alternative approach to training SSL models that can be used for training large neural networks with a limited memory budget.
- They evaluated a variety of strategies to customize the training procedure at the different blocks and the proposed strategies are also transferable to other SSL frameworks.
- They conducted various ablation studies to investigate individual component of their framework, such as comparing different feature pooling strategies for the intermediate block output.

Weaknesses
- The objective formulation for blockwise training is somewhat unclear to me. How do the authors define the loss in sequential or simultaneous blockwise training?  Similarly, what is the formulation for other frameworks, such as SimCLR?
- It lacks experiment details. For example, what training parameters, such as epoch and batch size, are associated with the results in section 4? How do authors conduct supervised tasks, e.g., loss and model change?
- It would be interesting to extend the current results to different model architectures. e.g., different CNN/transformer-based models, and include more results on different SSL frameworks, such as SimCLR, and MoCo.
- The authors mentioned one of the main advantages of this learning strategy is its ability to handle larger neural networks, it would be interesting to include more comparisons by varying the model size to support this claim. 


clarity,_quality,_novelty_and_reproducibility: This paper discusses an alternative learning strategy for SSL, which provides some insights for future research in SSL. However, the quality of the paper should be improved for publication consideration and more systemic comparison should be conducted to generalize the current conclude.  The proposed approaches seem to be reproducible based on existing publicly available codebases.

summary_of_the_review: Please see my comments in Strength And Weaknesses.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes multiple tricks to push the performance and of the self-supervised Barlow Twins step in a block-wise setting. Most prominently, it is reported than a block-wise training of a ResNet architecture with Barlow twins leads to a high top-1 accuracy of a down-stream classifier (70+% which is one percent away from end-to-end training). To reach this performance a new method to extract a low-dimensional vector from the ResNet feature map and a noise injection strategy are described.
strength_and_weaknesses: The biggest result of this paper is certainly to demonstrate how competitive can be the Barlow Twins in this block-wise setting. Although this had been demonstrated for the CPC loss by Lowe et al, this result had not been reproduced with other losses on a dataset as hard as ImageNet as far as I know. This alone is likely to be useful intermediate step torwards back-prop-free self-supervised learning method which perform well on ImageNet.

Although it certainly took a lot of work to make it work, I do not see big conceptual innovation but this is not always necessary to make a good paper. Let's recap why I think some results were already published by other authors:
- Barlow twins was already shown to work very well on ImageNet in the original paper,
- Lowe et al. 2019 demonstrated than block-wise CPC worked well,
- Lowe et al. 2019 inspected some training variants including simultaneously or sequential trainings of the blocks,
- The noise addition is likely to have an effect similar to drop out. It is unfortunate that we do not have a comparison between the suggested noise model against regular dropout.

The combination of feature expansion and pooling is probably the most creative innovation here. It is indeed expected that some kind of reshaping and post-processing is necessary to transform the feature map into a low-dimensional latent variable that is relevant for the loss function. Inspecting which variant is best is certainly useful for future work.

I find it very curious that the network with lower top1 accuracy at early layers perform well for later layers. This curiosity is also put forward in the paper as an interesting side result. This might indeed hide some important effect which explain the performance difference. A deeper analysis of this might make the paper stronger, why does it happen with the block-wise barlow twins with the specific pooling strategy? Is that an effect of the pooling loss only or the loss function too?
clarity,_quality,_novelty_and_reproducibility: The paper is clear, as argued before, the conceptual novelty is limited but this is counter balanced by the good performance. I hope that the code will be published to lead the field forward towards back-prop-free self-supervised learning working on ImageNet.
summary_of_the_review: Most of the results are very clearly written. It is very valuable to see some tricks which make Barlow twins competitive in this setting. This paper would be probably more impactful or important if the experiments were targeted to support one strong result with a central story line rather than a list of somewhat unrelated tricks.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The manuscript proposes a strategy to train the network in a blockwise manner. Specifically, this strategy explores two local learning methods for blockwise BP training based on the basic blocks in the ResNet-50 model, i.e. simultaneous blockwise training and sequential blockwise training. In addition, the paper also explores some design details, such as pooling methods, parameter settings, and augmentation methods. At last, the manuscript presents experiments and ablations of the proposed strategies.
strength_and_weaknesses: Strength
The paper explores two strategies for blockwise training based on the Barlow Twins method. The authors perform more detailed experiments and ablation of the proposed methods. Overall, the paper describes the proposed method in detail and validates the method with experiments, and the paper is clearly written and well organized.
Weaknesses
Motivated by the fact that local learning can limit memory when training the network and the adaptive nature of each individual block, the paper extends local learning to the ResNet-50 to handle large datasets. However, it seems that the results of the paper do not demonstrate the benefits of doing so. The detailed weaknesses are as follows:
1)The method proposed in the paper essentially differs very little from the traditional BP method. The main contribution of the paper is adding the stop gradient operation between blocks, which appears to be less innovative.
2)The local learning strategy is not superior to the BP optimization method. In addition, the model is more sensitive to each block after the model is blocked, especially the first block. More additional corrections are needed to improve the performance and robustness of the model, although still lower than BP's method.
3)Experimental results show that simultaneous blockwise training is better than sequential blockwise training. But the simultaneous blockwise training strategy cannot limit memory. 
4)The blockwise training strategy relies on a special network structure like the block structure of the ResNet-50 model.
5)There are some writing errors in the paper, such as "informative informative" on page 5 and "performance" on page 1, which lacks a title.
clarity,_quality,_novelty_and_reproducibility: Clarity and Quality: The structure of the paper is clear. The paper provides a detailed description of the proposed method. The proposed method is compared with state-of-the-art self-supervised methods and a detailed ablation analysis is provided. However, there are some writing errors in the text that need to be further checked by the author.
Novelty: The blockwise training strategy proposed in the paper lacks innovation. Specifically, ResNet networks with block structure are trained in a blockwise way, i.e., using the BP optimization method within blocks and the stop-gradient operation between blocks.
Reproducibility: Some key settings descriptions are missing in the paper. It is not clear whether the augmentation method for the dataset in the blockwise training strategy and the parameter setting, , in the loss objective are the same as in Barlow Twins.
summary_of_the_review: The blockwise training strategy that uses a stop-gradient operation between blocks is not novel. This approach reduces the performance of the Barlow Twins method and the robustness of the model. It also does not achieve the purpose of limiting the memory and improving the adaptive capability of the model.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposed to extend the existing self-supervised learning methods for blockwise training scheme based on the Barlow Twins self-supervised learning framework. Reviewers raised some concerns about the limited novelty and weak empirical studies (e.g., overall improvements do not seem very convincing, only evaluated on a single SSL method and a single DNN backbone, lacking evaluations on state-of-the-art baselines and larger models/dataests, insufficient ablation studies, etc), etc. The authors have partially addressed some of the questions during the rebuttal but did not provide sufficient results and answers to convince the reviewers. Overall, the paper contains some interesting idea that may be worth publication, but the quality of the current manuscript is not strong enough for publication.
Decision: Reject