Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the author proposed a supervised contrastive learning (SCL)-based training method for deep neural networks in which the gradients only flow locally in each layer, solving some of the problems associated with long gradient flows in backpropagation. The authors also benchmarked its performance against early exit, associated learning (AL) and backpropagation (BP) for training some standard feedforward network architectures for classification tasks. The results showed that the proposed method outperforms early exit, AL on most tasks while BP is still superior for ResNet. 

In addition, the author also did some analysis on the effects of batch size on the final test performance and found larger batch size generally works better. The author also found that a nonlinear projection head performs better than identity or linear heads. 

strength_and_weaknesses: This work has several strengths:

1. The proposed method is able to cut the long gradient flow in training deep neural networks.

2. The proposed method decouples the parameter updating across layers, making parallel training possible, though it needs follow-up work to validate this claim.

3. The proposed method requires fewer parameter than similar method such as AL in training phase, and has the same effective parameter in inference phase. 

4. The test accuracy on simpler tasks and simpler architectures are better than AL and comparable to BP.

Weaknesses

1. One important aspect of learning procedures are the learning dynamics, which is not thoroughly studied. What are the learning curves like for the methods compared? How much longer or shorter does the proposed method converge compared to AL and BP? Is the training stable? Do we need to use different learning steps for each layer? How does the parameters change during training and use this to validate whether it actually addresses some problems with long gradient flows, such as the exploding/vanishing gradient problem for long gradient flows?

The research community would also benefit from some potential follow-up work such as the pipelining implementation and its possible extension to training other architectures such as RNNs. 

clarity,_quality,_novelty_and_reproducibility: In general, the paper is well written. The motivations are clearly explained. The experimental design is also sound. One novelty of this paper is that it brings supervised contrastive learning to solve the problem of long gradient flow, complementing some existing approaches such work such as target propagation and AL. The source code is not provided for reproducibility.

Some minor issues and typos:

1. Figure 1 notations are a little bit confusing. 

2. Page 4, line 7 after equation (1): $]r_1^{(i)}$.

summary_of_the_review: The authors proposed a supervised contrastive learning inspired approach to train deep neural networks without long gradient flows. The proposed method has several characteristics such as 1) it only requires local gradient computation and parameter updates 2) it enables parallel training via pipelining 3) it requires fewer additional parameters than similar methods in training phase and same number of parameters as BP in inference phase 4) the test accuracy are comparable to BP. The authors also studied the effects of batch size, non-linear projection head. However, a thorough study and comparison of the learning dynamics are missing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors present a novel method that decouples back propagation gradient computation in artificial neural networks. The authors identify issues that result from long gradient flows in back propagation, and propose a method that tackles some issues, which include vanishing/exploding gradients, and update locking. In short, the proposed method cuts long gradient flows into several local gradients with a ‘read-out’ at each layer and a corresponding contrastive loss.
strength_and_weaknesses: Strengths
-----------

Delog-SCL provides competitive results against Backpropagation and is parameter efficient in comparison to related work (AL). Naturally, as a consequence of gradient decoupling, Delog-SCL circumvents the gradient update locking problem found in Backpropagation.

The algorithm is promising, lots of insights can be teased from this method and it is a fair addition to the literature on decoupled gradient optimization in artificial neural networks.

Weaknesses
----------------
There is a lack of clarity on the novel contributions of the proposed method, Delog-SCL. There is no clear consensus on the prevailing issue that Delog-SCL aims to tackle, a host of issues have been presented, some of which some are unrelated, in particular the Vanishing/Exploding gradient problem. The issue of vanishing and exploding gradients is prematurely explored, for example there is little mention of Delog-SCL being extended to recurrent nets.

Some of the issues highlighted are naturally tackled by contemporary backprop decoupling methods, there was some justification that Delog-SCL is more flexible and simpler than AL, but this was not raised in light of a critical issue (beyond efficient parallelization).

Similarly, the claim that Delog-SCL tackles the issue of unstable gradients in the early layers was mentioned in the introduction and conclusion with little to no explanation on the reasons behind the instability and how this relates to Delog-SCL. This was not explored in the experiments nor methodology section.
clarity,_quality,_novelty_and_reproducibility: Clarity
------------

The paper lacks clarity on its contribution. This makes it slightly tricky to follow the manuscript.

Quality
-------------
I find the quality of the paper to be good but with lots of room for improvement.

Novelty
------------
I find the work to be novel.

Reproducibility
-------------------
It appears that there has been no code that has been released yet but practically the method is simple and can probably be reproduced.
summary_of_the_review: The work is an interesting alternative to the backpropagation algorithm as well as other existing decoupled gradient computation algorithms. However, the biggest shortfall of this manuscript is the lack of clarity on its contributions. The manuscript lacks a problem statement.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: UPDATE: FOLLOWING THE AUTHOR'S RESPONSE, I CHANGE MY SCORE FROM 5 to 6. 

The authors describe a method to reduce the problems associated with gradients in settings where backpropagation happens through many layers. In the context of supervised contrastive learning, the authors propose to compute contrastive losses not only at the output layer, but also at intermediate layers. This allows the authors to segment the neural network into blocks that each have their own losses, and can therefore be trained nearly independently (of course, layers closer to the output still must adapt to changes in the closer-to-input layers during training). The authors compare their method with the Associated Learning method, and find that their method performs about as well, but is easier to work with. The authors also compare with vanilla back propagation, and find that their method (as well as the associated learning baseline) outperform back propagation on a number of tasks and network layer types (but not on residual networks). 
strength_and_weaknesses: # Strengths:

- The paper addresses a question that remains relevant: how do we best minimize losses in deep neural networks? How do we prevent gradients from vanishing? 

- The paper is very clear and well-written, and easy to read. 

# Weaknesses:

- The authors did not include the code with their submission, and mentioned no intent to do so. There is also no appendix with experimental details such as layer sizes and learning rates. Reproducing the experiments would be practically impossible. 

- Besides the methods described in the submission, skip connections (e.g. in highway networks https://arxiv.org/pdf/1505.00387.pdf or residual networks) are an effective tool to reduce vanishing gradient problems. The authors should highlight this in their literature review, and ideally also perform experiments to check if adding skip connections would remove the need for the method described in this paper. It is interesting that the authors find that their method (as well as the AL benchmark method) do not outperform backpropagation when ResNets are used. This suggests that skip connections may remove the need for methods such as the one proposed in this paper, as well as the AL baseline. It would be an interesting experiment to check how e.g. VGG with added skip connections performs with and without the proposed Delog-SCL method. Skip connections have also been used in LSTMs, e.g. in https://aclanthology.org/C16-1020.pdf and https://ojs.aaai.org/index.php/AAAI/article/view/4613/4491 . 

- The authors write “We omitted Early Exit because this method performs much worse on more complicated datasets”. In my opinion, it would be better to include the Early Exist performance numbers, and not need the explanation of why Early Exit was not included. 


clarity,_quality,_novelty_and_reproducibility: - The main body of the paper is very clear, and of high quality. 

- The paper is not reproducible, as no code was included, and hyper parameters such as learning rates are not included in the paper or abstract. I recommend that the authors release the code, and additionally describe experiments in detail in an appendix. 
summary_of_the_review: UPDATE: FOLLOWING THE AUTHOR'S RESPONSE, I CHANGE MY SCORE FROM 5 to 6. 

For now, I mark this submission "5: marginally below the acceptance threshold". 

The main idea presented in the paper is interesting and useful. However, before the paper can be accepted, the authors should improve the reproducibility of the paper by releasing code and describing experiments in more detail in an appendix. The authors should also investigate whether skip connections could achieve the same that their method does.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: As neural networks get deeper, end-to-end backpropagation requires computing the gradients along longer paths, thus making gradient computations more prone to numerical instabilities and vanishing/exploding gradients. This paper proposes a new approach which divides the network into different submodules that are independently trained. Specifically, inner modules are trained using supervised contrastive learning, while the last layer is trained as usual. This enables the computation of short gradients, as well as the asynchronous training of modules, which beats the current state-of-the-art method and obtains comparatively results with end-to-end backpropagation.
strength_and_weaknesses: **Strengths:**
- S1. The approach is clear and simple, and provides results at times better than back-propagation, which I find remarkable.
- S2. The proposed method is a clear improvement over the competing method, beating it in every aspect as far as I can tell.
- S3. The ablation study is nice and agrees with existing knowledge in contrastive learning.

**Weaknesses:**
- W1. The motivation is rather weak. While the asynchronous training is interesting, having long gradients does not seem as much of a problem, as it can be overcome with architectural changes (which is clearly shown by the performance difference of the existing approaches in the ResNet experiments).
- W2. Despite contrastive learning being quite intense in computations, there are no mentions to memory and time consumption.
- W3. Needed work is left as future work. Specifically, pipelining as well as understanding where comes the improvement in performance come from.
- W4. Despite talking all the paper about gradients, there is not a single experiment exploring the gradients and showing that there was actually a problem in the chosen experimental setups.

**Comments:**
- C1. While parallelism is trivialized in the proposed approach, pipelining is also possible in traditional methods, as demonstrated by companies such as [OpenAI](https://openai.com/blog/techniques-for-training-large-neural-networks/).
- C2. Doesn't the fact that Early Exit perform well in CIFAR-100 show that it is not an appropriate experimental setup?
clarity,_quality,_novelty_and_reproducibility: **Clarity and quality.** While the methodology is clear, the writing would benefit from a bit of polishing. Parts like the abstract, or short phrases extremely similar to the table captions (e.g. Table 5 and below) could be improved.

**Novelty.** Here lies my biggest concern: the proposed method is extremely similar (to my eyes) to Greedy InfoMax. If I understood correctly, the only difference between both approaches is that the proposed method uses _supervised_ contrastive learning, while Greedy InfoMax uses _unsupervised_ contrastive learning. The authors should better clarify what makes this work novel.

**Reproducibility.** There is no code provided until paper acceptance, neither there is an appendix. Therefore, the paper lacks all the experiemtal details to perfectly reproduce the reported results.
summary_of_the_review: While the proposed approach is novel in the supervised setting, I have serious concerns of its novelty with respect to Greedy InfoMax. While this would not directly imply a rejection, the deferred extra work and the lack of proper experiments to fully understand the benefits and hurdles of the proposed approach in the specific setting it is used for, makes me lean towards rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes an algorithm that breaks long gradient flows in end-to-end backpropagation to multiple short ones, as the objective is the inefficiencies of regular backpropagation when going with deeper networks. The main idea of this algorithm is to use of supervised contrastive learning, where examples of the same category should be close in the embedding space. Specifically, in the backward path when computing gradients, the flow is stopped per components in the network, and a 'global' objective is replaced my multiple local ones. Effectively the proposed algorithm considers deep networks as cascaded shallow ones, where each intermediate layer is not 'hidden' per se; rather, it considers the input from the previous layer as an observation variable (so, not a learned variable), and returns an output embedding that can be evaluated with a supervised contrastive loss. A problem that this approach can help with is vanishing gradients, which makes sense since intermediate loss functions are introduced into the neural network. This is not surprising, as this was also used with Inception networks, of course with regular backpropagation so that to be able to train deeper networks. For what is worth, the abstract is not very clear on what is the algorithm supposed to do, and only explains that a 'novel methodology' is proposed.

This was a paper with both positive and negative ratings. All reviewers agreed that the importance of the theoretical contributions of the paper is not clear, and I agree with that statement. The difference with Greedy Infomax is not persuasive enough. The authors argue that augmentations are only possible with visual tasks, however, this argument is not entirely valid (eg a simple Google Search for 'Molecular Contrastive Learning ...' returns quite a few results). What is more, replacing the requirement for augmentations with an arguably an even more expensive one (labelled samples), is not that convincing.

Besides, introducing loss layers in every layer in between is not novel itself and it was introduced for a similar observation of vanishing gradients (see Inception family of architectures). It was not considered as an option of non-end-to-end training, however, I think this is a rather minor difference. In other words, I do not this it is fair to say that said algorithm breaks the long gradient flow to smaller ones, because the gradient is not the same (and not supposed to be the same), as the learning objective changes massively. And if the intent was the gradients to be similar enough, then likely there is substantial bias added to the gradient estimation, and an ablation and theoretical study would be warranted.
justification_for_why_not_higher_score: The novelty and contributions is not clear, and at the same time Delog-SCL does not consistently outperform AL across all models and datasets. There are some elements in the ablation that are substantial, e.g. Vanilla ConvNet on Cifar100. Upon further inspection, one finds that the "Vanilla ConvNet" is a 3-layer Conv Net that I assume has been hand-engineered by the authors to outperform on the task. All in all, either the theoretical contributions must be further boosted, or the empirical findings become more generalizable across settings/datasets and detailed.
justification_for_why_not_lower_score: The paper is generally on a good track, I expect with further imrpovements it can be accepted in a good conference.
Decision: Reject