Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: Briefing:
This paper proposes a new infoPro loss for locally supervised training that alleviates the problem from greedy supervised learning, which collapsing task-relevant Information at the beginning of the layers.
The infoPro loss requires an auxiliary network to infer the Information I(h,y) (For ImageNet, the paper used ASPP).

Strong points:

Analysis of the phenomenon when applying the greedy supervised learning (GSL): 
The two observations from the authors seem natural.

(1): GSL underperforms E2E training (little trivial for the reviewer)

(2): Each separated layer trained by GSL captures more discriminative features.

However, the information-based explanation of why the task-relevant info at the early stage of the network is essential (information collapse hypothesis) seems new and worth considering for the reviewer. 

InfoPro-loss:
Theoretical analysis of the loss seems credible to the reviewer.

Experiments:
Mutual Information at each layer index looked interesting.

Interestingly, the proposed method outperforms E2E training in the ImageNet classification task.

Hyper-parameter does not seem to be sensitive.

Week point:

Using the additional network might not be fair when strictly comparing the performance. But practically, when training the large network, the size of the auxiliary network seems ok.

Comments:

(1) It is not strongly required. Is it available to apply the method to the detection task?

(2) Adding the experiments for other widely used network s.a. VGG, would be meaningful.

(3) Can we use LR-ASPP in MobileNet v3 instead of ASPP? or Can we use a heavier Decoder for better performance?

(4) How come when we set K=4 and set a larger batch size? We do not need to waste the remaining memory.

(5) Training the network larger than ResNet-152 from the proposed method would be impressive.

(6) Can we replace the imageNet pre-trained (by E2E) backbone with that trained by the proposed method? Experiments or discussion verifying the transferability of the trained network would be required.

 
Rating: Nice paper, Accept 

(1) But the reviewer wants a discussion with the author about the comments mentioned above

(2) Literature checks from the other reviewers may affect the rating.


review_title: Reviewer1 Comments

final_score: rating: 7: Good paper, accept
review_content: # Summary

The paper analyzes the pitfalls of locally supervised learning from the point of view of information propagation and proposes a new auxiliary loss that can facilitate locally supervised learning. The proposed loss, "infopro loss", is then relaxed to a tractable upper bound, which is then used instead. To implement the loss, mutual information is approximated with a decoder, as well as a classifier. The authors further introduce now contrastive learning fits in the framework as a lower bound maximization process regarding mutual information. The experimental results on standard datasets demonstrate the efficacy of the proposed method.

I have enjoyed reading the paper quite a lot, and therefore recommend accepting the paper. Still, I have some reservations that I would love the authors to clarify via the rebuttal.

# Strength

The paper is well written. There are some issues (which I detail in the weaknesses section), but most are very clear and easy to follow. There are some parts that are repetitive, but it does allow readers to scheme through without missing the important point.

The part that I like most about the paper is how proposition 1 and appendix D is presented. They are theoretically well-motivated and gladly seems to work, despite the relaxation. I personally think appendix D deserves more attention in the main text, but this is my personal preference.

The results clearly show that the method improves over simple greedy locally supervised learning, as well as other attempts at this problem. 

# Weaknesses

## Section 3.1, regarding equation (1)

I am not 100% sure on this, but shouldn't the third last sentence of Section 3.1 read "...under the goal of retaining as much information of the input as possible"? I am not sure this is actually a constraint, and the I(h,x) term is not explicitly task-relevant information.

## Section 3.3, estimating I(h,y)

It is not clear to me how I(h,y) is finally approximated. Shouldn't p(y|h) disappear after approximation? In the provided approximation it still exists. If p(y|h) is somehow directly used, isn't q not needed at all?

## Section 3.3 final equation

In my opinion, even when it is not referred to in text, equations should have numbers so that future readers can refer to it.

## Regarding Asy-InfoPro

It is somewhat unclear whether the dynamic caching was used at the end. Are the experiments in Table 2 with dynamic caching? From my current understanding, it does not seem to be the case, which leads to my second issue.

This is assuming that the results are without dynamic caching as this seems most logical. The explanation in the "Asynchronous and parallel training" paragraph was not obvious to me during the first read. The second sentence could be rephrased and split so that it becomes clear that the distinction between the two modes is that transient feature maps are seen/unseen and that this has a regularizing effect.  This then brings up the important question, whether the dynamic caching version suffers from the same fate---it should not. Having this experimental verification would greatly strengthen the observations in this paper.

## Softmax vs Contrastive

I am curious as to why contrastive works better. Could it be a coincidence of better hyperparameter tuning? Because the softmax version is a direct estimate on mutual information, whereas the contrastive one optimizes the lower bound.  It would be nice if this was discussed in more detail (I do understand that there is already very little space though!)

## Computational overhead

Is the computational overhead including the memory transfer cost? Are the results the actual physical measures observed via monitoring the resource usage? Or are they theoretical? I might have missed it, but this is not very clear to me.

## Typos and grammar errors

There are quite a few grammatical mistakes throughout the paper. For example, at the beginning of Section 2, it is more natural to write "we start by considering" than "we start with considering". In the italic question in the second paragraph of Section 2, "..., even the former..." should be "..., even though the former...". While I generally did not find these errors to be critical, I would suggest a thorough proofread.

I also found a typo in Appendix B. The last sentence should refer to equation (10) not (9).

## Early stopping, and the choice of the number of epochs

The training process in this paper does not utilize early stopping. While this is somewhat mitigated by the fact that multiple runs are performed, this is in fact another source of overfitting to the dataset and is strictly speaking tuning hyperparameters on the test set. This is a practice that should be avoided.
review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a strategy for training feed-forward networks in a more memory-efficient manner by employing local as opposed to end-to-end supervision. End-to-end/global (E2E) supervision as the dominant paradigm in training deep networks considers a loss function at the very end of the network for backpropagation of the resulting gradients, whereas local supervision injects supervisory signals (such as the same E2E objective, e.g. classification) at intermediate layers in the network. The benefit of such intermediate supervision is the ability to train larger networks in smaller chunks piece by piece, where each individual training is more memory efficient due to reduced need to store activations (and weights and biases) in GPU memory. As a drawback, however, it had been shown earlier that such local training is less optimal than global training in terms of the achievable generalization performance. The authors propose a new training strategy that aims at combining the memory efficiency of local supervision and piecewise training with the error performance of global training. Considering a given intermediate layer, the paper motivates to maximize the mutual information between the activations in this layer and the input signal to retain relevant information, while minimizing the mutual information of the activations and a nuisance variable, where the nuisance is defined as having no mutual information with the target variable (e.g. the classification prediction). The authors argue that this local supervision allows to train the features at the intermediate layer such that they carry relevant information from the input to the target variable without resorting to direct supervision with the target variable. Direct computation of the nuisance variable is infeasible and the authors propose a bounded approximation. 
Empirical results are discussed on five common vision datasets and two CNN architectures in relation to the existing state of the art.

### Strengths
**[S1]** The paper is largely written well and addresses a relevant problem. Contributions and claims are laid out well.

**[S2]** The method has potentially multiple benefits: (a) more memory efficient training without loss of performance, (b) speedups for asynchronous training, (c) use of larger batch size or larger models.

**[S3]** The method is motivated reasonably well by the argument of minimizing the loss of mutual information with the input variable while minimizing the mutual information of nuisance variables with respect to the target quantity.

**[S4]** The inclusion of Greedy SL+ to account for the additional degrees of freedom of the proxy networks in InfoPro is a good attention to detail in the empirical evaluation.

**[S5]** The paper looks at a reasonable set of tasks and studies the performance on relevant data, for instance Imagenet and Cityscapes semantic segmentation (but consider [W1] below)

### Weaknesses:
**[W1]** Baselines of empirical comparisons: With the exception of Fig 3, the empirical results are compared to only a comparatively simple baseline (Greedy SL/SL+), but not the same state of the art mentioned in section 4.1 (DGL, DIB, BoostResNet). This seems a rather odd omission, particularly as Fig 3/Section 4.1 elaborate on alternative methods (DGL, DIB, BoostResNet). It would support the paper's case to either include the results of those in table 2 or elaborate on their absence.

**[W2]** Fig 3: The scale of the y-axis is chosen in a way that a casual reader may visually misinterpret the absolute difference in error of the shown methods. For instance, at first glance DGL at K=2 seems twice as bad as Infopro, while in actuality is only about 14% worse (error from 7.76% to ~8.8%)

**[W3]** A methodical concern is the choice of $\phi$ and $\psi$ and the limited discussion around their choice (section 3.3, App. E): What is the sensitivity of the optimization with respect to their size and structure, what happens if I make them larger or smaller, how small can I make them? How would they look like for objectives other than classification?

### Further comments:
**[C1]** It would be insightful to consider what the implications for networks with recurrent structures would be.

**[C2]** It took a while to infer in section 3.3. that $\mathcal{R}$ is the reconstruction of the input data. A short sentence there may help future readers go through more smoothly.

I feel that I have learned something from the paper. It discusses the contributions, motivations and method reasonably thorough. My concerns are largely around the empirical substantiation of the claims, see [W1].
review_title: Insightful, some shortcomings in empirical evaluation

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper analyzed the reason why locally supervised training led to performance degradation. And based on the analysis, the author proposed the information propagation loss (can be understood as the combination of a classification loss and a reconstruction loss) aiming to prevent information collapse. Equipped with the proposed method, 40% memory footprint can be reduced demonstrated by their experiments, which is surprising. 

Strength 
(1) Deep neural networks need heavy memory cost, especially for object detection and segmentation task. Reducing memory cost in the training can enlarge the batch size in the training, thus speed up the training process. This is a very promising direction.   
(2) The paper is well motivated by analyzing traditional locally supervised training. Moreover, the analyze of information in features is also informative and reasonable. 
(3) The experimental results are good. The proposed method can achieve comparable performance with 40% less memory footprint. 
(4) The paper is clearly and is easy to understand.
(5) The proposed method is simple but efficient. And it can be used in different tasks, like classification and semantic segmentation.

Weaknesses 
(1) This is seldom comparisons with other methods in this paper. Only comparisons with the baseline are provided.
(2) Is the computational overhead is measured by training time or theorical computation cost? The comparison of real training time is more informative.
(3) Comparisons with other methods for reducing the memory consumption, e.g., [a] should also be added since they share similar aim. [a] Chen et al. Training Deep Nets with Sublinear Memory Cost.
(4) Recent contrastive learning works show that hyper-parameter τ in contrastive loss plays a very important role, like MOCO. Though τ = 0.07 is fit for MOCO, the setting here is different from MOCO. In this paper, the contrastive loss makes use of label information. If the ablation study for training hyper-parameter τ in the contrastive loss function is provided, it will be good. 
(5) In the experiments, the contrastive loss outperforms cross-entropy loss. And the authors attribute this phenomenon to using larger batch size for contrastive loss. However, I think the strong regularization on intra-class representation may be the key factor. Anyhow, ablation study of different batch sizes for contrastive loss should be added here.

review_title: Good motivation but lack of comparisons

Meta-review: All reviewers agree that the paper brings new knowledge in the field of locally supervised learning, and as such it should be accepted.  The authors should keep all reviewers' comments into account when preparing their camera ready version.
Decision: Accept (Poster)