Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: ### Summary

This paper proposes a differentiable architecture search approach for splitting a deep network into locally-trained blocks to achieve training speedup. The approach achieves better performance than using backprop on small datasets (CIFAR10 and TinyImageNet), and comparable or slightly improved performance on ImageNet with 2x claimed training speedup. Learned network architecture choices seem to transfer between datasets.

### Strong points

S1: The method outperforms comparable baselines, and significantly improves performance over standard backprop for small datasets (CIFAR10 and TinyImageNet)

S2: The learned network architectures transfer between tasks.

S3: On ImageNet, the method achieves slightly improved performance with claimed 2x training speedup.

### Weak points

W1: This seems like a pretty complicated approach to only get 2x speedup in training, and besides slight improvement in performance, that seems like the only benefit this method achieves for realistically large datasets like ImageNet.

W2: "Speedup" is not really defined. i assume this is the wall-clock time to achieve convergence, but I couldn't find a definition of convergence or stopping criterion, or how speedup is measured.

W3: Complicated optimization tricks seem necessary to get the bilevel optimization to work (Section 3.3: "In bilevel optimization, meta variables (α, β) depend on the learning trajectory of layer and auxiliary weights (θ, φ) (i.e. a sequence of values of (θ, φ) during inner optimization). As a consequence, there exists a risk of overfitting the meta variables to a specific episode"). If I have the choice between 2x training time with straight-forward optimization, versus 1x training time with complicated bilevel optimization that needs to be tuned, I would probably choose the former option, since I need to spend a lot less time debugging and tuning parameters.

### Recommendation

I think this paper is marginally below the acceptance threshold. The results seem decent, though a little underwhelming (see W1), but the biggest concern for me is that key concepts are unclear or ambiguous (e.g. definition of speedup is unclear, see W2, "confidence of \alpha at lower layers", see C2, and how the "backprop" gradients are computed and used in equation (2), see Q2).

If these issues were clarified I would be inclined to increase my score. But as it stands, this feels like a complicated method without much payoff (only 2x training speedup with slightly improved performance). It's possible that I'm missing something or not understanding the potential impact of the results, and explicit discussion of this and/or improved clarity of the description and evaluation of the method may help.

### Questions

Q1: What is the absolute wall-clock time for training that the speedup is reported on?

Q2: Given that the gradients used to train each block are a combination of local and backpropped gradients, how does this method avoid the locking problem, and thus provide speedup? 

### Other comments

C1: The "2.02x" speedup number looks a little funny; are two decimal places justified? Perhaps better to just say "2x speedup"?

C2: In Section 4.4, the paper says, "Surprisingly, SEDONA yields much more confident values of $\alpha_1^{(l)}$ (i.e. close to either 1 or 0) at lower layers than upper layers,", but when I look at Figure 4, at first glance I thought that \alpha is _less_ confident at lower iterations, according to the provided definition, compared to later iterations (which are all at 1). But then I realized the plot uses a log scale. Consider using a linear scale here instead? Or at least describe the confidence in terms of the log scale.

C3: I find the usage of all-caps CONV to be a little odd, at least I've never seem that before. Maybe just say "convolution"? Doesn't seem like CONV saves much space.

C4: I find this key statement to be unclear: "denotes whether layer l should utilize local gradients (i.e. the last layer of a block) or backpropagated gradients (i.e. the inside layer)." Is this saying that if a local gradient is used for a layer, then it is a candidate for being the last layer of a block, and if backpropped gradient is used, then it is a candidate for being the inside layer of a block? Some rewording of this sentence should help.

C5: Typos, in Appendix B: "to flat the learning landscape" -> "to flatten the learning landscape", Section 4.1: "to let auxiliary networks in the pool computationally lightweight" -> "to let auxiliary networks in the pool be computationally lightweight", Section 3.3: "Such warm start" -> "Such a warm start", Section 1: "If local signals are lousy representative of the global goal" -> "If local signals are not representative of the global goal"
review_title: Constrained architecture search that seems to mildly improve performance and speed up training by 2x, but somewhat unclear and hard to understand

final_score: rating: 7: Good paper, accept
review_content: The paper proposes a method for decoupled training of neural networks called SEDONA. In the spirit of recent trends in greedy layer-wise and indirect training, SEDONA allows gradient information to flow either from the next layer as in backpropagation or from an auxiliary head, trying to make a prediction using the current layer's output. Since a direct search for the best decoupled configuration results into probing a combinatorial number of splits, authors propose a continuously relaxed formulation which they later discretize. Transferring the found decoupled configurations to datasets different from the "pretraining" ones shows improvements in terms of validation accuracy and training time.

I find the overall topic of asynchronous or backpropagation-free training of neural networks very interesting and the submitted paper particularly relevant to this topic. The proposed search algorithm is novel to my knowledge and may be found useful in the community. However, I think authors should have put more effort into understanding SEDONA. 

The major points are the following:
1. The effect of discretization is not studied. I would be interested in seeing how well does a continuous configuration perform and what kind of accuracy loss (if any) is caused by discretization. 
2. I wonder why SEDONA performs better than backprogation (sometimes much better as in Tiny ImageNet)? Is it because not following the true (stochastic) gradient acts as regularization? One way of assessing this would be to also provide training loss or accuracy values. Why is your top1 error value for ResNet-152 is about two percent higher that in the original paper by He et al, 2015?
3. If the update directions configured by SEDONA lead to such a good performance, how do they correspond to the true gradients? Do they correlate positively? To what extent can such decoupled feedback implement credit assignment in a neural network?

I believe that addressing these questions is very important prior to publication of the method.

Minor comments:
1. $\theta$ does not appear on the RHS of (1).
2. Auxiliary head ensembling is not explained very clearly. 
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: Summary Of Contributions: This paper proposes to automate the design of auxiliary network and its allocation under decoupled neural network scheme, a design that speeds up network training and potentially boost model accuracy. The approach is validated with leading performance on ResNet and VGGNet under various datasets.


Strengths:  The idea to search auxiliary network for decoupled neural network is novel, and the proposed method is verified on multiple widely used datasets.

Weaknesses: 

1.	It’s unclear if the proposed differentiable search algorithm works better than DARTs with direction comparison, as the proposed method is basically DARTs with some changes (e.g. weight sampling, extra inner optimization steps per outer optimization), that are also applied to typical NAS tasks, a comparison with DARTs is needed to validate how much of the gain is from the proposed method rather than the fact of automating network design itself.

2.	It seems that the decoupled neural network scheme has nearly no gains in terms of Top-1 without the auxiliary heads ensembled as shown in Table 3. The details should be provided on how the ensemble works, are ensembled auxiliary heads kept during the testing stage? If so, Table 3 cannot tell if the gain is from the ensembled aux heads or decoupled neural network scheme. Network trained with Backprop method and identical extra aux head for ensemble might reach the same level of performance.

3.	Experiment with DGL in Table3 (b) is problematic, DGL in its original paper only provides ResNet-152 result with K=2, and it is better than Backprop according to their experiment. Yet the authors set K=4 in Table 3 and got a contrary result. This brought up the suspect of cherry picking of hyperparameters and can only be addressed with stricter comparison.

4.	One of the advantages to adopt a decoupled neural network scheme is training speedup, while the network based on compared methods is instantly available, the proposed method requires extra time to search before training. With this extra cost, how much the training speedup benefit is still left? The search cost should be indicated and the overall time cost should be discussed.


review_title: Major concern is about the experimental results

Meta-review: The paper proposes a novel method for greed layer-wise training by considering the learning signal from either backprop or from the additional auxiliary losses. SEarching for DecOupled Neural Architecture learns to identify the decoupled blocks by learning the gating parameters similar to gradient-based architecture search algorithms, such as DARTs.  The empirical experiments demonstrated the effectiveness of SEDONA on CIFAR and TinyImageNet using various ResNet architectures. Several issues of clarity and the correctness of the main theoretical result were addressed during the rebuttal period in the way that satisfied the reviewers. The ideas in this paper are interesting and are broadly applicable. Additional experiments / discussions on the tradeoff between initial search cost and accuracy should be included in the final version. 
Decision: Accept (Poster)