Initializing ReLU networks in an expressive subspace of weights

Abstract: Using a mean-field theory of signal propagation, we analyze the evolution of correlations between two signals propagating forward through a deep ReLU network with correlated weights. Signals become highly correlated in deep ReLU networks with uncorrelated weights.  We show that ReLU networks with anti-correlated weights can avoid this fate and have a chaotic phase where the signal correlations saturate below unity. Consistent with this analysis, we find that networks initialized with anti-correlated weights can train faster by taking advantage of the increased expressivity in the chaotic phase. An initialization scheme combining this with a previously proposed strategy of using an asymmetric initialization to reduce dead node probability shows consistently lower training times compared to various other initializations on synthetic and real-world datasets. Our study suggests that use of initial distributions with correlations in them can help in reducing training time.

Reviews and author responses:

The authors propose an initialization method called random asymmetric anti-correlated initialization (RAAI), which initializes residual networks in an expressive subspace of weights. RAAI is a combined initialization of anti-correlated initialization (ACI) and random asymmetric initialization (RAI), where the last initialization method is already proposed. Through both theoretical analysis and empirical evaluations, the authors show that RAAI has both a chaotic phase and low dead node probability, which accelerates the rate of convergence. Pros:
1. Very interesting theoretical results. Through mean-field theory, the author shows that a Relu network with anti-correlated weights can be more expressive by taking advantage of a chaotic phase. 
2. The paper is well-written and the proofs are clear.

Cons:
1. Poor empirical evaluations. I think the main drawback of this paper is that the proposed initialization is poorly evaluated empirically. The authors only evaluate RAAI in a simple teacher-student learning setup with non-standard residual structures. I suggest authors at least evaluate RAAI using a standard ResNet-18 on CIFAR-10 as a baseline comparison. A large-scale evaluation such as Imagenet would be better.
2. Unclear relationship between ACI and RAI. I am not sure why the authors need to combine RAI with ACI. From section 2, it seems ACI can achieve a chaotic phase by itself. I suspect it is because combining ACI and RAI can provide more expressive initial weights. It's good to clearly state their connections in the paper. Since RAI has been proposed before, it would also be good to disentangle ACI and RAI by comparing them separately.
3. (Possibly) unstable signal propagation. It seems RAAI is hard to be generalized to a deep network setting due to signal exploding. According to Figure 5, RAAI only accepts very small \sigma^{2}_{w} to ensure finite signals. 
4. The difficulty of choosing k. The authors mentioned that some experiments need to tune the anti-correlation strength k. How hard is this tuning? Would be great to provide some ablation studies about hyperparameters.
5. Why expressiveness of initialization is needed? Although this work proposes to initialize weights in an expressive subspace, it would be better to explain why expressiveness is needed for initialization. Overall, I think this is an interesting work that proposes an anti-correlated and asymmetric initialization. The paper is well-motivated, and theoretical results and numerical verifications are well-desgined. Anti-correlation is a good and novel way to improve the expressiveness of initialization, and thus it improves the rate of convergence. My major concern is its empirical evaluation given only a teacher-student setting with shallow residual networks considered. I would like to raise my score if more experiment results are provided.

This paper develops an initialization scheme which combines the anti-correlated weights with the asymmetric initialization to reduce the dead node probability of ReLU networks and improve its expressiveness.  This paper develops an initialization scheme which combines the anti-correlated weights with the asymmetric initialization to reduce the dead node probability of ReLU networks. It is claimed to reduce the training times on both synthetic and real-world datasets. However, the experimental results contradicts the claims. 

1. For the results on synthetic data shown in Figure 6, 7, 8, the best results are achieved with Adam, but the proposed RAAI does not converge faster than ACI and sometimes is slower than RAI.

2. For the result on real-world dataset, RAAI seems better, but no details of the hyper-parameter tuning process is given. Since different methods use different $\sigma_w^2$, it is reasonable to believe these initializations may correspond to different optimal learning rates.
3. It is unclear to me why the experiments in the main content only focus on the synthetic teacher-student setting. 

4. The analysis seems only applicable to feed-forward ReLU networks without normalization layers. For many practical systems, best result is achieved for networks with normalization layers, skip connections, and even attention mechanisms.

5. The concept of expressivity is not well-explained in the paper. It is not clear to me why a network with less dead nodes is more expressive. For feedforward ReLU networks, if there is no dead node, then it degenerates into a linear network, which might be the least expressive.

6. The claim "...it performs better than the best-known initialization schemes on tasks of varying complexity" seems too strong. The only compared baselines are He, ACI and RAI.  I feel the method is just a combination of anti-correlated weights and the asymmetric initialization, and the experimental results are not convincing enough.

The paper studies the signal propagation properties of deep ReLU networks at initialization time. For a standard ReLU network, the correlation between activations of two training examples converges to one as the depth goes to infinity. To resolve that, the authors proposed to use correlated weights at initialization time to avoid this pathology. With correlated weights, the correlation could instead converge to a value smaller than one (so-called chaotic phase). Empirically, the authors found the networks initialized with correlated weights train faster. Strengths:
- The idea of using correlated weights is interesting. However, this was first proposed by Li & Saad.

Weaknesses:
- The paper is poorly written and very hard to follow. There are many typos and grammar errors (see detailed comments below). Other than that, the misuse of \citet and \citep makes it unreadable. The random asymmetric initialization is one important component of the proposed initialization scheme, but the authors do not clearly explain what it is in the paper.
- The motivation is relatively weak. One can achieve a chaotic phase with many other activations. For example, ELU activation function has similar advantages as ReLU and can be initialized carefully to be in the chaotic phase. In addition, deep networks in the chaotic and ordered phase are either hard to train or impossible to generalize (see [1]). Both chaotic and ordered phases should be avoided when designing an initialization scheme.
- The contributions are neither novel nor significant. This work basically combined random asymmetric initialization with anti-correlated initialization, both of which were proposed before by other papers. 
- The experiments are all run on toy problems with relatively shallow networks (10 layers). No implementation and tunning detail is provided. The improvement of using the proposed initialization scheme is marginal on three real-world datasets.


Detailed comments:
1. Abstract: "intiialization" - > "initialization"; 
2. Introduction: "network depth and width" -> "network's depth and width"; 
3. Introduction: "ReLU CNN's produce" -> "CNNs with ReLU activation function produce"; 
4. Introduction: Inconsistent verb tense: mix of "we found" and "we find". 
5. Removing Ref in "Ref. Poole et al".
5. For many figures, the captions are not informative. For Figure 6 and 7, the authors should state the basic settings and datasets. For Figure 12-14 in the appendix, the captions do not match the axis labels.

[1] Disentangling Trainability and Generalization in Deep Neural Networks. ICML 2020. The paper is not well-motivated and its contributions are fairly minimal in both theory and empirical results. In addition, the writing is hard to follow and can be significantly improved. Overall, this paper has several substantial issues that make it unsuitable for publication at this time.

This paper proposes and studies the initialization of ReLU networks with correlated Gaussian random variables. Using the language of mean field theory, the authors define an ordered phase as two distinct signals becoming perfectly correlated asymptotically, and a chaotic phase as two signals losing all correlations. They find that while He init (uncorrelated Gaussians) only results in ordered phases, there exists an initialization with negative correlations that results in a bounded chaotic phase. They suggest that the use of anti-correlated initializations can speed up training. Strengths
- The proposal to initialize a neural network with anti-correlated Gaussians is novel (to the best of my knowledge).
- The paper gives a good overview of prior work, and is very clear about its contributions.

Weaknesses
- While the work is interesting and clearly defined, its significance might be of limited interest to the broader deep learning community.
- The experimental section was not particularly convincing: Figures 7 and 8 do not show the proposed RAAI as clearly superior to the competing initialization methods.

I think the authors can improve their work significantly by showing that their proposed method results in experimental benefits on a competitive benchmark/task with a modern neural network architecture, rather than limiting themselves to simple ReLU feedforward networks on small datasets like MNIST. I'm in favor of rejecting the paper at this point in time. The proposal of an anti-correlated initialization is novel and the technical analysis is interesting, but the significance of the contributions are not well-supported by the experimental results.

Meta-review: The paper proposes an initialization method to initialize residual networks in an expressive subspace of weights. Although the reviewers highlighted some positive aspects, they found the contribution to be limited compared to prior work. Some reviewers also raised some concerns regarding the experimental results not backing up the claims made in the paper. The authors did not respond, so I can therefore not recommend acceptance. This will hopefully provide useful feedback for a potential revision.

Decision: Reject