Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose Identical Initialization (IDInit) that initializes weights using identity matrix and its variants. They consider how to initialize non-square matrices, residual structures and convolutional operations using identity-like methods. Empirical evaluation demonstrates its good performance and fast convergence. 
strength_and_weaknesses: The paper is well-written and easy to follow. The authors also present extensive empirical evaluations to support their methods.

However, I have a severe concern regarding the effectiveness of IDInit. The identical initialization has been well studied in ZerO paper [1], where the authors theoretically prove that a simple identical initialization leads to training degeneracy in practical networks. The major reason behind this training degeneracy is that for a dimension-increasing matrix (e.g., the $D_{i+1} > D_{i}$ case in the current paper), a simple "identity + zero-padding" causes the rank constraint ((ii) in Theorem 1 in ZerO). 

To initialize dimension-increasing matrix, IDInit replicates the inputs to span over the output dimension. However, this causes a similar rank constraint (i.e., a symmetric problem) as each replica in the dimension-increasing matrix receives the same gradient (excludes some special cases). ConstNet paper [2] also shows the degeneration caused by insufficient feature diversity (i.e., the symmetry), which is related to the replication here. In addition, the authors only perform trainability analysis from a signal propagation perspective, without considering the potential degradation caused by the replication. 

I highly recommend the authors to empirically evaluate if IDInit can pass the simple rank verification in Figure 3 of ZerO paper. On the other hand, I suspect the surprisingly good empirical evaluation of IDInit comes from randomized operations (e.g., dropout) in networks, which directly breaks the training degeneracy. I suggest the authors to carefully consider this training degeneracy issue, as it affects the effectiveness and universality of the proposed method.

Other comments:

1. The need of zero preserving initialization (IDIZ) is unclear. The authors motivate IDIZ by arguing that Fixup or ZerO method will not work if the multiplier in batch normalization is initialized as 0. However, this is not a standard case because the multiplier in BN is usually initialized as 1. The motivation also requires a special case where BN is applied after the last conv layer. These reasons can not convince me why IDIZ is really needed. Do authors initialize the multiplier in BN as 0 in their empirical evaluation? If so, I suggest the authors to provide more details about this case.

2. How does IDIC compare to the simple IDI in convolution case? The authors mention that IDIC is used to improve feature diversity but it's unclear why it is needed. ZerO paper [1] directly utilizes IDI in convolution (Algorithm 2 in ZerO) and it works well.

3. Comparison to ZerO. Since ZerO is the most relevant work to IDInit I believe, I suggest the authors to empirically compare them, including both performance and determinacy analysis (since both methods claim the benefit as being deterministic).

[1] https://arxiv.org/abs/2110.12661

[2] https://proceedings.mlr.press/v119/blumenfeld20a.html
clarity,_quality,_novelty_and_reproducibility: The paper is mostly clear with high quality. However, I believe it lacks enough novelty. Identical initialization has been studied in ZerO paper [1], where IDInit and the proposed variant (Eq 14) in convolution are already considered. Its deterministic benefit is also studied in ZerO. It would be great if the authors can provide more details about the connection and difference between the two methods. Another concern is the reproducibility given the training degeneracy issue discussed earlier, which affects the effectiveness of the proposed method.
summary_of_the_review: Identical initialization is a very interesting topic and IDInit looks promising. My major concerns are its effectiveness (training degeneracy problem), necessity (motivations behind IDIZ and IDIC) and novelty (compared to ZerO). I would like to discuss these issues with the authors, and I am happy to raise my score if the authors can address my concerns.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The reviewed work proposes "identical initialization" as a novel initialization scheme for neural networks. The guiding principle of this initialization scheme is to ensure that each layer, at initialization, propagates activations identically.

The motivation behind this choice is that the resulting isommetry property of the network prevents vanishing or exploding gradients and thus allows for the stable training of deep architectures. 

Numerical experiments on classification for CIFAR10 and Imagenet show improvements due to the new initialization scheme. 
strength_and_weaknesses: ### Strength
The main strength of the paper are the empirical improvements apparently achieved by the proposed initialization scheme.

### Weakness
The first-order weakness of the paper is the poor quality of its writing. There are numerous grammatical mistakes and vague language such as "Therefore, if all values are set to 0, it can obtain non-zero gradients of a suitable magnitude." Due to the above, despite numerous attempts, I am still not entirely confident if I have correctly understood the approach proposed by the authors. 

A second weakness (possibly related to the first) is that the motivation for the proposed approach over FIXUP and ZerO is not clear. In particular the fixup approach, that similarly initializes the residual branches as products of identities and zero seems closely related. 

clarity,_quality,_novelty_and_reproducibility: As mentioned above, the paper is poorly written, with many grammatical mistakes and vague language. This makes it difficult to discern the novelty of the proposed approach compared to existing ones. Given the authors' awareness of FIXUP and ZerO, the claim "To our best knowledge, it is the first trial to put identical-like initialization into practice" appears to be false.
Regarding the numerical experiments, I am somewhat concerned by the low performance of the baseline in Figure 6. 
summary_of_the_review: In summary, the empirical results suggest that the authors may be up to an interesting result. However, the writing, presentation, and comparison to existing work need major improvement. In its present state, the paper is not ready to be published at a major ML conference.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new initialization scheme for both vanilla convnets and convnets with residual structures. This initialization is a simple and special case to maintain dynamic isometry at initialization time. Specifically, they initialize the weight matrix to be identity. The authors also propose a spatial shift scheme to improve the representation learning ability. Using this initialization, they can train nine-layer vanilla convnets and 110-layer residual networks on CIFAR. It is also shown this initialization method is helpful for transformer training.
strength_and_weaknesses: Strength:
1) This paper is motivated by the dynamic isometry formulation and propose a very simple initialization (identity initialization). The simplicity is good and empirically helpful on varies settings.
2) The proposed two techniques on improving the feature diversity are interesting.

Weakness:
1) Missing Reference: This paper misses discussion with DiracNet [1] and ISONet [2]. DiracNet reparameterize the convolutional kernals and essentially initialize it to be an identity. ISONet explicitly initialize the conv kernels to be identity both in vanilla convnets and residual convnets. It is necessary to discuss the differences and advantages over these previous works.
2) Some terms and examples are confusing.
- the robustness in this paper actually means insensitivity to hyperparameters. However, this term in machine learning usually means the network is insensitive to input noise.
- Table 1 is also confusing. Most of the entries in the “w/ BN” category should be put in the “w/o” because Zero \gamma, Fixup, Skip Init, ReZero are designed for training residual networks without normalization.
- Figure 7 can be replaced by a Table becuase the current figure does not have caption and it is not straightforward to get the point from the figure.
- The experiment setting and conclusion of section 4.4 is also confusing. Is the proposed initialization has any randomness? It seems the initialization is deterministic. In this case, why the std is not 0 in Table 1?
3) The central message of the paper seems to be unclear. The authors mix two information together: 1) the trainability of network without batch norm; 2) the proposed method improves over the original initialization. The first part is well-motivated because BN has many disadvantages. However, the proposed method only shows the trainability of a network with 100 layers on CIFAR. The ImageNet experiments are based on networks with varies normalization layers and the proposed method only has marginal improvement over the baselines.

[1] Zagoruyko et al. DiracNets: Training Very Deep Neural Networks Without Skip-Connections.
[2] Qi et al. Deep Isometric Learning for Visual Recognition. ICML 2020.
clarity,_quality,_novelty_and_reproducibility: The clarity of this paper can be improved a lot. I listed a few examples in the second point of the previous weakness section. I think the author should address them and further polish the storytelling and presentation. Besides, I feel I don’t obtain much insight from reading this paper. The authors may also consider to visualize the distribution of the weight matrix, e.g. how the weight matrix value changes through the training.

The paper has its own novel part. However, the central identity initialization has been proposed an analyzed before (ISONet and DIractNet, listed in the weakness).

It provides code. I checked the relevant implementation. I think the proposed method is simple and also easy to reproduce.
summary_of_the_review: In summary, this paper is interesting as it shows simply initializing the kernels to be identity can make vanilla networks trainble. However, the writing, experiments, and presentation need to be improved a lot. I cannot recommend acceptance for this paper before this improvement.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes to initialize each linear layer of a deep neural network to be identity so as to accelerate training.
strength_and_weaknesses: Strengths: 
1. The experiments are done on many datasets.
2. The code is attached for reproducibility.

Weaknesses:
1. It has been shown (Thereom 5 in [1]) that the identity initialization may not lead to convergence for deep linear neural networks. 
2. For residual layer, this work proposes to initialize the weight matrix with tiny values so as to avoid the dead neuron problem. This is not sensible. Because of the shortcut connections, weights with zero initialization still receive non-zero gradients.
3. The overall idea, identity initialization, is not novel. For example, see [2]. The proposed variant in this paper is technically trivial.

References:
[1] Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networks, Peter L. Bartlett, David P. Helmbold, Philip M. Long, ICML 2018.
[2] A Simple Way to Initialize Recurrent Networks of Rectified Linear Units, Quoc V. Le, Navdeep Jaitly, Geoffrey E. Hinton, arXiv 2005.


clarity,_quality,_novelty_and_reproducibility: Clarity is fine. Novelty is close to none. Reproducibility is good.
summary_of_the_review: This paper proposes identity initialization for deep neural networks to accelerate the training. I found the idea lack of novelty and the claim of solving the dead neuron problem insensible.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a new initialization scheme called "identical initialization". Unfortunately, the reviewers clearly agree that the current form of the paper is below the bar for acceptance. Common concerns include poor quality of writing, unclear main message, unclear motivation for the proposed approach over FIXUP and ZerO (despite the authors trying to address this point). Some reviewers also point out some missing references.

I'm therefore not able to recommend acceptance, and I encourage the authors to address the concerns of the reviewers in a revised version.

justification_for_why_not_higher_score: All reviewers agree this is a clear rejection
Decision: Reject