Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper suggests an adjustment to the resnet architecture, which enables successful training despite having deterministically initialized weights.
main_review: Strengths:
* (Significance) The paper explores common-sense modifications that are important for understanding the operation of residual networks.
* (Significance) I found the results of the paper to be interesting.
* (Clarity) The majority of the paper is clearly written.
* (Reproducibility) The paper provides code, helping reproducibility.
* (Quality) The experiments include multiple seeds for every configuration, with a clear description of the used hyperparameters. The tasks used for experiments are highly relevant (e.g: Resnet50 for Imagenet). As such, I found the results to be highly reliable.

Weaknesses:
* (Significance/ novelty) Getting rid of batch normalization was already achieved by fixup, in a very similar method to what the paper presented. (Which did not require initialization with only ones and zeros). The experiments do not show a benefit for using the new modified resnet over fixup. Other works, like [SkipInit](https://arxiv.org/abs/2002.10444) and [ReZero](https://arxiv.org/abs/2003.04887) also offer similar methods to improve convergence speed and get rid of batch normalization. 

* (Significance) The paper pride its initialization on being deterministic, but this is poorly defined. I could argue that my initialization would be just as deterministic by using constant seed or importing weights saved on storage. 

* (Quality- Empirical) I was not convinced that the 4th contribution (lower standard deviation) is correct. Comparing standard deviation over 5 seeds does not give you a statistically significant result, this could easily be the result of random chance. I was also not convinced this is important: If my goal was to reproduce my run, I wouldn't have changed the seed.

* (Novelty) Initializing residual networks to only zeros and constant was already done in [this work](https://arxiv.org/abs/2007.01038). This paper cites this work as requiring "random noise to improve the performance". However, the prior work also concluded that "standard GPU operations, which are non-deterministic, can serve as a sufficient source of symmetry breaking to enable training" (quote from the abstract). Given that this paper also uses GPUs and did not mention using them on deterministic mode (It's stochastic by default), the specified distinction between this work and the prior work is not valid.

* (Clarity) I was confused by section 2.2. As far as I can tell, the degeneracy problem there is a direct result of the choice of padding $\mathbb{I}_{1,2}$ with zeros when the network is wide. Since this is not a trivial solution to the expanding width problem, the title "Width expansion leads to training degeneracy" is misleading, and the addressed problem in this section is poorly defined. I am also confused by why this problem is different from the first dead neuron problem (Skip connections obviously can't solve dead-neuron problems for neurons it is not applied on). 

* (Clarity) Section 2.4:  my understanding was that training degeneracy refers to a problem where padded zero values cause the gradient to remain zero  (section 2.2). I do not understand why $C_f$ and $C_b$ are good measures for that kind of degeneracy. Also, $C_f$ and $C_b$ aren't properly defined well for zero-initialized weights.

* (Quality) The paper concludes with results I consider to be negative. I thought this was interesting, as some of the changes made in this paper seem trivial and I did not expect them to harm performance. However, there was no proper ablation study (Figure 3 helped, but wasn't sufficient), so it remains unclear what element exactly caused the small degradation in performance. For example, I wonder if the single-convolution-skip architecture is responsible, but there is no way to disentangle the results from the entire zero-initialized-hadamard network. 
summary_of_the_review: The paper suggests some interesting modifications to the resnet architecture. However, the changes aren't very novel, and the paper did not convince me that these new adjustments are beneficial. The results in the paper appear to be (slightly) negative, but there is no proper ablation study for them to be informative
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors propose some modifications to the classic ResNet architecture and initialization scheme, so that the network can be trained with fully deterministic initialization, even without any normalization layers. The authors motivate their design choices by identifying and analyzing the problems with the forward and backward dynamics of some naive ideas. Then the authors present their approach and explain why each choice they make can solve the specific problems they have presented. Then they evaluate their proposed method along with some baselines on two popular image classification datasets, showing that their deterministic initialization method is on par with the ones with random weight initialization. Finally, they present related work and conclusion.
main_review: ### Strength:

#### 1. The motivation of the paper is clear and sound.

#### 2. The idea of combining Hadamard transform with identity mappings to initialize ResNet-style architectures is not previously seen in the literature.

#### 3. Deterministic initialization reduces source of randomness, as is shown by the standard errors of trial results of different methods.

#### 4. The paper is easy to comprehend.

#### 5. Code is shared anonymously for reproducibility.


### Weakness:

#### 1. While the approach as a whole is new to my best knowledge, most ideas have been investigated in published work. For example, identity initialization of convolutional kernels was presented in Xiao et. al. (2018) under the name of Delta-Orthogonal initialization; additional residual connections appear in related work such as Hardt and Ma (2017); Hadamard transform layer is used in binary neural networks, among other neural network applications.

#### 2. The experimental results do not show performance advantage of the proposed method over previous ones such as the original ResNet and the Fixup initialization (in the no normalization setting).

#### 3. The mathematical / technical contributions are limited; though this may have the virtue of improving the overall readability.

*Reference:*

*Xiao et. al. (2018). Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks.*

*Hardt and Ma (2017). Identity Matters in Deep Learning.* 

### Other comments and concerns:

#### 1. I am interested in the final weight distributions shown in Figure 3 (Left). I wonder whether this figure implies the proposed method can obtain solutions "closer" to identity mapping.

##### 1.1. Do the plots correspond to the same network architecture, only different initializations?

##### 1.2 What are the initial weight variances of different initializations (as it could be that they start from very different distributions)?

##### 1.3. Since the network block contains a scalar multiplier, there can be identical blocks (in terms of the input-output mapping) with very different weight distributions. Could that be the case here? 

##### 1.4. Could you provide more justification as to the connection of close-to-zero weight distribution to weight pruning? In my understanding, a more direct argument is to analyze the network's sensitivity to weight perturbation; due to the presence of scalar multipliers and/or normalization layers, small weight scale does not necessarily imply small sensitivity.


#### 2. In "Experiments - Hyperparameter settings", you mention "learning rate warmup is essential for ZerO to achieve a large maximal learning rate". This seems to suggest that the proposed initialization still suffers from training instability resulted from gradient explosion.
##### 2.1. To address this concern, could you provide some comparison (e.g. gradient norm plots) of your method vs. the baselines in the first hundreds of steps with and without warmup schedule?
summary_of_the_review: The paper introduces a novel method to initialize a ResNet-like architecture deterministically with binary weights (except for the Hadamard transform where additional rescaling is required). The paper contains some novelty in its model design, has good clarity, but does not show convincing practical significance compared with previous work. Some claims are hypothetical and may require further justification or be supported by further evidence.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: To initialize a ResNet with only zeros and ones, this paper analyses the problem of all-zero initialization and then proposes a  method named ZerO to achieve the goal by augmenting the standard ResNet architectures with a few extra skip connections and Hadamard transforms. The proposed idea is interesting and somewhat novel. However, it seems to introduce a new structure rather than only initialization, and empirical results are not really good.
main_review: Pros:
- This paper is easy to follow.
- The proposed method is interesting and somewhat novel.
	
Cons:
- This paper seems to propose a new structure (containing extra skip connections and Hadamard transforms) that contradicts the topic ( an initialization). The proposed method ZerO overly relies on auxiliary structures, compared with FixUp [1] that only uses a multiplier and bias. Therefore, the proposed method is not a pure initialization for ResNet.
- In Table 2, does ZerO only use a structure with AugSkip? Wether other initializations (i.e., Fixup, Kaiming, and Xavier) choose vanilla ResNet? There is no description to mention it. In addition, ZerO performs worse than Fixup. Although it can reduce the gap by running 180 epochs, it costs too much on time.
- It is suggested to change notation like $W^2$ to $W^{(2)}$.


Additional Questions:
- As a fully-deterministic structure, the initial weights are the same with different seeds. Why is the std not equal to 0? What else makes an effect on the training?
- Will the auxiliary structures influence time consumption?
- Kaiming initialization is proposed for relu activation function and ResNet adopts relu. Therefore, is it weird that Kaiming is worse than Xavier in Table 2?

[1] Hongyi Zhang, Yann N. Dauphin, Tengyu Ma. Fixup Initialization: Residual Learning Without Normalization. ICLR 2019.
summary_of_the_review: Although the idea is somewhat novel, for the unremarkable empirical results, incongruent method contradicting the topic initialization, I regret to reject it weakly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a deterministic weight initialization scheme called ZerO for residual networks.  ZerO works by first augmenting existing ResNet architectures with extra skip connections and Hadamard transforms and then initializes network weights with only zeros and
ones (instead of random weights). The authors show that the proposed scheme has various benefits such as improving reproducibility, training networks without batch normalization, and improved performance.
main_review: Strength:
The idea of ZerO is novel and has various benefits such as improving the reproducibility of models and training networks without batch normalization. 

Weakness:
1. One drawback of ZerO is that it can only be applied to residual networks.
2. Can the authors also provide a comparison with the baselines for space-time complexity? I think that the number of skip connections added is significant which may add to the complexity.
3. How about applying ZerO for other core CV applications such as segmentation and object detection. Does the behavior of ZerO remain the same for such applications?
summary_of_the_review: Overall the paper presents some interesting ideas. However, there are some concerns such as the space-time complexity of models and limited applicability.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new initialization scheme for ResNet. The key feature is the initialization values are set to all zeros and ones, rather than random. The development of this initialization scheme is motivated by avoiding the "dead neuron" problem and "training degeneracy" problem for networks with zero-initialized weights. Some toy examples are discussed to illustrate the motivation. Last, experiments on standard benchmarks are conducted to verify the initialization scheme.
main_review: Strengths:
1. It is interesting to show that ResNets can be trained to get competitive performance with zero-and-one initialized weights. The conventional view is that the weights should be initialized with random values for symmetry-breaking.
2. The problems (e.g. dead neuron) discussed in the paper are well-illustrated. In Section 2.1, the authors use simple and clear examples to demonstrate how zero-initialized weights can cause training to get stuck and why short connections can rescue. In Section 2.2-2.3, the authors explain in a similar manner why Hadamard transform can deal with degeneracy problem caused by the dimensionality expansion.
3. Experimental results confirm that randomness can be removed from weight initialization. The authors compare several initialization schemes on the standard ResNet models on CIFAR-10 and ImageNet and show theirs achieves competitive results in terms of convergence speed and test accuracy.

Weaknesses:
1. The motivation of the zero-and-one initialization is unclear. What are the theoretical advantages of removing randomness in initialization? It seems to me that keeping randomness in initialization has the advantage of potentially avoiding bad local minima and saddle point. Empirically, despite the authors have shown some comparison with He and Xavier methods, the zero-and-one initialization does not show *significant* advantages.
2. The zero-and-one initialization is of a heuristic nature and lacking of theoretical background. There is no unique way to avoid the dead neuron problem. For a trivial example, one can simply add a shortcut connection from the dead neuron to a output neuron and one from a input neuron to the dead neuron. It would be desirable to motivate the proposed initialization scheme with some theoretical optimality.
3. Additional costs of the zero-and-one initialization. The extra shortcut connections and Hadamard transform result in additional computational cost and model complexity. As they are cheap operations, I consider this issue a minor one.
4. In Section 1, the authors state "In order to design networks robust to vanishing gradients at initialization, He et al. introduced
skip connections". However, it is contradictory to the claim in the original paper of He et al, which states "We argue that this optimization difficulty is unlikely to be caused by vanishing gradients."
summary_of_the_review: The paper is technically correct but lack of strong motivation, theoretical advances and empirical performance.
review_title: null

Meta-review: This paper suggests an architecture with a deterministic initialization which has only 0/1 values.
The reviewers were mostly (marginally) negative, mainly because of the low novelty and significance of this work. 

Specifically, the main novelty issues were:
1) Improving convergence speed and removing BatchNorm: was already done, in a quite similar manner, and it achieves better or similar results. (Fixup , ReZero: https://arxiv.org/abs/1901.09321, https://arxiv.org/abs/2003.04887, and few others as well)
2) Initializing a network with deterministic initialization: was also done (ConstNet, https://arxiv.org/abs/2007.01038). I think the main difference from the previous work is the additional Hadamard connections, which help break the symmetry. However, it is unclear what is the benefit of this modification, as the previous work could train without it (albeit on CIFAR).

Specifically, the main significance issues were:
1) Reducing standard deviation: The authors' response confirmed there is no statistically significant benefit (p=~ 0.1) for variance reduction when comparing with Kaiming initialization for ImageNet.
2) General network performance: The results do not seem better than the baseline (Xavier init is not a proper baseline in a network with ReLUs).
3) Sparsity claims: The network appears to be losing accuracy even with 20% sparsity, which isn't even useful for efficiency. For comparison, the lottery ticket hypothesis showed you can get to 90% sparsity and get better results. So, this is a nice observation, but not a major contribution.

Therefore, I recommend the authors to better distinguish themselves from previous works (What are the changes? Why are these important?), and improve their empirical results so they highlight the usefulness of the suggested method (e.g., improve the SOTA in some benchmark).
Decision: Reject