Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a method to scale the initial parameter blocks by a set of scalar values such as to achieve better optimization properties during the course of training. This paper follows up on previous approaches of similar nature but does it more efficiently and in a simpler way, achieving better results. The paper is clearly written, and it was a pleasure to read.



main_review: This paper proposes a method to scale the initial parameter blocks by a set of scalar values such as to achieve better optimization properties during the course of training. This paper follows up on previous approaches of similar nature but does it more efficiently and in a simpler way, achieving better results. The paper is clearly written, and it was a pleasure to read.

Overall, this is an important area and this paper achieves good initialization with only 1% of the total compute cost. The paper is also written in a way to convey both intuition as well as good detailed implementation details. 
One area where the paper can potentially improve would be to improve the overall accuracies compared to previous methods.

Questions:
- For the section starting with Line 153, it is mentioned that if S and \tilde{S} were the same, not enough randomness would be captured. However, is it not possible to keep S and \tilde{S} the same for the same t, and S can be different for t+1? (i.e. the step t from line 3 of Alg. 1)

- Line 189 mentions the added complexity of 2nd order gradients, however table 2 doesn't show significant increase in running time, so whether or not there are 2nd order derivative terms for penalty method is irrelevant? 

- Line 230: Why was only SGD used and Adam not used for the image datasets? what happens to the results if Adam were to be used instead of SGD?

- Typo on line 108
- Minor, but it would be good to discuss Admin before line 303.


limitations_and_societal_impact: yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This research proposes a novel initialization method for architecture agnostic networks by learning a scalar coefficient of a random weight tensor. Compared with the existing initialization method MetaInit, the proposed method is more computationally efficient and can be directly combined with different optimizers.
The motivation and heuristic of this paper is simple but interesting. However, the reviewer thinks this idea is worth digging deeper and requires more validations, not limited to an initialization method. 

########################################################################################

The authors have addressed most of my comments and concerns.
However, the reviewer still thinks that the scope of this paper is a bit narrow, if the scope of the current paper is limited to improving the initialization, the authors should provide some theoretical proof and analysis to enrich the paper.
Therefore the reviewer still keeps the score.

main_review: (1) In line 38, is “the variance” right? I think it should be the learned scalar coefficient. Please check it.
 
(2) The advantages of the proposed method mainly lie in that it is independent of the network architectures. And it can be combined with network architecture search in future works. In ideal cases, the proposed method should generate a proper initialization for arbitrary networks. The authors also claimed that other architecture-specific initialization schemes are not always portable
to new architectures. Therefore the effectiveness of the proposed method on networks with more complex architectures, like DenseNet, GoogleNet, should be further validated.

(3) In line 291, you claim that the acceleration achieved by GradInit is even more significant than FixUp. Can you report the specific results including the comparisons of initialization optimization time, training time, training epochs between FixUp and your method? Because your method uses a smaller batchsize and this will cause a longer time for a single epoch in the practical training experiment though you may need fewer epochs to train the model.

(4) The main concern of the reviewer is that this idea is worth digging deeper. For example, considering the motivation of this paper “the norm of each network layer is adjusted so that a single step of SGD or Adam with prescribed hyperparameters results in the smallest possible loss value”, a more straightforward and intuitive solution is to design a new linear layer with scale alpha and bias beta (different from the *trainable* linear layer in batch normalization). This linear layer will be adjusted according to the motivation dynamically during the training process, not limited to the initialization.

limitations_and_societal_impact: The motivation and heuristic of this paper is simple but interesting. However, the reviewer thinks this idea is worth digging deeper and requires more validations, not limited to an initialization method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes GradInit, a method to learn a set of scaling parameters ahead of training to better initialise networks.  Those parameters, learnt using very few batches in the training set,  scale the weights after initializing them with known initialisers like Kaiming or Xavier. The method aims to provide an automated easy to use approach for neural network initialisation that is agnostic to the model architecture and task at hand.
main_review: The paper is well written and touches upon an important problem. 

I am not very familiar with the line of research investigating the network initialisation problem. Having said this, I would expect a paper that looks into this problem to follow a systematic evaluation evaluation protocol and, as a result, include substantially more results, specially as far as model architectures are concerned. For example, in the paper only two types of architectures ResNet and VGG are evaluated on image classification tasks. I think this is a particularly weak point of this paper specially since the method is framed as an _initialisation scheme for any architecture_.

If this work (or a future version of it) could compare a much wider set of model architectures (maybe tasks too), it would certainly be of very high value for the community at large.

It would be interesting to report the training/validation curves instead of only the accuracy after the first epoch. I am a bit unsure if comparing accuracy after first epoch given a method that effectively “looks” at the training set ahead of the training, is very informative of the method’s real performance (compared to others such as Kaiming).

From the language models experiments, I only find Figure 2 to clearly show the advantage of GradInit. If the range of $\eta_{\max}$ and $\beta_2$ values considered was wider and still showed GradInit being unaffected, that would strengthen the evaluation section.
limitations_and_societal_impact: - Limited evaluation of the proposed method.
- No clear advantage over existing techniques.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce a new automated and architecture agnostic method to initialize the weights of a neural network. The method incorporates a pretraining step where a scalar coefficient for each of the weight matrices is optimized to minimize the loss on the first step of the training pipeline, subject to a constrained gradient norm. The authors show that their initialization method achieves better performance than other initialization schemes on a variety of image classification tasks applied to a medley of different architectures and a machine translation task applied to a Post-LN Transformer architecture.
main_review: Originality:
Other initialization schemes have been proposed in the past such as MetaInit. However, GradInit uses batches of training data and is more computationally efficient as well as memory efficient. It is also more adaptable to different optimizers and hyperparameters.

Quality:
This is a high-quality paper with most of the claims supported through experimental results. However, I would have appreciated more experiments with non-residual architectures. The authors claim that their method generalizes to all kinds of architectures, but in their experiments, they only compare their method on VGG-19, ResNet, Wide ResNet, and DenseNet architectures for CIFAR-10, the ResNet architecture for ImageNet, and the Post-LN Transformer architecture for language. Perhaps the authors could show that their initialization method works for other more exotic architectures from the Neural Architecture Search literature which are proven to have higher performance than ResNet models. 

In the Table 4 results, the authors claim that the FixUp $Acc_{best}$ is 75.7, but the FixUp paper claims that they received a final accuracy of 76% for the same experiment. I am not sure why there is a discrepancy here.

Clarity:
Most of the paper is written very clearly. I especially enjoyed the method section where the optimization procedure and the algorithm were invaluable to my understanding of the method. However, there were a few parts of the paper that were not as clear to me:
- How is the grid search for $\tau$ performed? Is there a held-out validation set that $\tau$ is fine-tuned on? 
- Table 1 is unclear. I assume the model was consistent across the experiment. However, Table 1 seems to indicate that there are two models: VGG-19 and w/o BN.

The paper also relies heavily on the Appendices. Certain sections do not make sense without reading the Appendices. For instance, the "GradInit further stabilizes feedforward nets with BN" section relies on Figure 3 to be understood.

Significance:
While the paper does show an improvement in results compared to the other considered initialization schemes, the improvements are very marginal (less than a percentage on the other initialization techniques; some of which are much simpler to implement). This questions the merit of using such a complicated initialization process in practice. In the language setting, adding GradInit to Admin does not seem to show any improvement. 

On the other hand, it seems like using GradInit has several other appealing properties like increased stability and efficiency which can be used in newer models that might be more unstable than the current ones.

Other miscellaneous items:
- Line 70: how can there be infinite variance, but a finite 2-order moment?
- Line 81: "possbile" should be "possible".
- Line 138: "is that is" should be "is that it is".
- Line 225: "labbeled" should be "labeled".
- Line 245: "decay the learning rate by 10" is not clear.
limitations_and_societal_impact: The authors have adequately addressed the limitations of their work.
review_title: null

Meta-review: All reviewers were positive about this paper: it suggests an initialization procedure GradInit which improves upon a previous method MetaInit, in accuracy, but especially in terms of computational efficiency. A recurring and remaining concern from the reviews was that the authors claim an "architecture agnostic method", but don't show it in many architectures. I therefore recommend the authors to tone down this claim. Also, I was not sure about the significance of the result that GradInit enables training without learning rate warmup in Post-LN Transformer, since it is not clear to me if warm-up is necessarily more expensive than running GradInit. Therefore, I recommend that the authors should mention this in the final version.
Decision: Accept (Poster)