Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes random weight factorization, a new parametrization for linear neural network layers inspired by weight normalization. It argues that the parametrization reduces the distance between different parameter configurations in the loss landscape and reduces spectral bias, leading to performance improvements across a range of tasks.
strength_and_weaknesses: Strengths:
+ The proposed method is very simple.
+ The method is scalable and potentially widely applicable.
+ The paper presents results on a broad range of settings for the experiments.

Weaknesses:
- My understanding based on Tab. 3 in the appendix is that the optimizer setup is fixed across the parametrizations that are compared in each benchmark. As I read it, the paper argues that random weight factorization outperforms the alternatives, i.e. the best case performance is better. However, then the optimization setup would need to be tuned for each method independently. As it stands, the paper can roughly support the statement "For a range of benchmarks there exists some setting of the hyperparameter where the proposed method performs best". However, this is not all too relevant, it would either need to typically perform best across a range of hyperparameter setups (i.e. be more robust) or consistently deliver the best optimal performance. To me, the experimental setup is critically flawed and would need to be addressed for me to recommend acceptance (I'm open to discussion on this point and do not strictly expect all experiments to be rerun with a hyperparameter sweep for each baseline).
- There are no error bars in most experiments and I did not find any mention of multiple runs with different random seeds, which significantly weakens any reported performance gains. Also as a result, I'm not sure how to interpret e.g. the ablation study in Fig 6, it seems extremely noisy across the board with no observable trend beyond high errors for a mean of 4.
- The reported performance gains are mostly incremental (although this is probably not unexpected with a relatively small change such as the weight parametrization on something as basic as feedforward neural networks with best practices for training them that have been established over the past decade).
- The range of architectures is quite narrow, it seems like the parametrization would be straight-forward to plug into CNNs and Transformers if I'm not missing anything.
- I don't find the argument around the proximity of parametrizations too convincing, I'd expect these to be increasingly far away from the origin, especially in high dimensions. This probably would make them hard to reach by gradient-based optimization, in particular in conjunction with L2 regularization (although I suppose it depends whether that is applied to the weight matrices or the actual parameters). I would appreciate some more discussion around this or some small experiment on an actual neural network architecture.

Other:
* I haven't seen any of the benchmarks before, so find it difficult to judge the reported performance gains.
clarity,_quality,_novelty_and_reproducibility: **Clarity** Good, but given how simple the method is, this is to be expected.

**Quality** I think the experimental setup is flawed and

**Novelty** New as far as I'm aware, although highly similar to weight normalization, but this is acknowledged and discussed explicitly. The parametrization is not particularly involved, but discovering small variations on something as basic as linear layer parametrization is not trivial.

**Reproducibility** Hyperparameters seem to be reported exhaustively in the appendix. However, I could not find any discussion of how they were obtained, so while the experiments may be re-runnable, the overall process of the study is not reproducible.
summary_of_the_review: This is a simple, but potentially broadly useful idea. To me this puts the bar on the empirical evaluation rather high. Unfortunately, I believe that the evaluation falls short of that, so I do not recommend acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes random weight factorization as an initialization of the conventional linear layers. Theoretically, it is shown how this factorization alters the underlying loss landscape and effectively enables each neuron in the network to learn using its own self-adaptive learning rate. Empirically, the proposed initialization method can result in improving the training on a variety of tasks, including image regression, shape representation, computed tomography, inverse rendering, solving partial differential equations, and learning operators between function spaces. 
strength_and_weaknesses: Strength:
Overall, the paper is well-organized and easy to follow. This paper studies how to better initialize and factorize the weight parameters of the multi-layer perception. Specifically, they suggest using random weight factorization to factorize the original weight parameters. They first theoretically show that the space of this factorization is flexible enough to be arbitrarily close to the global minima. After that, they show that the proposed weight factorization can be recognized as a gradient descent with a self-adaptive learning rate. I also admire the tremendous experimental work in this paper.
Weakness:
1. My biggest concern is that I didn’t find much difference between the proposed method and the adaptive activation function method. I check the algorithm in Appendix B. It seems that the random weight factorization is only applied once in the initialization. After that, the model was trained with the standard gradient descent method. If we omit the initialization step, this is almost like training a model with an adaptive activation function. I am also concerned about how to ensure that the initialized scale factor s^{(l)} is better than the vector of ones. Please correct me if I am wrong.

2. The author claims that the proposed method can effectively mitigate spectral bias. But I am not aware of any such analysis in the experiment part. 

clarity,_quality,_novelty_and_reproducibility: Clarity, Quality, and Reproducibility are good. Novelty is minor.
summary_of_the_review: I didn’t find much difference in this work compared to the adaptive activation function method. In addition, the random weight factorization method only applies to the initialization.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a simple but well-performing technique of reparameterizing each weight matrix of implicit neural representations (INRs) as a product of diagonal matrix (with diagonal elements being equal to the exponential of a single parameter) and another matrix. Empirically, it turns out that the reparameterized INR converges to a better-fitting solution to the signal being trained.
strength_and_weaknesses: __Strength__
- The method is very simple and easy to use, can be used as a drop-in replacement of the weight matrices of INR. I especially like the fact that the method may not incur too much additional compute / memory on top of the vanilla INR training procedure, as the training cost is one of the central issues of INR.

__Weaknesses__
- The "exponentiation" in the main algorithm (in appendix B) is not well-motivated, and is not well discussed in the main text. Could authors give a little more explanation why using $\exp(s)$ instead of $s$ is a good thing to do? Perhaps a formal ablation study or theoretical justification may help.

- The explanation regarding SIREN is not clear enough to me. In page 4, authors argue that (1) using RWP is better than using the SIREN (2) the benefit of SIREN is mostly from using the reparameterization $\omega_0 \cdot \mathbf{w}$ rather than the periodic activation. I am not really sure about how authors showed this in Appendix G and H. It seems like in Appendix G and H, authors try various values of $\omega_0$ of SIREN to show that the peak performance of such fixed parameterization is lower than the value achieved by RWP. But it is not clear from the text with which activation function RWP and fixed $\omega$ are trained with. Did you use periodic activation function, or the ReLU? In either case, I do not see how this experiment leads to a conclusion that having periodic activation is less important than having a scaling factor in SIREN.

- Related to the previous point, I think that perhaps showing that RWP reparameterization boosts the performance of INRs even when the underlying model architecture uses a periodic activation function (like SIREN) is a right way to prove the benefit of RWP. Indeed, in many cases, SIREN is known to achieve way better/faster convergence than the ReLU-based models used in the experiments.

- In fact, I wanted to test the claims of the authors by myself but the code has not been provided (unfortunately). Providing codes will help readers validate the claim.

- I wonder if such reparameterization will help boosting the performance of meta-learned INRs (via MAML or hypernetworks). Is there any related experiment?

- In Theorem 1, I am not sure how meaningful having a small "distance" (which has not been properly defined, by the way) is, because the reparameterization also have a critical impact on the scale of the gradient of each component, s and V. In other words, despite having a smaller distance by the reparameterization, the effective number of optimization steps can be similar to the original parameterization.

- Results like Theorem 2 has already appeared in the work of Arora et al. (2018), which should be mentioned in the text. In fact, the theoretical results of Arora et al. may help authors explain the good performance of the proposed method.

---
Arora et al., "On the optimization of deep networks: Implicit acceleration by overparameterization," ICML 2018.
clarity,_quality,_novelty_and_reproducibility: The paper is quite clear overall, could be improved by:
- giving the definition of the "distance" measure in Theorem 1 explicitly, and
- having a formal algorithm in the main text and explaining why authors propose using $\exp(s)$ instead of $s$.

Regarding quality and novelty, I do not think any more work is needed.

About the reproducibility, having codes publicly available will be helpful.
summary_of_the_review: The proposed method is potentially a very useful technique for training implicit neural representations, but I have several concerns regarding the empirical validations of the technique, especially on how the method can be combined with the state-of-the-art architecture, SIREN.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method for improving and accelerating the training of coordinate-based networks by proposing randomized weight factorization. This factorization generalizes weight normalization, but can be applied in the coordinate-based network setting for signal memorization. It is done by factoring each of the weights in the layers into a multiplication between a diagonal scale matrix and a dense weight matrix. The paper demonstrates that this factorization theoretically improves the distance to a local minima, resulting in easier optimization and robustness to poor initializations. This is experimentally validated across a number of signal fitting tasks which coordinate-based networks are applied to, including image and 3D shape memorization, neural radiance field optimization, and partial differential equation solutions.
strength_and_weaknesses: In my opinion the strengths of the paper are:
- The paper is extremely clear in description of the proposed method and exposition of theoretical and experimental results. This is a plus, as it will make the impact of the paper significantly higher since it can easily be adapted into application pipelines. 
- The claims of improvement are backed up both theoretically and experimentally, and seem significantly better than the comparable baseline methods. The breadth of experimental experiments for which coordinate-based networks are used is great, and I find it valuable that in all of these situations the randomized weight factorization provides the same improvement.

In my opinion, there are no glaring weaknesses in the paper. One thing which would be interesting to be directly discussed would be the aspect of computational efficiency. Specifically, does the factorization increase the size of the models, since there are more parameters being optimized after the factorization. Additionally, does the improvement in loss landscape significantly improve the training time of coordinate-based networks across all applications. This is an important question for those who work in neural rendering (ex: with neural radiance fields), and this could be a simple but applicable solution for this.

clarity,_quality,_novelty_and_reproducibility: The paper is quite clear, and the theoretical and experimental results are of high quality and justify the claims made in the paper. The proposed method is related to many existing methods and is simple, but I find it to be useful and the benefits are justified very well and in a new way, resulting in a novel contribution. I believe the result is reproducible.
summary_of_the_review: This paper proposes a new factorization for fully connected coordinate-based network architectures, theoretically proves that it has some properties which are amenable to better initialization (resulting in better results), and experimentally validates that this is the case. I find this to be a convincing story, and this is a topic which is very interesting for those who work on applications involving coordinate-based networks. Thus, I believe this paper would have a high impact and recommend it for acceptance.

POST REBUTTAL UPDATE: After reviewing the authors' response, and the other reviews and responses, I am not inclined to change my score. The authors have empirically addressed the hyperparameter comment, which I believe was valid, and have shown that their factorization leads to improvements. The theoretical section has been improved with respect to prior work. I believe the improvements are noticeable, and I don't believe comments on vastly different architectures are relevant since the paper specifically focuses on coordinate-based networks.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes a method for improving and accelerating the training of coordinate-based networks by proposing randomized weight factorization. This factorization generalizes weight normalization, and can be applied in the coordinate-based network setting for signal memorization.

The reviewers raised a major concern on experiments:

(1) The proposed method slightly outperformed baselines in their investigated applications. However, considering the provided sample standard deviations, all the p-values for the corresponding two sample t-tests are large. Therefore, the improvements are not statistically significant.

(2) The proposed method is not generic, and cannot be applied to other popular architectures in as natural language processing and computer vision.
justification_for_why_not_higher_score: NA
justification_for_why_not_lower_score: NA
Decision: Reject