Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extend the Tensor programs that was initially proposed for real valued neural networks to complex valued neural networks (CVNN), and as a result, providing the foundations to calculate the the neural tangent kernel (NTK) of any CVNN architecture. Later, they show that the NTK of any CVNN with a certain type of activation functions ( that contains majority of common used activation function ) is as expressive a real valued neural network by comparing their limiting NTKs, as a result CVNNs can not offer anything more that real valued neural networks. They also provide some examples of activation functions that differs CVNNs from their real valued neural networks.   
strengths_and_weaknesses: Tensor programs is a series of papers that build the foundations for understanding wide neural network. It was initially proposed for real valued neural network to calculate the NTK of any architecture. This work extend it to CVNNs and thus provide a theoretical foundation to analysis CVNNs. The results of this theoretical foundation, other than calculating the NTK of any CVNN can be very valuable, one that is discussed in this paper on conditions on activation functions that distinguishes CVNNs from their real valued neural networks. The foundation that is built in this work also has the potential to be used to extend the theorems to analyze CVNNs outside of NTK regime where weights move from initialization like Tensor plus programs [1]. 



[1] Yang, Greg, and Edward J. Hu. "Feature learning in infinite-width neural networks." arXiv preprint arXiv:2011.14522 (2020). 
questions: How would this results be affected if we replace the output layer instead of Real(WH), we use abs(WH)? can tensor programs still be applied to these kind of networks? how does it affect the results? 
limitations: It would be very useful to see the of effects of choosing different activation functions on performance of CVNNs and real neural works in real world datasets. How can zrelu be superior over other cost functions? What are the task that CVNNs are better than their exact real valued counterpart? A few carefully experiments on real valued neural networks can to answer these question can significantly boost the quality of this paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Complex-valued infinitely wide neural networks are considered in the paper. Through studying the kernel limit of complex-valued networks, the authors showed that real-valued backpropagation reduces complex NTK to a real NTK, under certain conditions on the activation function. Interestingly, many of practically used activation functions satisfy the condition, and therefore, using real-valued backpropagation may not be suitable to leverage all the potential of complex-valued networks.
strengths_and_weaknesses: The theoretical findings in the paper are interesting and intriguing. Complex-valued neural networks are relatively less studied, and whose theoretical understandings are lacking. This paper contributes towards this direction and the reported results read sound and correct. Moreover, there are empirical studies corroborate the theoretical findings. The organization of the paper is good and most of the results are easy to understand.

I will discuss some weaknesses of the paper in the limitation section.
questions: I am interested to know if complex-valued neural networks can beat the state-of-the-art real-valued neural networks in applications, such as computer vision and time series prediction.

Is there any concrete motivation of studying the infinite width limit of complex-valued neural networks? As mentioned in the introduction, complex-valued neural networks have richer representation capacity and faster learning. Is overparameterization in complex-valued neural networks necessary for a good performance?

Is it possible to suggest a proper backpropagation method based on the theoretical findings in the paper in order to further leverage the potential of complex-valued neural networks?
limitations: The major limitation comes from the NTK analysis. In practice, neural networks are often trained with a relatively large learning rate for a sufficiently large number of iterations. Therefore, the working regime of neural networks is actually not NTK. The theoretical findings undoubtedly enhance the understanding of complex-valued networks; they may have limited implications for practice. For example, in the infinite-width limit, real-valued backpropagation may not be appropriate. However, in practice, the theory does not apply due to aforementioned reasons. Whether using real-valued backpropagation is suitable remains open.

I am fond of the theoretical findings in the paper. But the practical value of the paper is rather limited to my knowledge. Therefore, I have a neutral initial rating of the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper derives the equation for the neural tangent kernel (NTK) for complex-valued neural networks trained via real-valued backpropagation (backpropagation on the real and imaginary components separately). Their main theoretical result shows that, under common choices of activation functions, the NTK in the infinite-width limit is the same as that of a real-valued neural network.
strengths_and_weaknesses: This is an extremely interesting paper for the community working on complex-valued neural networks. The results are easy to follow and, by any point of view, extremely interesting.
questions: - Please provide a proofreading because the paper has several typos.
- A small introduction to questions of holomorphicity or complex backpropagation could be helpful to broaden the understanding of the results.
limitations: NA
review_title: null

Meta-review: metareview: This paper extends Yang's Tensor Program paradigm to the setting of complex-valued neural networks and shows that under certain conditions the infinite-width training dynamics under real-valued backpropagation are equivalent to those of real-valued networks. Numerical evidence supporting the main conclusions is presented and a short discussion argues for the necessity of backpropagation algorithms specifically designed for complex networks.

Even though the idea is not entirely novel, the proof that real-valued backpropagation is unsuitable for complex-valued neural networks will likely surprise many readers, and it is useful to present a solid theoretical explanation, even if it only holds in a certain infinite-width regime. The extension of Tensor Programs to complex networks is also a nice result in itself, if not groundbreaking. The reviewers generally appreciate these strengths of the paper.

The paper would be more impactful if further effort were devoted to the practical implications of the results, including a direct comparison with a backpropagation method designed for complex networks. It would be particularly useful to characterize the types of functions that are more readily learned with a method based on, e.g., Wirtinger calculus.

In light of the the above strengths and weaknesses, this is a borderline paper. Ultimately, I believe it falls just above the threshold and I recommend acceptance.
confidence: Less certain
Decision: Accept