Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes new algorithms for CCA and Partial Least Squares problems based on game-theoretic formulations, following previous work by Gemp et al. The proposed algorithm relaxes some constraints and works well empirically.
strength_and_weaknesses: While the proposed algorithm seems to be promising empirically, its description and motivations are unclear and under-developed in the manuscript.

The key elements that are presented to motivate the algorithm are its lack of hard constraints, which are supposedly "softly" enforced by Lagrange multipliers, and reduced number of hyperparameters. Yet, in the presentation of the algorithm, I see no description of Lagrange multipliers, or soft-enforcement of constraints, and at least for the full-batch version, Algorithm 1 in [Gemp et al](https://arxiv.org/abs/2206.04993) seems to have only the step-size as a hyper-parameter, just like the present paper, and the projection on the sphere doesn't seem like a big issue in practice.

Overall, I encourage the authors to provide a clearer and more formal description of the proposed algorithm, and its motivation (ideally in a way that doesn't involve stop-gradients as currently done in eq.(12)), and a precise comparison to previous work.
clarity,_quality,_novelty_and_reproducibility: Difficult to read, and little difference with previous methods.
summary_of_the_review: The paper is not ready for publication in its current state. I encourage the authors to state the motivations more clearly and improve the presentation of the proposed method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the application of EigenGame formulation to solve the Generalized Eigenvalue Problem (GEPs). The previous work had defined the implementation $\alpha$ and $\mu$ - EigenGame, while this one defines the $\delta$- EigenGame version. EigenGame is a reduction of the problem to compute eigenvalues to the problem of computing a Nash equilibrium in a ''quasi''-collaborative/potential game.
In the previous line of work, in order to compute this strict-pure Nash equilibrium, a Riemannian Gradient Descent is employed, (as an analogue of Best-Response in a potential game). This work describes an alternative description of the game, which implicitly incorporates the constrained optimization via Lagrange multipliers.
strength_and_weaknesses: This question falls within a general research thread that has recently attracted considerable attention in the literature. To start with a positive point about the current submission, the current authors proved that, indeed, the unique equilibrium of the new game is again the solution of the Gen. EigenProblem. In other words, they proved that the reduction is meaningful . I will return to this general positioning later. The main contribution of the paper is to explain that experimentally a Lagrangian Multipliers usage in the optimization process can accelerate the Game Dynamics Process.

An objection that I raise collectively with the ''EigenGame'' reductions for all versions, is the inclusion of the stop-differentiate operator, introduced by Gemp et al from the alpha-version. Under this way, we lose completely the ''game-theoretic'' intuition. Actually, it would be interesting to understand even if the utility function is smooth/continuous. An important weakness of this series of work is the finite sample complexity. See the discussion of the discussion at the initial work of \alpha-EigenGame.
clarity,_quality,_novelty_and_reproducibility: The paper could be better written in the experimental section.
To be more precise, even if I am theorist, I would vote positively for an experimental mainly paper if the majority of the main draft has been devoted to explain  (i) the validity & generality of the experimental setup (ii) the intuition between the advantages of the model.

About the novelty, I expect to earn more intuition about the importance of the application in Deep Learning framework. I invite the authors to be more explanatory of the improvements and the necessity of Deep CCA and the potential impact of an ADMM optimization-style for EigenGame.


summary_of_the_review: Currently, I feel that the work is below the threshold. However, at the first glance, it is clear that the paper provides an interesting implementation details about simplifying difficulties of previous versions of EigenGames. However, as standalone work, it seems a bit incremental  improvement. In the case that authors would not like to focus on the theoretical aspect of the optimization (which is fare), it is necessary to explain the importance of the applications and the burdens that are overpassed thanks to this EigenGame version.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents a top-k eigenvalue solver for stochastic eigenvalue problems. The proposed approach undertakes an optimization viewpoint using Lagrange multipliers. The effectiveness of the proposed scheme is validated through numerical experiments on simulated datasets.
strength_and_weaknesses: 
Strengths:
-) The discussion of related work is sufficient. 
-) Numerical experiments confirm that the method is competitive from a qualitative perspective.
-) The proposed scheme appears to be well-fit for parallel computing, although this direction is not explored in the paper.

Weaknesses:
-) The paper is poorly written and some concepts, like the ones presented in Section 2 before Section 2.1 are not explained well (also, Eq. (13) is empty). Eqs (16), (17) etc. are poorly motivated, and it is not clear how they appear.
-) The novelty of the algorithm seems to lack. On several occasions, the paper feels to step on the work by Gemp et al. and the authors try hard to overstate the potential differences; to the point that it disrupts the natural flow of the submission. 
-) Unless I missed it, there are no running time comparisons between different schemes. Note that faster convergence does not translate to a better algorithm if there are no running time advantages (or maximum qualitative accuracy captrured).

Minor comments:
-) The \hat operators below Eq. (15) are misaligned.
clarity,_quality,_novelty_and_reproducibility: All good.
summary_of_the_review: My main concerns are: a) novelty, and  b) advantage over existing methods since no running-times are presented.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors generalized the Hebbian rule through the corresponding Lagrangian of the Rayleigh Quotient form, which leads to a stochastic algorithm for generalized eigenvalue problem, including canonical correlation analysis and partial least square. 

The major issue of the paper is the novelty. These generalized Hebbian rule has been proposed in kernel community [1], in which the finite-sample convergence has already been proved in a more general setting in RKHS. The proposed ignored this important literature. Despite the missing reference, the algorithm is only a small modification of Hebbian rule in [2]. 

Another issue is the clarity of the presentation. All the reviewers raised the confusion about the algorithm description and motivation.  

In sum, the paper is not ready to be published yet in current version.

[1] Xie, Bo, Yingyu Liang, and Le Song. "Scale up nonlinear component analysis with doubly stochastic gradients." Advances in Neural Information Processing Systems 28 (2015).
[2] Terence D Sanger. Optimal unsupervised learning in a single-layer linear feedforward neural network. Neural networks, 2(6):459–473, 1989.
justification_for_why_not_higher_score: 
The proposed algorithm has been investigated in literature before in a more general setting.
Decision: Reject