Online approximate factorization of a kernel matrix by a Hebbian neural network

Abstract: We derive an online algorithm for unsupervised learning based on representing every input $\mathbf{x}_t$ by a high dimensional vector $\mathbf{y}_t$ with pairwise inner products that approximately match input similarities as measured by a kernel function: $\mathbf{y}_s \cdot \mathbf{y}_{t} \approx f(\mathbf{x}_s, \mathbf{x}_{t})$. The approximation is formulated using the objective function for classical multidimensional scaling. We derive an upper bound for this objective which only involves correlations between output vectors and nonlinear functions of input vectors. Minimizing this upper bound leads to a minimax optimization, which can be solved via stochastic gradient descent-ascent. This online algorithm can be interpreted as a recurrent neural network with Hebbian and anti-Hebbian connections, generalizing previous work on linear similarity matching. Through numerical experiments with two datasets, we demonstrate that unsupervised learning can be aided by the nonlinearity inherent in our kernel method. We also show that heavy-tailed representation vectors emerge from the learning even though no sparseness prior is used, lending further biological plausibility to the model. Our upper bound employs a rank-one Nystrom approximation to the kernel function, with the novelty of leading to an online algorithm that optimizes landmark placement.

Reviews and author responses:

 With an increasing number of neurons perhaps the small value of λ actually causes more artificial correlation than a larger value (since the matrix is being inverted). This wouldn't be a problem with the pseudo-inverse in the Nyström method.

Additionally, the paper uses a heuristic of two times the number of input dimensionality for examples shown in Figure 6. Its not clear if the downstream task performance would also decay when too many neurons are used (without fixing the previously mentioned issue). 

The paper studies the linear similarity matching problem, which represents input $x_t$ by a high dimensional vector $y_t$ such that $y_s \cdot y_t \sim f(x_s, x_t)$ for a kernel function $f$. 

Main contributions of the paper:
- The authors propose an online algorithm for linear similarity matching, which can be interpreted as a neural network with Hebbian and anti-Hebbian connections.
- The authors provide experiments to demonstrate that the proposed method can aid unsupervised learning tasks. I like the authors' idea to use an online neural algorithm to approximate nonlinear kernel similarity. However some concerns are:

- I feel the authors need to argue more the contribution and importance of the proposed algorithm. Is there other online algorithms for similarity matching, and why one should choose this algorithms over others?
- In the experiments there is no comparison to demonstrate the advantages of the proposed algorithm over other algorithms. There is a comparison with the kernel PCA on the Half-Moon dataset, but the finding is that the proposed method's performance is not much worse compared to the kernel PCA.


Some Typos:
- page 2, 'which are defined by more a general nonlinear kernel similarity' -> 'which are defined by a general nonlinear kernel similarity'?
- Page 2, 'OUr work' -> 'Our work'.

---------------------- Score after revision -----------------------

Thanks the authors for adding the performance comparisons with other methods. I think the revised version is better than the previous one. However, I want to keep my original score, since there is some strange behavior of the proposed method in Fig 5 (as also mentioned by the authors): the error turns to increases with the dimensionality when the dimension is large. I think this behavior deserves more study and analysis. I feel the importance of the work is not well argued in the paper, and hope the authors could address that.

This paper studies the problem of feature mapping for reproducing kernels. The focus is when the data size is too large, the SVD for the kernel Gram matrix would be prohibitively difficult. This paper proposes to use the least-squares loss to achieve a low-rank approximation to the Gram matrix, to use Legendre transformation to disentangle quadratic terms in the least-squares loss, and to use a two-phase Hebbian parameter update rule to achieve the approximating feature map. strength: the idea is nice. After training, the obtained function maps data points directly to their images in the feature space with low computational complexity.

My concerns:
1. The one-dimension approximation (4) may lose precision. The introduced error could be significant. Users of the algorithm may not be able to estimate the introduced error. Users may end up having a totally different reproducing kernel if they shift to using the feature images.
2. The feature map is obviously not unique. For example, for any such map x--> y=F(x), and any orthogonal matrix Q, QF(x) is exactly another feature map as good as F(x). Consequently, you can not expect that W, q, or L in (8) would converge in the first phase of training.
3. Some minor mistakes in using the Legendre transformation: the sum in t on the right-hand side of (4) does not cover (1/2)q^2f(w,w). However, we see that the author(s) made it cover this term in (5) and (6). We see that it is easy to fix this problem by scaling the term (1/2)q^2f(w,w) by 1/T.
4. The optimization problem (8) is far from convex (especially in terms of W). Compared with matured randomized matrix decomposition techniques many with convergence guarantee and designed for reducing space and time complexity (e.g. Halko, N.; Martinsson, P. G.; Tropp, J. A. Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Rev. 53 (2011), no. 2, 217--288.) (these algorithms could be plugged into kernel-based empirical feature maps to achieve the same purpose as claimed in this paper), the convergence and precision of the proposed algorithm are both difficult to verify.

========================== after reading the authors feedback
Most of my points are well addressed. I am happy to raise the score to 8. I believe that combined with some regularity assumption on kernels, good theoretical results can be derived for the proposed algorithm as future work. This is a nice paper with a good idea. However, we believe that the introduced methodology is not fully developed.

 About point 1. I agree that the authors have provided excellent and comprehensive empirical comparisons. However, compared with existing randomized matrix decomposition algorithms guaranteed to converge with overwhelming probability, empirical comparisons in the current paper appear not satisfactory. This was my concern.

Kernel-based algorithms are reduced to classical linear models when linear kernels are adopted. For classical linear models, the dimension reduction algorithms are well developed and well understood.

The other three points I raised are well addressed.

This paper considers the problem of finding a low-rank approximation to a kernel matrix. In the low-rank approximation problem, one would want to find a low-rank matrix such that the Frobenius distance between the low-rank matrix and the original matrix is minimized. The paper shows that this optimization problem can be upper bounded by the cost function of a Hebbian neural network with a recurrent layer. Using some experiments the paper shows that this neural network can produce accurate low-rank approximations to some synthetic datasets as well as the MNIST dataset for the Gaussian and Arc-cosine kernels. The paper sells the result as an online algorithm for finding a low-rank approximation to a kernel matrix which is not quite the right term for their algorithm. In online algorithms, one typically can only have a single pass over the input dataset while the presented algorithm needs many passes over the input until the neural network converges. 
Moreover, even though the expression of the low-rank approximation problem in terms of a neural network is novel and interesting, it would be much more compelling if the paper presented some convergence analysis for the training of the neural net. In particular, it is not clear if SGD on this network would find the optimal solution or how many epochs are needed for training the model. Basically, the paper presents an algorithm but it does not analyze the algorithm in terms of convergence, generalization, or runtime.


Minor issue:
It seems to me that you need a factor T in the term -1/2 q_i^2 f(w_i,w_i) in Eq(5).


---------------------- Score after revision -----------------------

The revised version of the paper looks better than the initial submission and it has addressed my concerns. Therefore, I raise my score to 6. I generally like the idea of the paper and think it is novel to express the low-rank approximation problem in terms of a neural network but I have the following concerns about the presentation and contributions of this work. 

The paper sells the result as an online algorithm, while the presented algorithm needs many passes over the input until the neural network converges. It should be stated clearly that your algorithm requires multiple passes.
Moreover, I think it would be much more compelling if the paper presented some convergence analysis for the training of the neural net. In particular, it is not clear if SGD on this network would find the optimal solution or how many epochs are needed for training the model. Basically, the paper presents an algorithm but it does not analyze the algorithm in terms of convergence, generalization, or runtime.


 I want to thank the authors for the revision, which I have read and I think strengthens the work. Overall, I maintain my score of a borderline accept. It is an interesting work with useful conceptualization of the online kernel similarity learning. However, the failure to perform well with increasing number of neurons means the approach is not 'robust' to model misspecification. Nonetheless, I think the fact that the learned landmarks perform better than a k-means approach points to further improvement of the methodology. The paper is not clear whether the gap to the Nyström optimal can be narrowed with an online approach. This is where some more analysis and theory could be useful. Thus, I tend to agree with the another reviewer that the work is somewhat preliminary.

Minor points:
Abstract: "In this paper attempt " -> "In this paper, we attempt"

Introduction: 
"In this paper, tackle" -> "In this paper, we tackle"

"our upper bound with will lead to a " -> "our upper bound will lead to a "

Fig. 3 "<= 16" ->  "≤ 16"

"algorithm leanrs" -> "algorithm learns"

" Random Rourier method" -> " Random Fourier method"

"100 inits" -> " 100 initializations"

"We obesrve" -> "We observe"

"surpasses ours neural" -> "surpasses our neural"

It's not clear why the paper chooses the weight decay penalty for the classifier with the test set error. This is methodologically invalid strategy. A validation set or cross-validation set should be used. I understand that this may be for the best case scenario, but I think it is unwise to take these shortcuts. 

Please capitalize journal and conference venue names appropriately in the bibliography. 

Also check the bibliography in general as it is not proofread "nystr\” om" 

In appendix, the line after equation 24 has a missing reference. 

At bottom of page 13, the format is messed up. 

"he original paper" -> "the original paper"

For the random Fourier features, make sure the kernel size $\sigma$ is correctly related to the $\gamma$ used in the other methods. 
 

 Dear all, we appreciate the feedback and reviews! We have uploaded a revised paper https://openreview.net/references/pdf?id=gzwzyWCGxQ

**Overview of changes**
We have implemented 2 major changes
- Add comparisons (quantitative and conceptual) to existing methods for approximating kernel matrices (Nystrom and neural Fourier methods)
- Heavily discuss the ‘’biological networks for kernel similarity matching’’ work that was brought to our attention by the author Yanis Bahroun.
We have also fixed most of the minor concerns addressed by reviewers. We’ll now go section by section and discuss the changes.

**Title** 
- change to “Kernel similarity matching with Hebbian neural networks”
    - This title makes more sense in relation to previous work. Notable the kernel similarity matching problem was already defined in previous work, so we will use their terminology “kernel similarity matching”.

**Abstract & Introduction** 
- Rewritten to strongly emphasize
	1) this is is an extension of work by Yanis Bahroun et al. on neural kernel similarity matching
	2) this is strongly motivated by biological networks
	3) state that we compare to non-biological methods for kernel approximation

**Section II: Kernel similarity matching** 
- We have upgraded the lemma to a theorem and brought the proof from the appendix into the main text, as suggested by Reviewer 3.
- We have removed Fig. 1 to make space
- We have added the notation N,M,T to the old figure 2 (recommended by reviewer 3)
- We have fixed the (1/T) issue in Eq. 6 that was identified by Reviewer 1 and 4.

**Section III: Neural algorithm** 
- We state that convergence of our algorithm is treated as an empirical issue, but we use "two time scale" approach that has been empirically successful for training GANs

**Section IV: Experiments**
- merged previous experimental sections into one big section
-added comparisons to various Nystrom methods and to the random Fourier feature method for the half moons dataset
- replace the input-output scatter plots with more conventional “mean square approximation error vs. dimensionality” plots

**Section V: Related work**
- we have added a related work section to discuss at a conceptual level how our method relates to
	1) the neural random Fourier method
	2) non-neural, but widely used, Nystrom methods

**Section VI: Discussion**
- state the novelty of our work 
- state that this method is not as robust as the sampling methods that are widely used

 Hi,

First of all, I would like to reiterate that in my opinion this work is novel and complements a variety of existing work. I personally really enjoyed reading it. I think that this work also has a lot of potential in connection with Neuroscience. The SM framework has proved useful for modeling a variety of neural structures [1-2] and for unsupervised and supervised learning tasks [3-4] and this work certainly proposes a way forward. 

Question 1: 
In that work, I used kernel nonnegative similarity matching for a clustering task. How it was shown in the original work of Pehlevan et al. [5] the output is sparse but not exactly winner-take-all, thus when comparing to other methods, it is important to acknowledge that and when measuring the performance, I used the maximal response neuron as an assignment. 
It is in my thesis that I provided some quarantine and proof of concept for Kernel Unconstraint Similarity Matching, which is the offline objective that you also proposed. Your work is more exhaustive and the experiments you present very compeling. 

Question 2:
Yes, indeed it amounts to that. The doctoral thesis of [6] provides a great description of a variety of competing models that can be efficiently be trained online. The respective papers forming the thesis were cited in my work. 


[1] Lipshutz, David, et al. "A Biologically Plausible Neural Network for Multichannel Canonical Correlation Analysis." Neural Computation 33.9 (2021): 2309-2352.

[2] Bahroun, Yanis, Dmitri Chklovskii, and Anirvan Sengupta. "A similarity-preserving network trained on transformed images recapitulates salient features of the fly motion detection circuit." Advances in Neural Information Processing Systems 32 (2019): 14201-14212.

[3] Obeid, Dina, Hugo Ramambason, and Cengiz Pehlevan. "Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks." Advances in Neural Information Processing Systems 32 (2019): 15403-15412.

[4] Qin, Shanshan, Nayantara Mudur, and Cengiz Pehlevan. "Contrastive Similarity Matching for Supervised Learning." Neural Computation 33.5 (2021): 1300-1328.

[5] Pehlevan, Cengiz, and Dmitri B. Chklovskii. "A Hebbian/anti-Hebbian network derived from online non-negative matrix factorization can cluster and discover sparse features." 2014 48th Asilomar Conference on Signals, Systems and Computers. IEEE, 2014.

[6] Chitta, Radha. Kernel-based clustering of big data. Michigan State University, 2015.

 Dear reviewer,

Thank you for your review!

*The paper sells the result as an online algorithm for finding a low-rank approximation to a kernel matrix which is not quite the right term for their algorithm. In online algorithms, one typically can only have a single pass over the input dataset while the presented algorithm needs many passes over the input until the neural network converges.* We are revising to clarify our use of the term “online.” We just mean that the algorithm sees only one input pattern at a time. Sometimes this term additionally includes the setting in which performance is evaluated, for example cumulative loss in a single pass through the input patterns. Our algorithm is expected to converge in a single pass through an infinite stream of input patterns. If applied to a finite batch of examples, it is true that multiple passes would be necessary for convergence. We will state this clearly in our revision.

*Moreover, even though the expression of the low-rank approximation problem in terms of a neural network is novel and interesting, it would be much more compelling if the paper presented some convergence analysis for the training of the neural net. In particular, it is not clear if SGD on this network would find the optimal solution or how many epochs are needed for training the model. Basically, the paper presents an algorithm but it does not analyze the algorithm in terms of convergence, generalization, or runtime.*
The neural net performs stochastic gradient descent-ascent (GDA). We are revising the manuscript to cite more theoretical research regarding convergence of stochastic GDA.  

*Minor issue: It seems to me that you need a factor T in the term -1/2 q_i^2 f(w_i,w_i) in Eq(5).*
Thanks, we will correct this.


 Dear reviewer,

Thank you for your review!

**Weakness 1:** *This approximation yields something like the Nyström method. Yet no comparisons are made to baseline the approach in terms of Nyström landmark selection algorithms such as Musco and Musco's approach "Recursive Sampling for the Nyström Method" NIPS 2017.*
This is a great suggestion, and we are adding comparisons with several Nyström methods, including randomly sampled landmarks and k-means landmarks. Both of these methods can be implemented as online algorithms, and the latter in particular is very close to exact for Half Moons and MNIST. While the Musco and Musco algorithm has runtime linear in the number of points, their recursive sampling does not appear to qualify as an online algorithm (process a single pattern at a time)

**Weakness 2:** *The experimental results are limited but clear. Nonetheless, I'd like to see results of the performance metrics (similarity and downstream tasks) across different choices of neurons.*
Could you clarify what “different choices of neurons” means? Different numbers of neurons? Different kernel functions?

**Weakness 3:** *While the method concentrated on proper kernel methods (yielding positive semidefinite kernel matrices), it is not clear if the the PSD assumption is necessary. For example stochastic neighbor embedding (SNE) can use different kernel sizes for different input points, which doesn't maintain PSD nature. The restriction of f and their implications could be elaborated on.*

We can add text to the discussion. Essentially there is no guaranteed that our method is optimizing a lower bound if you do not use a kernel. Although it is possible this method could still generate useful representations if f is not a proper kernel.

**Weakness 4:** *In the conclusion the wording about the discussion of the relationship to Nyström is somewhat confusing. In particular, the statement "we start with a different objective" doesn't make it clear that this paper uses the same cost as the Nyström approach, but that Fu (2014) used a different objective, which is functional approximation in the RKHS.*

The reviewer is right that there is potential for confusion. We are making revisions to clarify how our bound is different from Nyström. 
Minor points: 

**Minor points:** we are working on revising these. For "nearly convex-concanve", it means the function is not necessarily convex concave everywhere, but is locally convex concave around the saddle point.

 Dear reviewer,

Thank you for your review!

**Response to concern 1:** Chklovskii, Pehlevan, and coworkers have written numerous papers on online neural net algorithms for linear similarity matching. These are important papers, but linear similarity matching is computationally limited. Therefore it is important to extend to the nonlinear case. Sengupta et al. (2018) have introduced a similarity threshold, a constant offset into the similarity matching objective function.

Yanis Bahroun introduced an online neural algorithm for kernel similarity matching. (He informed us on OpenReview; we were not previously aware of his work.) This uses random Fourier features (RFF) to approximate a shift-invariant kernel, and then applies linear similarity matching. The RFF algorithm has the theoretical advantage of becoming exact in the limit of infinite dimension, whereas our algorithm does not. The RFF algorithm is limited to kernels that are shift-invariant, while our algorithm is more general. 

We are carrying out empirical comparisons of our algorithm versus the RFF algorithm, and will add them soon to the manuscript. So far, we are seeing that our algorithm empirically outperforms RFF for finite dimensions that are fairly large. 

**Response to concern 2:**
Compared to linear similarity matching, our algorithm is shown to be better for downstream classification. Our algorithm is not meant to outperform kernel PCA, but deliver similar performance with an online neural implementation. However, we are working on experiments to compare with other non-neural methods just to better place this method in the context of previous works.


 Hello Yanis, thanks for the comment!

We apologize that we were not aware of your work. We are revising to cite your 2017 paper as having introduced neural algorithms for Kernel Similarity Matching, which simplifies our problem setup.

We are trying to understand the empirical analysis in Section 4.2 of your paper, and have a few questions. (1) After applying NCMDS to obtain a soft clustering, is that turned into a hard clustering by finding the maximum? (2) What exactly do you mean by “Nystrom-based spectral clustering”? Does that mean applying k-means to the representations from the Nystrom approximation?


 Dear reviewer, 

Thank you for your review!

**Response to point 1:** It is true that our algorithm depends on an approximation. To evaluate the precision of the approximation, we provide some empirical comparisons. Figures 4 and 6 show that the kernel similarities f(x_s, x_t) closely match the feature similarities y_s dot y_t for the Half Moons problem and for MNIST. Also, for linear kernels at least, this rank 1 approximation is sufficient (the optimal bound yields an inequality for the linear kernel).

**Response to point 2:** For the objective in Eq. 1, any orthogonal transform of the learned $y$ will yield the same energy. We are optimizing not Eq. 1, however, but the lower bound of Eq. 5. This lower bound in general breaks the continuous degeneracy when the kernel is nonlinear. 
If there is concern about convergence, it should be for a different reason. Our neural algorithm uses gradient descent-ascent (GDA), for which the convergence guarantees are not as strong as pure gradient descent or pure gradient ascent. We will expand our discussion of convergence, and citations of the relevant theoretical works on convergence.

**Response to point 3:** Thanks for pointing this out. We are revising and will fix this.

**Response to point 4:** We are revising the manuscript to be more explicit about our desiderata for algorithms. Our most basic requirement is that

(A) We want an algorithm that doesn’t have to explicitly compute every element of the similarity matrix. 

The reason for requirement A is that the NxN similarity matrix is huge when the number of input patterns N is large. The dimension reduction algorithms of Halko et al. (2011) require at least one pass through every element of the similarity matrix, violating requirement A. In Section 2.1, Halko et al. (2011) mention in passing the alternative sparsification and column selection approaches. These randomly sample elements or columns to yield a surrogate matrix that can be factorized. Superficially, these alternative approaches might seem to obviate the one pass through the similarity matrix. It turns out, however, that one pass is still required to compute the sampling distribution. Therefore, as far as we can tell, all of the randomized matrix decomposition techniques of Halko et al. (2011) violate requirement A.
Our algorithm satisfies requirement A, and in addition satisfies two other requirements:

(B) We want an online algorithm, meaning that a single input pattern suffices for each learning update.

(C) We want an algorithm that is implemented as a biologically plausible neural network. 

We were willing to sacrifice some precision for the sake of these requirements, and the empirical results show that the sacrifice is not so large in two example cases. 


 Kernel Similarity Matching was introduced in these exact terms in [1] and [2] for clustering and nonlinear dimensionality reduction. None of these two works were cited here. The main distinction, which is important, is that they performed a Nystrom-like method when we approximated the kernel with random Fourier features, which were discussed as being respectively data-dependent and data-independent approaches. The current work has some novelty with respect to the two papers I mentioned, but that were not cited in the present paper. 


[1] Bahroun, Y., Hunsicker, E., & Soltoggio, A. (2017, November). Neural networks for efficient nonlinear online clustering. In International Conference on Neural Information Processing (pp. 316-324). Springer, Cham. 
https://www.researchgate.net/profile/Yanis-Bahroun/publication/320165071_Neural_Networks_for_Efficient_Nonlinear_Online_Clustering/links/59d1e788aca2721f43697292/Neural-Networks-for-Efficient-Nonlinear-Online-Clustering.pdf

[2] Bahroun, Y. (2020). Exploration and extension of the similarity matching framework: feature learning, nonlinear methods and transformation learning (Doctoral dissertation, Loughborough University).
https://repository.lboro.ac.uk/articles/thesis/Exploration_and_extension_of_the_similarity_matching_framework_feature_learning_nonlinear_methods_and_transformation_learning/12683804

The paper describes an online stochastic optimization of a neural network to learn an embedding such that the inner-product between embedded points approximates a non-linear kernel based similarity. This can be a radial basis function network that approximates the embedding of a Gaussian kernel. It can also use a homogeneous kernel to approximate cosine-similarities. The network approach can be seen as a sort of Nyström approximation where each output dimension is associated with a different Nyström support point (landmark). The approximation cost is in terms of minimizing an upper bound on the squared error of the similarities. Experimental results show that the method is able to learn meaningful representations.  Strengths: 

The problem is interesting and follows a long line of kernel methods and scalable approximations. 

The paper's approach is well motivated. 

The paper is mostly clear and well-written. 

The results are illustrative and capture some of the nature of approximating non-linear similarities in low-dimensional space.

Weaknesses:
This approximation yields something like the Nyström method. Yet no comparisons are made to baseline the approach in terms of Nyström landmark selection algorithms such as Musco and Musco's approach "Recursive Sampling for the Nyström Method" NIPS 2017. 

The experimental results are limited but clear. Nonetheless, I'd like to see results of the performance metrics (similarity and downstream tasks) across different choices of neurons.

While the method concentrated on proper kernel methods (yielding positive semidefinite kernel matrices), it is not clear if the the PSD assumption is necessary. For example stochastic neighbor embedding (SNE) can use different kernel sizes for different input points, which doesn't maintain PSD nature.  The restriction of $f$ and their implications could be elaborated on. 

In the conclusion the wording about the discussion of the relationship to Nyström is somewhat confusing. In particular, the statement "we start with a different objective" doesn't make it clear that this paper uses the same cost as the Nyström approach, but that Fu (2014) used a different objective, which is functional approximation in the RKHS. 

Minor points: 

page 2, " OUr work" 
page 3, Aspects of the proof of Lemma 1 could be incorporated into the text.
 What is meant precisely by "nearly" convex-concave, and is it due to the nature of the Legendre transform?
page 3, comma after 2 will help the reader parse the double verbs in "is derived as an upper bound to Eq. 2 is thus" 
Figure 2 could have the variables from the paper $M,N, T$. Their choice makes them a bit hard to remember.
page 4, "gaurantee"
page 5, "Hebbian rules are been defined" 
page 5, the number of training iterations could be converted to epochs? (how many times the training data set has been seen)

 The paper is a solid approach to show feasible online update rules for learning representations that capture non-linear similarity.  Not all aspects of the theory or results are explored.  Key comparisons versus other scalable techniques Nyström are not made, perhaps to focus on the online nature. Nonetheless, some more comparisons and elaborations on the nature of similarities could make this approach more useful in practice. 


Meta-review: This paper studies the problem of kernel similarity matching using Hebbian neural networks. Specifically, the authors propose to compute the approximate feature map for the kernel using the least square loss function, Legendre transformation, and Hebbian parameter update rules.

Reviewers generally agree that the proposed method is interesting. However, there are major issues with the current manuscript, both theoretically and empirically. Theoretically, there is no guarantee for the convergence of the method. In fact, the non-convergence of the approximation error, when the dimensionality (number of features) increases, indicates that the proposed method is not consistent, in contrast with other methods such as Kernel PCA or Nystrom approximation-based methods. As observed by the authors, this may be related to the unstable convergence of the stochastic gradient descent ascent optimization procedure. For consistency,
the approximation error needs to approach zero as the number of features becomes large. 
Overall, empirical results compared with existing methods are not  satisfactory. As the authors themselves point out in their discussion, "this method does not provide the same sorts of theoretically guarantees or empirically observed robustness of sampling based methods".

The authors are encouraged to take  reviewers' comments and suggestions into account to improve their current work.

Decision: Reject