Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning
Authors: Andrew Bond
Zafer Dogan
Venue: NeurIPS 2024 poster (2024)
Keywords: Training Dynamics
High-Dimensional Analysis
Scaling Limit Analysis
Subspace Learning
Primary area: learning_theory

Abstract: Subspace learning is a critical endeavor in contemporary machine learning, particularly given the vast dimensions of modern datasets. In this study, we delve into the training dynamics of a single-layer GAN model from the perspective of subspace learning, framing these GANs as a novel approach to this fundamental task. Through a rigorous scaling limit analysis, we offer insights into the behavior of this model. Extending beyond prior research that primarily focused on sequential feature learning, we investigate the non-sequential scenario, emphasizing the pivotal role of inter-feature interactions in expediting training and enhancing performance, particularly with an uninformed initialization strategy. Our investigation encompasses both synthetic and real-world datasets, such as MNIST and Olivetti Faces, demonstrating the robustness and applicability of our findings to practical scenarios. By bridging our analysis to the realm of subspace learning, we systematically compare the efficacy of GAN-based methods against conventional approaches, both theoretically and empirically. Notably, our results unveil that while all methodologies successfully capture the underlying subspace, GANs exhibit a remarkable capability to acquire a more informative basis, owing to their intrinsic ability to generate new data samples. This elucidates the unique advantage of GAN-based approaches in subspace learning tasks.

Exploring the Precise Dynamies of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning
Andrew Bond KUIS AI Center Koç University abond190 edu.
Zafer Dogan
MLIP Research Group, KUIS AI Center Electrical and Electronics Engineering Koç University zdogan@ku. edu. edu. tr
Abstract
Subspace learning is a critical endeavor in contemporary machine learning, par- ticularly given the vast dimensions of modern datasets. In this study, we delve into the training dynamics of single-layer GAN model from the perspective of subspace learning, framing these GANs as novel approach this fundamental task. Through rigorous scaling limit analysis, we offer insights into the behavior of this model. Extending beyond prior research that primarily focused on se- quential feature learning, we investigate the non- sequenneltial scenario, emphasizing the pivotal role of inter-feature interactions in expediting training and enhancing performance, particularly with an uninformed initialization strategy. Our investi- gation encompasses both synthetic and real-world datasets, such as MNIST and Olivetti Faces, demonstrating the robustness and applicability of our findings to practical scenarios. By bridging our analysis to the realm of subspace learning, we systematically compare the efficacy of GAN-based methods against conven- tional approaches, both theoretically and empirically. Notably, our results unveil that while all methodologies successfully capture the underlying subspace, GANs exhibit a remarkable capability to acquire a more informative basis, owing to their intrinsic ability to generate new data samples. This elucidates the unique advantage GAN-based approaches in subspace learning tasks.
Introduction
Subspace learning a widely explored task, especially with the growth dimensionality modern datasets. It is important to identify meaningful subspaces within the data, such as those determined by principal component analysis (PCA). However, due to the high dimensionality of the data,
common employ online methods such Oja's method [1] and GROUSE (2). Meanwhile, Generative Adversarial Networks (GANs) 3 primarily used as generative models, have also
demonstrated the ability to learn meaningful representations of data 44 [5) Inspired by this, explore how single-layer GAN models can be viewed as a form of subspace learning.
We seek to improve the understanding of GAN training by relaxing some common assumptions made in previous analysis of GANs [6]. Specifically, we focus on the training dynamics of the gradient-basee learning algorithms, which can be converted into continuous-time stochastic process characterized by an ordinary differential equation (ODE). Furthermore, the dynamics of the model weights form stochastic process modeled by stochastic differential equation (SDE). Understanding these two equations provides the relevant information to understand convergence behaviour of the
"Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

training. We extend previous work to discriminators with same dimensionality the generator. Finally, we discuss the new training outcomes that can arise in the nigher-dimensionaliin case.
Our work explores what happens when the training switches from sequential learning in the single- feature discriminator case [6], the non-sequential (multi-feature) learning of our discriminator. We show that the non-sequential learning of features not only allows for faster learning and convergence, but also a higher maximum similarity with the true subspace compared the sequential case, when everything else is kept the same. This shows that contrary to the general approach of making discriminators much weaker than generators, it is still possible to use powerful discriminator. In fact, doing so can lead to much faster training with better performance, through careful choice of learning rates.
We further show that our new framework can be used to analyze the cases where we assume different dimensionalittes between the true subspace, fake subspace, and discriminator. Through the of simple uplifting trick on the relevant Grassmannians, we are able to extend our analysis to arbitrary dimensionalities. To understand how GANs compare with existing subspace learning algorithms, we provide both theoretical and empirical comparisons with existing such algorithms. We see that the features learned by GAN model more meaningful and represent the data better as compared to Oja's method, due to the requirement of being able to generate new data from the underlying data distribution.
Finally, we test our approach using two prominent roal-world datasets: MNIST and Olivetti Faces, comparing our method against sequential discriminator, and show that all of the key insights gained through the theoretical analysis are visible training on this dataset as well. This shows that our analysis has very practical applications, and lead to interesting research directions exploring these ideas in more powerful GAN architectures. This testing is additionally done on the case where we assume different dimensionalities for each component of the model, showing that the results are as expected. We release all our code at SolvableMulls e:MulliFFaa
Overall, our contributions are as follows:
1. We investigate the distinctions between multi-feature and single-feature discriminators and fully characterize the learning dynamies through a rigorous scaling limit analysis.
2. We introduce a novel method for analyzing cases where the true feature dimensionality is unknown, enabling broader future analyses under uncertain conditions.
3. We position the feature GAN models and compare against existing algorithms, both theoretically and empirically.
type of subspace learning algorithm,
4. We further validate our findings on image datasets (MNIST and Olivetti Faces), highlighting the practical implications of our insights for real-world applications.
2 Related Work
2.1 Training dynamics for GANs
Much of this work is inspired by Wang et al. [6,, which was one of the first to undertake this task. However, in this case, they use single-feature discriminator, and show that the choice of learning rates relative to the strength of noise is what determines the outcome of the training process: convergence, oscillations, or mode-collapse.
There have been further attempts to understand the convergence of GANs through other approaches, not just the dynamics of the gradient- optimization. Heusel et al. 7 showed that under reasonable assumptions on the training process and hyperparameters, using a specific update rule will guarantee that the GAN converges to local Nash equilibrium. Mazumdar al. [8] introduced a new learning algorithm under which the local Nash equilibria are the only attracting fixed points, so that the training algorithm tends to move towards these points. This type of analysis is very similar to our approach, focused on understanding fixed points. Other types of models have also been analyzed in the high-dimensional regime, such as linear VAFs (. two-layer neural networks solving a regression task [10), and two-layer autoencoders [T]. A review of these methods can be found in [2]

---

2.2 Subspace learning
Subspace learning is a heavily explored field, with many algorithms. However, when the noise of the data has con-zero variance, most approaches fail, and the general technique used to solve the problem some type of online PCA-based method. In Wang et al. [13], similar analysis of dynamies through ODEs is performed for multiple algorithms which learn subspaces with non-zero variance of noise. This analysis allows for steady-state and phase transition analysis these algorithms. Balzano et al. [14] presents survey of different online PCA algorithms used for subspace learning, in the case where only some of the data is visible at each timestep, and discuss how possible to find unique subspace of given rank which matches all the provided data.
3 Background and problem formulation
3.1 True data model
Our data Vu is drawn from the following generative model, known as a spiked covariance model 15
F Uc, + V
Here, E Raxd is the true subspace we wish to learn represented as an orthonormal basis, C2 R
zero-mean random vector with covariance matrix A, ag R is standard Gaussian vector, and er represents the noise level.
The spiked covariance model is very widely studied, due the non-triviality of learning U whenever Wr > The key property of this model is that the top d eigenvectors the data covariance Ely given by the columns of U. If there exists a strict eigengap between the top d corresponding cigenvalues and the other eigenvalues, then the reconstruction loss function is proven to have as a global minima [16].
3.2 Online subspace learning algorithms
Subspace learning is very important task in machine learning, most commonly performed by algo- rithms such as PCA or ICA. However, these approaches involve costly operations such as calculating covariance matrices or calculating matrix inverses, infeasible in high dimensions. Therefore, is very common to use an online version of these algorithms,, processing samples one at time.
Online subspace learning algorithms typically fall into two categories: algebraic methods and geometric methods. Algebraic methods are based on computing the top eigenvectors of some representation of sample covariance matrix. Assuming strict eigengap, the top eigenvectors will yield the true subspace. Meanwhile, geometric methods optimize a certain loss function over some geometric space (Fuclidean space Grassmannian manifold). We review two subspace learning algorithms here, Oja's Method [1] and GROUSE (2]. While GROUSE was introduced for the missing data case, can be used for full data too. For further details about these algorithms and their categorization, we direct the reader to [14]
However, suggest third category of online subspace learning algorithms, which call the generative methods. Such methods, including single-layer GANs, do not have information about the specific task, and instead aim to learn the data simply by sceing the data and attempting to generate data from the same distribution.
3.2.1 Oja's method
Oja's method 1 is a classical algebraic approach to online subspace learning. Given an orthonor- malized initial matrix Xg, we perform the following update every timestep given a data sample y
X II [Xa + ry Xs]
Here, IT is an orthonormalization operator, and is the learning rate.
3.2.2 GROUSE
GROUSE 2 performs gradient descent on the Grassmannian manifold, which guarantees orthonor- mality of the updates. In the full data case, again start with an orthonormal initial matrix Xo, and
3

---

at each timestep, given data sample yk, our update is:
Xi+1 X + (cos 02 - 1) Pk
W
rk
W
+ sin 0x
PR
WEll
|
[Wx]
Here, 0g = T|| Palls We = arg min 1ly:
X₁ w/ 12 PA = X Wg. Fg Yk
and T is our P&
learning rate.
3.3 Generative Models
Here, we focus specifically on GANs. A GAN model seeks to learn a representation of the underlying subspace through the use of two components: a generator and discriminator. The generator learns the subspace by trying to generate new samples from the subspace, while the discriminator acts as classifier, attempting to distinguish data from the true subspace from data produced by the generator.
Note that measuring performance through cosine similarity measure the generalization performance of the generator, as it doesn't depend on any specific instance of generated data and instead provides concrete measure of how similar the generated data will be.
actually be viewed
way to
3.3.1 Generator
We assume that the generator also follows a spiked covariance model:
yx = V&C₂ + V
(4)
However, we do not assume that 1jG 17r, that the covariance of čk. A, is the same as A. The goal of the generator is to learn Vg.
3.3.2 Discriminator
The learning in the GAN model critically depends on the choice of discriminator, which aims to separate the data from the true and generated subspaces.
The most common approach when training GANs is to use a discriminator that is weaker than the generator. If the discriminator is too strong, then it will easily learn to distinguish between true and generated samples, leading to vanishing gradients for the generator and thus preventing learning. However, weak discriminator results sequential learning, where the generator only able to learn a subset of the features at a time. In multi-feature cases, this will lead to very slow learning.
Motivated by this, we seck to analyze model in which the discriminator has the same strength the generator. Thus, we let W e xx and define the discriminator as
D(y; W) = D(y W)
(5)
where D R. -> some function (see the assumptions below). Since this discriminator is able to focus on all the features at once, this means the generator is also able to learn every feature at once. This is in contrast to the single-feature case (where W IR") analyzed previously. While this is strong assumption the discriminator, we show below how this assumption can be relaxed.
3.3.3 Training procedure
GAN training modeled a two-player minimax game, where the discriminator attempts to maximize some loss function and the generator attempts to minimize it. This is used as way to learn a "surrogate" subspace which represents the true subspace. Therefore, the GAN model can be seen as a form of subspace learning, except that the focus is on generating new samples from the subspace.
Specifically, let L(y. y; W) be loss function depending the discriminator weights, and true and fake samples. denotes the true distribution and I denotes the generator distribution, the minimax game can be represented as
min max E W).
Following the approach of Wang et al. (6] and order to compare the sequential and multi-feature cases, use the following loss function:
A (r
A + +
C 9: W) F(D(y" w)) F(D(S
(6)

---

Here, F, F are functions affecting the outputs of the discriminator, H is an glement-wise function used for regularizing the weights of the generator and discriminator, and À controls the strength of the regularization. As À - oc, the matrices V, W will become orthonormal.
The standard approach solve this minimax game is using stochastic gradient descent (SGD). At timestep k, given a sample yą from the true subspace and a sample yx from the generator subspace, we perform the following updates:
V Ve+1 = Vv -v, f(y: W&),
n
WE+1 = WR + Vw, C(y Wo)
Here, r denotes the learning rate of the discriminator, and i denotes the learning rate of the generator. Note that while it is common to use batch of data at time when using SGD, we focus on single element at a time order to simplify all the analysis.
Development of ODE
Similar to [6]. make the following definitions:
Definition 4.1. X = [U, VR. Wa] E Rax3d is called the microscopic state of the training process
time k.
Definition 4.2. The tuple (Pxx Qg. RR. Sa. Z } is called the macroscopic state of X₃. time k. where W.W. We S V.Vi and Z₁ == W We. The macroscopic
Ps U V.. V.. Q₂ U Wk. R₂ V state be written in matrix notation M X Xx, in which we get
I
P
Q
M
PT
S
R
(8)
Q
R
Z₁
4.1 Macroscopic dynamics
To analyze the macroscopic dynamics, we reduce to special case, which leads slightly modified set of the assumptions from Wang et al. [6]
(A.1) The sequences ck2 Cj are i.i.d. random variables with bounded moments of all orders, and } is independent of {.x}
(A.2) The sequences { }. { are both independent Gaussian vectors with zero mean and covariance matrix I..
(A.3) ((A) = log cosh A - D(x) = ||||| and F(x) = F(x) = 2. We note that the first derivative of H exists, the first four derivatives of F(D(.)), F(D())) exist, and all the derivatives are uniformly bounded. Thus, our choices satisfy the conditions assumption
(A.3) from Wang et al. (6).
(A.4) Let [U, Vo. Wol be the initial microscopic state. For 1,... n, we have E. EI Volf, + [Wolf.] < C/n², where C' is some constant not depending on n.
(A.5) The initial macroscopic state M, satisfies E M. Mn S C/VR, where Mÿ a determin- istic matrix and C' is some constant not depending on n.
(A.6) The columns of the discriminator matrix W are orthonormal, that W w = la.
Assumptions (A1) and (A2) are the usual i..d assumptions common in machine learning. (A3) is important for deriving the update equations. (A4) and (A5) are used guarantee that the macroscopic state can converge. Our assumption (A6) of orthonormal discriminator matrix allows us to simplify the equations since the Z matrix of the macroscopic state is always just Lg.
Under these assumptions, as well letting À oo, we obtain modified Theorem 1 from Wang al. 6 specifically considering the reduced case of equation (13). Note that our choice of F, F, D means that equations become dimiraay-dimeesionaal version of the original equations.
5

---

Theorem 4.3. Fix T >> 0. Under Assumptions (A.1) (A.6), it holds that
C(T) Vn
max OSASAT
MA M
(9)
where C(T) is some constant depending T but not n, and M(t) R+ (0)
3ddd3d is
deterministic funtiiooo Moreover, M(t) is the unique solution of the following ODE:
d P: =(0,R) P.L.),
d 2. =(AQ, PARR, H.Q.)
dt
d
R: T(PE AQ: S,AR, + H.R,) + +F(A L,)RR
(10)
dt
d S. F(R.R, AR.R. S.L. L.S.), dl dl
d
-
Zz =0
dt
with the initial condition
M(0) = My. where
L = - (R He = (1
THE 2
R AR; (1
TIT
AQ,
TG + r
(11)
2
2
sketch of the proof of this theorem can be found in Appendix proof of the original theorem in 6
The proof closely mirrors the
4.2 Microscopic dynamics
The microscopic dynamics are concerned with how the terms U, V, W change over time. Following previous work, we consider the empirical measure
n 8((1, v, w] Vn] Vala,, (We]e.))
N (U, V, W)
Σ n
(12)
where 8 is the delta measure. This is a discrete-time stochastic process, which can be embedded in continuous time as f4 uk. with k = [nt]. Then, as n + co, this process converges to deterministic process yr, which is the measure of the solution of the SDE
dü, = 0, di, = FW,, + Leve)dt,
(13)
dw; Tuū (u"AQ: AQ, wh; dt + AdB.,
where A is some diffusion term, negligable due to our assumption on the discriminator (A.6). From this equation and the convergence of the measure, we can obtain the following weak PDE
d dt (pr (ūū v, w)) = (me, (w:AR: + Love Vep) + (me: i AQ, + wh,
V
(14)
where yo is bounded, smooth test function. The ODE in the main theorem can be derived from this weak PDE.
Simulations
In order to demonstrate that the ODE properly represents the training dynamics of the GAN model, we first perform simulations and show that the empirical results match the ODE, seen in Figure I To understand how the training dynamics change based on the generator learning rate, we fix the discriminator learning rate as r = 0.2 and fix the generator learning rate 7 = 0.04. We show the results on different noise levels. In all cases, we let A Ā = diag([v3 v5)).
We set Pu = Qu 0.1 + I, and we ensure that the empirical setup is initialized with exactly matching P and Q values. We note that the ODE will never learn when the initialization exactly 0, and so we must provide some level of similarity to start training. However, this is not very restrictive, as our experiments show that even random matrices will have approximately 0.001 / for both P and Q which is sufficient to escape the fixed point around O.

---

1.0
Cosine Similerity 2
Similority
Cosine Simillarty
Similarity
44
Figure 1: ODE results for learning rate 0.04, 0.2 and four different noise levels, with The columns represent 7/G 1/r 2, 1,3, 4 respectively. At = 5 or higher, the generator is unable to learn anything. all cases, the green and red represent the two diagonals P. and the blue and yellow represent the two diagonals of Q. We see that the simulations do match the predicted ODE results.
5.1 Off-diagonal simulations
key insight found from the multi-feature discriminator that the interaction between different features can help learning. When the macroscopic states are initialized to non-diagonal matrices, we see that the dimension with smaller covariance is actually able to attain better results and reach similar cosine similarity to the dimension with higher covariance. Such an outcome is not possible in the sequential learning regime, due to the lack of interaction between features. In sequential learning, features are learned one at time, and once feature has been learned, the training will focus different feature instead. This phenomenon can be seen in Figure 2 showing that the off off-diagonal initialization allows for not only faster training (which also happens in the diagonal initialization case), but also higher steady-state values compared to the sequential learning case. We are unable to provide a detailed characterization of these fixed points, neat closed-form solution cannot be obtained.
Coainn Simileity
$7903
Figure ODE results when initialized with off-diagonal entries. We focus on the case GJG fr as that noise level is seen above to be ideal for learning. Additionally, in all cases, 7 = 0.04, 7 0.2. The solid lines are with our approach, while the dashed lines are using the discriminator in Wang et al. use an initialization of 0.1, 0.01, 0.001, 0.0001 for each component of the
(6). From left to right, macroscopic states. can be seen that our approach outperforms the single-feature discriminator in every case, with the gap becoming larger the initialization approaches O.
Unknown number of features
While this type of analysis can provide interesting insights, it has a very restrictive assumption that we know the number of features d. This done so that the macroscopic states are welledefined. However, we now seek to extend this analysis to the case where the true subspace has d features, the generator subspace has p features, and the discriminator learns g features, where we do not assume that d = q. While this analysis can be performed under any assumptions on the relative size of d. p. and g, we focus on the single case d < 4 n.
L
To simplify the demonstration of this approach, we make the assumption that U =
so that U
contains the first d standard basis vectors. introduce the idea of uplifting (inspired by the work in [77 the matrices U, W to the dimensionality of V.
7

---

12
10:
Oja
GAN Multi-feature
8-
GAN Single-feature
6
50000 100000
150000
Figure 3: The graph shows the Grassmann distance over time on the Olivetti Faces dataset, for Oja's method (Blue) and the GAN model (Orange), as well as the single-feature GAN model (Green). We use the same hyperparameters all previous experiments, measured with respect full PCA decomposition which acts as a surrogate for the true subspace.
First, since an orthonormal matrix, lives in the Grassmannian Gr(d, n)) of d-dimensional subspaces of R", Similarly, W E Gr(g, n). Our goal is to embed U and W into Gr(p, n). Once we do this, we can again calculate the macroscopic states we are interested in, To this, we use the following map:
La
pxp-d
(15)
On-d
On On-dxd
This produces new matrix Ü E Gr(p,n), n). We can perform a similar trick with W to obtain matrix W. The important details about this uplifting trick are the following: (1) Due the construction. we preserve orthonormality of all the matrices, (2) the subspaces of interest are found as the first d columns of the matrix Ü and the first g columns of the matrix W, and (3) the analysis of the diagonal case is unchanged under this uplifting (In the diagonal case, there is no interaction between the different dimensions, so we ignore the other dimensions. In the non-diagonal case, these additional dimensions only provide minor noise, and so don't affect the training at all).
7 Real image subspace learning
order to demonstrate the practicality of this analysis, we test our approach on the MNIST [8] and Olivetti Faces [19] dataset, and compare our approach with the single-feature discriminator from Wang et al. (6]. Here, we include some qualitative results regarding the learned features, and provide quantitative analysis the performance differences between the multi-fcature and single-feature discriminators. We include the Olivetti Faces results in Figure 4 and the MNIST results can be found in Appendix
To perform these visualizations and measure performance, we first perform PCA on the entire dataset and extract the top K (16 or 36) features. We then use this an approximation of the true subspace U, which allows us to compare the distances. We then track the Grassmann distance between the true and learned subspaces for both the multi-feature and single-feature approaches. The Grassmann distance between two d-dimensional subspaces of an n-dimensional space is given by
d
1/2
d(U, v)
Σ
02
(16)
where the 0, are the principal angles between the subspaces. Here, lower distance means better similarity between the subspaces. If the two matrices are orthonormal, the principal angles are the singular values of the cosine similarity matrix, explicitly connected with the macroscopic
states.Figure shows the Grassmann distances for the sequential and multi-featur learning cases

---

GAN Multi-feature
GAN Single-Feature
Oja
Epoch 1
Epoch 200
Final Epoch
Figure 4: We provide results on the Olivetti Faces dataset, well-known dataset. We show the top 16 learned features for all approaches at 3 stages of training: after the 1st epoch, the 200th epoch, and the end of training. We train all approaches for 500 epochs, equivalent to approximately 50 timesteps of simulated training. It can be clearly seen that while Oja's method learns quicker than the GAN model, eventually the GAN model outperforms it. Additionally, we see that the features learned by the GAN model are much more diverse and meaningful than those learned by Oja's method (whose learned features are more similar). For the single-feature GAN model, we can see that the learning is significantly slower, and never approaches anywhere close the other two results.
the Olivetti Faces dataset. This provides empirical justification real dataset, showing first that the phenomenon of faster training identified by the ODE in Figure Tapplies to practical settings as well. Furthermore, due having restrictions off-diagonal entries of the macroscopic states, see that the results in Figure allso apply to practical datasets, since our multi-feature discriminator attains better performance even in less time.
8 GANs as subspace learning algorithm
In the linear setting, GANs attempt to perform subspace learning. However, GANs do not fall into either of the categories introduced earlier. The other subspace learning algorithms all seek to minimize the following loss function
((U) Ex X
(17)
known as the reconstruction error. This is because the global optima of this loss function is the true subspace itself, and so, we can view this as a prior included the subspace learning algorithms. GANs do not have such information, and instead seeks to learn the subspace simply through seeing the datapoints. Therefore, we can consider GANs be third type subspace learning algorithm, which we call the generative algorithms. We seek to understand how well the GAN model is able to

---

learn a subspace compared to the existing subspace learning algorithmss We compare both analytically using the derived ODEs, as well as empirically on synthetic and the MNIST dataset, in order to see under what circumstances GANs learn subspace at comparable rate.
8.1 Learned features
Figure@lin the Appendix compares the features learned by the GAN model to the features learned by Oja's method. Both models are initialized to exactly the same weights, and trained on the same data at the same time, for single epoch. For the GAN model, we use the same hyperparameters previous experiments above. For Oja's method, we used learning rate of 0.1, which experimentally we found to produce the best results. We clearly see that the features learned by the GAN model are more meaningful and more clearly resemble the true data, while most of the features that Oja's method learns aren't 't very interpretable. This suggests that because the GAN needs to be able to generate the images, this acts as a form of regularization on what types of features are learned.
the
Conclusion
Our investigation into single-layer GAN models through the lenses of online subspace learning and scaling limit analysis has provided valuable insights into their data subspace learning dynamies. By extending our analysis to include multi-feature discriminators, we' we've ve unearthed novel phenomena pertaining to the interactions among different features, significantly enhancing learning efficiency. This advantage is particularly pronounced in scenarios of near-zero initialization, where the generator achieves higher maximum and steady-state performances compared to the sequential discriminator. Moreover, the interaction between dimensions enables the generator to closely match variances across dimensions, a feat unattainable in the sequential scenario. the context of subspace learning, we see that in higher noise levels, the GAN is able more consistently outperform Oja's 's method wide range of generator, discriminator, and Oja learning rates.
Introducing an uplifting method for analysis in arbitrary dimensionalities enables us to better model uncertainties inherent real-world subspace modeling. Practical validation on the MNIST and Olivetti Faces datasets reaffirms the applicability of our theoretical findings, underscoring the supe- riority of overparametrization in single-layer GANs over data availability. This prompts intriguing avenues for research in multi-layer GANs, probing whether similar phenomena persist in more complex architectures. Exploring these directions holds promise for further advancements in the field. Finally, we observe that GAN models excel in acquiring more meaningful feature basis compared to Oja's method when applied to the real-world datasets, which we attribute to their ability to generate new data samples.
Acknowledgements
We acknowledge that this work was supported in part by TUBITAK 2232 International Fellowship for Outstanding Researchers Award (No. 118C337) and an Al Fellowship provided by Koç University & is Bank Artificial Intelligence (KUIS AI) Research Center.
References
[1] Erkki Oja. Simplified neuron model as a principal component analyzer. Journal of Mathematical Biology, 15:267-273, 1982.
[2] Laura Balzano, Robert D. Nowak, and Benjamin Recht. Online identification and tracking of subspaces from highly incomplete information. 2010 48th Annual Allerton Conference on Communication, Control, and Computing (Allerton), pages 704-711, 2010.
[3] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial networks. Communica- tions of the ACM, 63:139 144, 2014.
[4] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. 2019 IFEF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 4396-4400 2018.
10

---

[5] Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen- Aahen-Or,,e and Dani Lischinski. Styleclip: Text-driven manipulation of stylegan imagery. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 2065--0744 2021.
[6] Chuang Wang. Hong Hu, and Yue M. Lu. A solvable high-dimensional model of gan. In Neural Information Processing Systems, 2018. URL WRLLhhtps://ppie URLattps::/ppp..eeaaaiiiii http:/ppp...ematicsccc.....ooo.e org/ CorpusID: 44033959
[7] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. In Neural Information Processing Systems, 2017.
[8] Eric V. Mazumdar, Michael I. Jordan, and S. Shankar Sastry. On finding local nash equilibria (and only local nash equilibria) in zero-sum games. 2019.
[9] Yuma Ichikawa and Koji Hukushima. Learning dynamics in linear vae: Posterior collapse threshold, superfluous latent space pitfalls, and speedup with kl annealing. In International Conference Artificial Intelligence and Statistics, 2023.
[10] Rodrigo Veiga, Ludovic Stephan, Bruno Loureiro, Florent Krzakala, and Lenka Zdeborov'a a. Phase diagram of stochastic gradient descent in high- high-dimensional two-layer neural networks. Journal of Statistical Mechanics: Theory and Experiment, 2023, 2022.
[11] Maria Refinetti and Sebastian Goldt. The dynamics of representation learning in shallow, non-linear autoencoders. Journal of Statistical Mechanics: Theory and Experiment, 2023, 2022.
[12] Luca Arnaboldi, Ludovic Stephan, Florent Krzakala, and Bruno Loureiro. From high- dimensional & mean-field dynamics to dimensionless odes: unifying approach to sgd in two-layers networks. Proceedings of Machine Learning Research, vol 195:1-29, 2023.
[13] Chuang Wang, Yonina C. Eldar, and Yue M. Lu. Subspace estimation from incomplete observa- tions: high-dimensional analysis. IEEE Journal of Selected Topics in Signal Processing, 12: 1240-120 1252,,25,..1 2018.
[14] Laura Balzano, Yuejie Chi, and Yue M. Lu. Streaming pea and subspace tracking: The missing data case. Proceedings of the IEEE, (06:1293-1314) 2018.
[15] lain M. Johnstone and Arthur Y. C. Lu. On consistency and sparsity for principal components analysis in high dimensions. Journal of the American Statistical Association. 104:682 693, 2009. httphh:tt//ppppp weteriissmaatiicchho..oo
orrg/orrpuusse..ooo.oooo : 30706008
[16] Bin Yang. Projection approximation subspace tracking. IEFE Trans. Signal Process., 43: 95-107, 1995.
[17] Ke Ye and Lek-Heng Lim. Schubert varieties and distances between subspaces of different dimensions. SIAM J. Matrix Anal. Appl., 37:1176- 2014.
[18] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-14) 2012.
[19] AT&T Laboratories Cambridge, The database of faces. URL httph:://///// h/wp::://wwww.// www...cccc.mc.ccmmmmmmmmm
cam ac uk/

---

A MNIST results
A.1
Comparisons with Single-Feature Diseriminator and Oja's Method
In order to demonstrate that our approach works in more practical settings, we train our model on the MNIST dataset. Then, in order understand what the model has learned about the dataset, compute the SVD of the generator weights V, and plot the left singular vectors, Each of these vectors correspond to a single feature learned by the model, and so viewing these will help understand the model performance. Finally, we perform the same tests using the single-feature discriminator, to demonstrate the effects of sequential vs non con-sequential learning of features.
Our theory and development in the paper operated under the assumption that we knew ev- erything about the true subspace. While this is not possible for these image datasets (since we cannot determine the true subspace U the distribution of c), we can still use the same assumptions and model structure. Therefore, the generator still samples a è from standard Gaussian distribution, and the choice of covariance and noise levels are determined through testing.
The dataset is flattened into multi-feature model, we train for single epoch. For the sequential discriminator, we train for 5 epochs. We focus on the d = 36 case, although it can be further scaled up as necessary. Through test- ing, we fix the covariance matrix Å La and nG = We use a generator learning rate of = 0.04 and discriminator learning rate of good representations of all 36 basis elements as seen in Figure [] the sequential discriminator is un- able learn even half of them in the 5 epochs. As be seen, the last 18 basis elements are just noise.
784 vector, so our ambient dimension n = 784. For our
0.2. While the multi-fcature discriminator is able to learn
This scaling becomes very problematic as the number of features increases. Even with just 36 features, a small amount given modern datasets, such model requires significantly more training and is still unable to perform well the multi-feature model.
Finally, we provide a comparison of the GAN learned features with the Oja's learned features in Figure 6 It can be seen that most of the GAN features are more visually representative of the dataset compared to Oja's method.
3
7
Generator Basls In Multi-Feature Case
Generator Basis In Single-Feature Case
Figure 5: Comparison between the generator basis vectors learned by the multi- and single- feature discriminators on 36 features. The nulti-feature model trained for epoch, while the single-feature model is trained for 5 epochs.
A.2 Grassmann distances
Figure 7 contains comparison of the Grassmann distances for the multi-feature and single-feature cases. Even after 5 epochs, the sequential discriminator still has a much higher Grassmann distance than the multi-feature model, even though has seen times as much data. Specifically, after one epoch of training, the multifeature discriminator has a distance of 2.46, while the single-fcature
12

---

?
8
GAN Learned Features
Oja Learned Features
Figure 6: After training both GAN and Oja's method the MNIST dataset with 16 features, SVD to extract their learned features, and compare them here. We can see that despite Oja's method learning quicker than the GAN model (seen in previous analysis), the GAN features learned are better representation the data, while most of Oja's features do not resemble the data. Note that Oja learning the first feature is due the order of training samples seen.
discriminator finishes with a distance of 3.17, showing significant gap. We tested with up to 20 epochs, but saw no improvements for the sequential discriminator past 5 epochs.
We also example of the training outcomes predicted by the ODE in Figure 7 Specifically, our choice of learning rates = 0.04, = 0.2 and noise level = is seen in Row T, Column 2 of Figure [ and see the expected result of the generator oscillating around its steady state.
Nuttihattue wotch
sequendaal epocns
Sequential
wmy
120000
15000C
300000
oru soon
Figure 7: In the first figure, using PCA the MNIST dataset, obtain the top 16 features of the data, and use these features as an approximation for the true subspace We then track the Grassmann distance between the learned subspace and this approximation, for both our model trained for 1 epoch, and the sequential discriminator trained for and epochs. It is clearly seen that our discriminator learns much faster than the sequential discriminator, while at the same time obtaining much lower distance, even when the sequential discriminator has sees 5 times as much data.
B Proof of main theorem
This proof mirrors the proof of Theorem in [6]. However, for completeness, we re-state the key results and sketch the proof here.
The proof relies on the following result, found in
13

---

Lemma B.I. Consider sequence of stochastic processes (*x (n) -= 0, , ,[n?]] (
with some constant T O. xf can be decomposed into three parts
(m) x k+1 x (m) 1 -( (")) + Pk + +
(18)
such that
(C.1) The process En' (a)
Pie is martingale, and IE| p 11 - C(T)/n for some e1 0;
(C.2) E 11 S C(T)/n++e for some e2 0;
(C.3) ((x) is Lipschitz funsciitzfunction, ie., [1ób(x) - (f()) < C||x - +||:
(C.4) [ 1
(1)) 1|2 S for all k $ [nT];
(C.5) 1 (1) xoll < C/n for some 63 > and deterministic vector x
then we have
- x h -|| S C(T)n min (2..0s))
|
(19)
where x(t) is the solution of the ODE
मे x(t (x((t))
with x(0) = =x--
(20)
The relevant stochastic process the macroscopic states introduced in Section states are decomposed as
The macroscopic
1 M M (MM)) + (M++1 M,Mx+1) + B& M (M())))
(21)
Note that macroscopic state can just be written n² dimensional vector, and it is equivalent using the Frobenius norm in the conditions above. We seek to show that this decomposition satisfies the conditions (C.))
Immediately, Condition (C.5) satisfied by the assumption (A.5) for the theorem. Additionally, (C.3)
satisfied by assumption (A.3) and Lemma 4 in the supplementary material of (6).
Next, we slightly modify Lemmas 2 and 7 in the supplementary material of (6 for our case.
Lemma B.2 (Lemma 7 of (). Under the assumptions (A. (A.6), given T >> O, we have
||E Pk.ill S Cn 0 (|| |We,sl))s
(22)
B Ws.ill Cn 10nn11110 ([|u=]1 |M, [[e,,]])-
The proof of this follows exactly from Lemma 7 of 6
Lemma B.3 (Lemma of (6). Under the assumptions (A.I) (A.6), given T > 0, we have
E
Σ EiAl + [Wald) C(T)n
(23)
Proof. We show that this holds for fixed First, we know that
C n17
Eg WK+1. - Wk.lll S (1 + ||u,11 + ||v,,>11 + [[wssll))
(24)
whenever 7 = 2,3,44 4, due to boundedness of h, f, J. Additionally, we can write
E W -iw-=1l.. 11,,, E[wa].. = 4F
[w>] [(
[[[[w++lle [wala,t]]
+ 6E
[[ws]]], (We+1kk- [wwlk.)]]
(25)
+ 4E
[[wee],, (W -[We]J))
+ EEE,
([w++1]aa (
14

---

Combining both of these, we get that
C 2-- (n=2 Bl|u,ll* + Bllvaall + 1 ww.:ell).
Elwa+1lit Ee
n
d
(26)
B(n -2 + Vell + (Walt,)-
which follows from assumption (A.4). Similarly, we get
B(V (Vll)) - E(n E +
Σ
(27)
Combining these for both terms and iteratively applying it, we get
E Veltt + (Wa)i,) S (n + E(volt + (Wll.)e
(28)
-=1
1=1
Then, due to assumption (A.4), we get the required result.
Once we have Lemmas B.2 and B.3, we can show that condition (C.4) is satisfied. Lemma B.4. Condition (C.4) is satisfied for our macroscopic state stochastic process.
Proof. We show that the expected norm squared of each macroscopic state is than some C(T). The cases of P&c and Sy are proven in Lemma 3 of (6. and require changes. Additionally, by our assumption (A.6) that the matrix W& is orthonormalized, we know that Z₁ = La. and so the
requirement is trivially satisfied for Zy Thus, it remains to show this for Q₂ and R We show this for Qu. and R₂ follows similarly.
E
U
W
E
UR.
E
x=1
(29)
E
W w1,
1=1
1=1
< C(T),
required.
Lemma B.5. Condition (C.2) above is satisfied, meaning that for all k = [nT], and for
given T 0, we have
E E M ((M))l < C(T)n 3/2
(30)
Proof. To prove this, we can split it into five parts, one for each of the macroscopic states, For the macroscopic state Zg, this just requires showing that
E E Z+ll S C(T)n 3/2
(31)
But the left side is just zero, since Zμ II for k. Thus, this is trivially satisfied. For the macroscopic state Ps, we want to show that
- n
EE E,P 6+1 Ps-
(Q.RA P&LA)I S C(T))-337
(32)
However, from the gradient of our update equation for V, averaging over č₁, āg see that
f WW VeLa
EgV. V 1++1
V2
(33)
n
15

---

Multiplying both sides by U on the left, we get
B&Px+1 Ps- 7 n.
(34)
But then, the left side the equation we wanted to show is just zero, and so the inequality is satisfied. Applying a similar process to the update equation for Wk. we want show that
E& OR+1 Qk (AQx PAAR, + H2Qx)l & C(T)n 3/2
(35)
and by averaging over Ce2 åk. Cg2 al and multiplying by U on the left, get
Ex Q Q AQ PAAR, + H₂Qu
(36)
Again, this results in the left side of the expression we want to show just being zero. Finally, we show the result for Sy. The case for R3 follows similarly to the previous results. Using the property that
S S₁ V K++1 V4) (V++1 Va) (VK+1 Va) Va);
(37)
and averaging over C22 äg, we get
EgS S R S&L ARRR + L&SS
7 n
72 n²
[W.R[A+VoLL]
[AR VAL
(38)
The second term in the sum above has expected norm
22
E
W
VoLu
W. + V&La
|| W A VoLaI1.
I ZZ R A 2|
(39)
CE [|[ + |S&||]
- C(T).
This concludes the proof.
For condition (C.1), the requirement of being a martingale is automatically satisfied by construction. To show that the remainder of condition (C.1) is satisfied, it suffices prove that
E M
M < C(T)n
(40)
Lemma B.6.
EE M 6+1
M S C(T)
(41)
Proof. We can break this into each of the 5 iacroscooic states separately. As before, doing this for Zy trivial. We show this for P& and Q5. and the rest follow similarly. For Pk, we get
P
Pall2
s On
'E
|||
Q
Ex
11
+ [|]
s Cn 2 1 +
1
(42)
x C(T)n
This finishes the proof for PR. For Q we have
E
E
1
2
||ca 1 f2 + | axll 11 + |
62x F2x + || 22x 11 P2 +
1|Qx (2
n²
< On -2
1 E|IU a 11.
E + BIIUU
au
11
E
+
E
Z
E||S
(43)
- Cn [1+ Z + S
< C(T))=
where in the last line, we used the previously calculated values for Z and TE| Sx The values fü and fex are the values of f = F² and f = F evaluated on the corresponding inputs. The conditions for the rest of the macroscopic states can be shown in the same way.
Given the previous lemmas, the proof of the theorem then follows immediately from Lemma B.I
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims made in the abstract reflect our analysis, and figures throughout text support claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We make clear the assumptions require for our analysis, and exactly what setting our analysis is performed in. We also discuss the areas we are unable to extend our analysis towards.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: For completeness, we provide proof our theorem the appendix, even though it closely matches the proof of similar theorem in a different work.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information about the true data model and generative model, the training procedure and loss function, and state the required ODE formulas. For each set of experiments, we specify the choice of hyperparameters we used to produce those figures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
18

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
We provide a link to our code the Abstract, from which all of our experiments can be reproduced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wttppcciccgg.ceee..eeeeoeeeceeee https//gggiieee/eeeuuubiiiiii www.nissssipiiiiioooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We are careful about specifying choice hyperparameters, training algorithms, and information about the data. We analyze wide range of hyperparameters in paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Much of our analysis is performed theoretically, based on ODEs that are deterministic. Some of the experiments are meant for producing visual results such as learned features, and for these, it does not make sense to consider statistical significance. Furthermore, the premise of our paper deals with convergence in high dimensionalities to a deterministic result. Therefore, for the experimental results we do provide, due to high dimensionalities involved, we can expect results very close to the deterministic result. Because of this, error bars would not provide meaningful information and instead would simply clutter our charts.
19

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: All of our experiments do not need any compute setup, and is doable using a single computer with single CPU. Therefore, is not important for us provide these specific details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttps://neuripptt http:///eerrrrps.cppliii.i.c http/puullc/Bthhsuuiiiiees
Answer: [Yes]
Justification: Our work focuses on theoretical results and analysis. We our own data model as well two freely available and widely used datasets (MNIST and Olivetti Faces).
Our work does not involve new data interactions with humans. Additionally, there legal implications for our work.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
20

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: While models such as GANs do have important societal impacts, due to the theoretical nature of our work and the toy GAN model used, we do not feel that any of these societal impacts are directly relevant to our current work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All data and models discussed in this paper do not have any high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
21

---

Justification: We have cited our original inspirations, as well as the MNIST and Olivetti Faces datasets where used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, pappesswittteeee thcode dantsatsssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing or any research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
22

---

Answer:
[NA]
Justification:
We do not use crowdsourcing or any research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
23


Reviews:
[{"review_id": "vpNbf4TGyG", "reviewer": "Reviewer_uM7k", "summary": "This work explores the training dynamics of a single-layer GAN model, especially for high-dimensional subspace learning, presented as a novel approach. By connecting the GAN models with analysis to subspace learning, this work compares the effectiveness of GAN-based methods with former approaches e.g., Oja’s method and GROUSE, both theoretically and empirically. The findings in this work reveal that GANs demonstrate a remarkable ability to acquire a more informative basis due to their inherent capacity to generate new data samples or in this case subspaces. The experiments in this work demonstrate subspace learning tasks in a more efficient way compared to the counterparts.", "questions": "Please answer the questions and concerns in weaknesses.", "limitations": "There is no special section for limitations. It is hard to view direct negative impacts of this work as this is a core machine learning problem that can be applied to many applications.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- This paper provides a novel perspective for a fundamental problem in subspace learning. The idea of GAN is imposed to learn subspaces with a generator and a discriminator with equal power.\n- This work provides some insights on technical issues implementing GAN to learn subspaces with some different hyperparameter settings in P7.\n- The intuition and motivation for implementing GAN for subspace learning are adequate with relevant backgrounds and theoretical analysis.", "weaknesses": "- Because the subspaces have to be orthonormal, we need to ensure that Eq. 7 maintains this property when performing gradient descents. How can this property be achieved? Or is this not necessary using the approach in this work? Commonly, this desired property can be achieved by geometry aware constraints applied to gradient steps.\n- This work borrows the idea of GAN with a minimax objective between the generator and discriminator. One common issue in GAN is the learning instability (e.g., mode collapse) in the learning stage. This work does not discuss this issue in depth. Is this problem not the case for subspace learning?\n- The efficacy of the proposed method is demonstrated on MNIST with promising performance. However, the dataset used in this work is relatively small with limited variances. The proposed method would be more insightful to work on a larger scale dataset e.g., a face dataset (MORPH).\n-The claim that the learned features are better than Oja’s method in P9 cannot be justified only from the depicted figure. This claim of learned features has not been quantified in what kinds of tasks and for what purposes.\n- The proposed method is limited to the linear setting. Would it be possible to use the GAN design for subspace learning in this method for non-linear subspace learning with more complex types of data? This work would be more insightful to provide relationships with the choice of non-linear functions as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback provided by the reviewer. \n\n- *Because the subspaces have to be orthonormal, we need to ensure that Eq. 7 maintains this property when performing gradient descents. How can this property be achieved? Or is this not necessary using the approach in this work? Commonly, this desired property can be achieved by geometry aware constraints applied to gradient steps.*\n    \nRegarding the orthonormality of the subspaces, it is required to enforce this property in some way. The easiest way to do this is by explicitly orthonormalizing the matrices after each gradient step. However, an alternative we have also tested is using the approach in [1], which allows for gradient updates which keep the matrix on the Grassmann manifold, thereby preserving the orthonormality condition. The last option is using the orthonormality regularization term $tr(H(W^T W))$, however this does not explicitly force the subspaces to be orthonormal.\n    \n[1] A. Edelman, T. A. Arias, and S. T. Smith. The geometry of algorithms with orthogonality constraints. SIAM Journal on Matrix Analysis and Applications, 20(2):303–353, 1998.\n    \n- *This work borrows the idea of GAN with a minimax objective between the generator and discriminator. One common issue in GAN is the learning instability (e.g., mode collapse) in the learning stage. This work does not discuss this issue in depth. Is this problem not the case for subspace learning?*\n\n(Wang et al. 2018) provides an analysis in the single-feature case of the different types of learning, including mode collapse. They find that mode collapse depends on multiple details, including the relation between the generator and discriminator learning rates, the noise level present in the data, and the covariance matrix chosen for the sampled subspace.\n\nWe did find that similar trends hold in the multi-feature learning case. However, since this is not a novel result, and is not directly related to our main goal of understanding the advantages of a multi-feature discriminator compared to a single-feature discriminator, we choose not to focus on this.\n\n- *The efficacy of the proposed method is demonstrated on MNIST with promising performance. However, the dataset used in this work is relatively small with limited variances. The proposed method would be more insightful to work on a larger scale dataset e.g., a face dataset (MORPH). -The claim that the learned features are better than Oja’s method in P9 cannot be justified only from the depicted figure. This claim of learned features has not been quantified in what kinds of tasks and for what purposes.*\n\nThe MORPH dataset is a commercial dataset which requires access to a license, which we do not have. Thus, we instead focus on a different face dataset, the Olivetti Faces dataset (a well-known face dataset). Included in the attached PDF above are results for both the GAN model and Oja’s method on this dataset. Further details can be found in our joint response to all reviewers, and in the caption of the figure. Our theoretical results driven by our work is highlighted in this new setting, in addition to the MNIST results.\n\n- *The proposed method is limited to the linear setting. Would it be possible to use the GAN design for subspace learning in this method for non-linear subspace learning with more complex types of data? This work would be more insightful to provide relationships with the choice of non-linear functions as well.*\n    \nNon-linear GAN models are not within the scope of this work, and as such, we do not have any results or analysis of such a situation. We do believe that this is possible, but we have not derived the necessary ODEs or proved the relevant theorem to enable such an analysis on a more complicated model. However, future work will seek to extend such an analysis to a situation like you have described.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EGVtOCYfDc", "reviewer": "Reviewer_6SvB", "summary": "This paper focuses on the training dynamics of the gradient-based learning algorithms, and converted into a continuous-time stochastic process characterized by an Ordinary Differential Equation (ODE). Empirical evidence demonstrates the correctness of the proposed method.", "questions": "Please could you think of other GAN baselines and evaluations metrics that can be used to strengthen the empirical evidence in this paper? For example the FID score is a popular metric usually used in this generative modeling scenario. Maybe demonstrating that the ODE can lead to similar FID score in order to demonstrates the equivalence between this new model and GAN?", "limitations": "It seems to me there is no discussion on limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: This paper focuses on the training dynamics of the gradient-based learning algorithms, and converted into a continuous-time stochastic process characterized by an Ordinary Differential Equation (ODE). Empirical evidence demonstrates the correctness of the proposed method. This is the first work that connects the ODE with the training dynamics with GAN, which is very inspiring.", "weaknesses": "I am afraid I am not an expert in this ODE area. But still, as a machine learning scientist, I feel the empirical section is lacking the supportive evidence and ablation studies. Maybe adding more comparisons with other GAN-related generative modeling can help better support the assumptions in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and effort. We would like to provide a more detailed summary of our paper, to clarify any confusions you may have about our work.\n\nWe focus on representing the training dynamics of a simplified (linear) GAN model using a system of ODEs, which represent the key states of the training (specifically the interactions between the true and fake subspaces, and the discriminator). We explore using a multi-feature discriminator, which is able to learn significantly faster than a weaker, single-feature (aka sequential) discriminator. The goal is to show that contrary to standard practice in GAN training of using a weak discriminator, it is actually possible to use a discriminator with equal power as the generator. In fact, doing so allows for both faster training as well as a better overall performance. As such, we believe that our results will open new research directions into understanding the training dynamics of more complicated GAN models, and the use of more powerful discriminators.\n\nWe provide numerical results showing that the ODEs do actually match with the GAN training, and we show that our analysis allows us to gain insight into training on real datasets, through out testing on MNIST. The purpose of using MNIST is not to claim that our approach is the best for this task, but instead showing that trends and insights gained from the ODE appear as we would expect. For example, in the Grassmann distance on MNIST diagram (Figure 6), we see the oscillating pattern which is visible in Figure 1, second diagram. \n\nFinally, we aim to position GANs in the category of online subspace learning algorithms, by showing how it learns a subspace compared to existing approaches. Specifically, we are able to empirically show that the features learned by the GAN model are semantically more meaningful than those learned by Oja's. It is possible to learn an arbitrary number of bases for a given subspace, but certain ones are more useful than others, and having features which actually match the true data is one way of seeing this.\n\nFID is a metric used to compare two distributions of images. However, it depends on the model used for extracting the features, and using a model with no exposure to the type of data we are using (specifically grayscale images of digits and/or faces) will mean that the results provided are not very informative. Furthermore, as mentioned above, we do not claim that our model will achieve the best FID, as we do not focus on perceptual quality of the results. Instead, we are focused on metrics such as Grassmann distance, which measures the subspace itself, in terms of the learned features. This is more directly related to subspace learning, which is our setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F7qrKzqfkq", "reviewer": "Reviewer_7DBz", "summary": "This paper proposes to learn the subspace from the observations by the GAN model. Taking the one-layer GAN model as the starting point, this paper provides a theoretical analysis from the perspective of training dynamics. Specifically, from the technical side, the proposed method trains both the generator and discriminator by using adversarial loss and subspace regularization.  Different from the original GAN dynamic training method, this paper further proposes an assumption that \"the columns of the discriminator matrix W are orthonormal\"\nThe proposed method is evaluated on both synthetic and real-world datasets.", "questions": "1) What is the unique advantage of the GAN model for subspace learning? Why not use the VAE-based framework, such as iVAE [RQ1] and PCL[RQ2]. In my understanding, subspace learning is a representation learning method, VAE-based methods explicitly model a mapping from observation to latent variables (encoder), which may be more efficient to use in subspace learning.\n\n2) What is the detailed difference between this paper and (Wang et al. 2018) on the theoretical side? \n\n3) What is the advantage of the proposed subspace regularization $ tr(H(W^TW))$ over other regularization methods [RW1,RW2] .\n\n[RQ1] Khemakhem, Ilyes, et al. \"Variational autoencoders and nonlinear ica: A unifying framework.\" International conference on artificial intelligence and statistics. PMLR, 2020.\n[RQ2] Hyvarinen, Aapo, and Hiroshi Morioka. \"Nonlinear ICA of temporally dependent stationary sources.\" Artificial Intelligence and Statistics. PMLR, 2017.", "limitations": "No clear negative societal impact needs to be listed.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "This paper provided a systematical analysis of GAN-based methods and conventional approaches, from both theoretical and empirical sides. \n\nMore simulation results are attached in the appendix to verify the effectiveness of the method.", "weaknesses": "I have a big concern about the contribution of this paper.  \n\n1) The reason for using GAN for subspace learning, instead of other frameworks (such as VAE), is unclear. The advantages and drawbacks of this choice should be discussed. Please refer to the questions for the details.\n\n2) This paper provides a theoretical analysis from the perspective of training dynamics (Wang et al. 2018). However, most equations and theoretical results are similar as (Wang et al. 2018), such as equations [1,5,6,7,8,9 10].  Only the assumption that \"the columns of the discriminator matrix W are orthonormal\" is newly added. However, given that the loss function in (Wang et al. 2018) had already constrained this. \n\n3) Technically, some GAN-based empirical methods [RW1,RW2] tried to add subspace regularization into the GAN model.  Some methods apply the disentanglement constraints to achieve the identification of the latent space, such as [RW3]. \nTechnically,  what is the advantage of the proposed subspace regularization $ tr(H(W^TW))$. \n\n4) In the experiments, the comparison with VAE-based methods [RQ1,RQ2], the GAN-based empirical methods [RW1,RW2] should also be involved. \n\n[RW1] Liang, Jie, et al. \"Sub-GAN: An unsupervised generative model via subspaces.\" Proceedings of the European Conference on Computer Vision (ECCV). 2018.\n[RW2] Jiang, Hongxiang, et al. \"Orthogonal Subspace Representation for Generative Adversarial Networks.\" IEEE Transactions on Neural Networks and Learning Systems (2024).\n[RW3] Xie, Shaoan, et al. \"Multi-domain image generation and translation with identifiability guarantees.\" The Eleventh International Conference on Learning Representations. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "While we appreciate the reviewer’s comments, we respectfully disagree with his/her assessment of the level of novelty. We refer the reviewer to our joint statement to all reviewers to clarify the novelty of our paper.\n\n- *This paper provides a theoretical analysis from the perspective of training dynamics (Wang et al. 2018). However, most equations and theoretical results are similar as (Wang et al. 2018), such as equations [1,5,6,7,8,9 10]. Only the assumption that \"the columns of the discriminator matrix W are orthonormal\" is newly added. However, given that the loss function in (Wang et al. 2018) had already constrained this.*\n\nRegarding the similarity to the previous approach, our loss function and design was intentionally and carefully chosen to be similar to (Wang et al. 2018), in order to provide a fair comparison between the two approaches. Specifically, our main goal was to understand how multi-feature (stronger) discriminators compare to single-feature (weaker) discriminators, with all else being equal. Note that this is not a trivial extension, and furthermore, by introducing the uplifting method, we are the first to show that such an analysis provides novel insights into the training of GAN models with real datasets. The prior analysis results were only focused on synthetic datasets and only allows for sequential learning (one feature at a time), and as our experimental results have shown, it is not straightforward to apply the previous analysis to real datasets. As such, we believe that our contributions are significant, and has the potential to open new research directions into a better characterization of GAN training dynamics.\n\n- [Q1:] *What is the unique advantage of the GAN model for subspace learning? Why not use the VAE-based framework, such as iVAE [RQ1] and PCL[RQ2]. In my understanding, subspace learning is a representation learning method, VAE-based methods explicitly model a mapping from observation to latent variables (encoder), which may be more efficient to use in subspace learning.*\n\nOur purpose of the experiments is to show that such insights gained from the theoretical analysis is actually transferable to real situations. Furthermore, we focus on the linear case, while both [RW1] and [RW2] are in general non-linear models with significantly more advanced setups. VAE models are outside the scope of our work, as our setting is understanding how GAN methods fit into subspace learning in general. Furthermore, while there are limited works on this type of analysis for VAE models such as the paper [1] cited in our paper, they are focused on other issues such as posterior collapse, and the ODEs provided (Appendix A of [1]) are far too complicated for any similar analysis to be performed.\n\n[1] Ichikawa, Y., \\\\& Hukushima, K. (2023). Learning Dynamics in Linear VAE: Posterior Collapse Threshold, Superfluous Latent Space Pitfalls, and Speedup with KL Annealing. ArXiv, abs/2310.15440.\n\nWhile it is possible to view the VAE models from a subspace learning approach, our key goal is to understand how GANs exist compared to other traditional subspace learning methods. Additionally, the VAE models you have mentioned are non-linear models relying on MLPs, and so do not provide a fair comparison.\n\n- [Q2:] *What is the detailed difference between this paper and (Wang et al. 2018) on the theoretical side?*\n\nThe following are the key differences between the papers on the theoretical side:\n- We focus on the advantages of a multi-feature discriminator compared to a single-feature discriminator. This includes understanding the increases in speed that comes with a multi-feature discriminator, as well as the increases in overall learning of the true subspace, measured in terms of Grassmann distance between the true and learned subspaces.  \n- We introduce a method for extending the analysis to cases where the dimensionality of the true subspace is not known. This method allows for analyzing more realistic situations and real-world scenarios, where such information is not known and not learnable. \n- We show how GAN models can be viewed as a new type of subspace learning algorithm, which we can call the data-driven approaches, compared to the algebraic (Oja's) and geometric (GROUSE) methods discussed in our paper. We use the characterization of training dynamics to provide a systematic comparison of the different approaches' performance.\n\nWe refer the reviewer to our joint statement to all reviewers for a more detailed explanation of these key points.\n\n- [Q3:] *What is the advantage of the proposed subspace regularization $tr(H(W^T W))$ over other regularization methods [RW1,RW2] .*\n\nThe choice of normalization $tr(H(W^T W))$ is a simple way of enforcing that the discriminator matrix W remains orthonormal during training. By using a sufficiently high lambda value, this will force the matrix to be orthonormal, which is required by our assumptions. However, the use of this term is not explicitly meant for regularization, just for enforcing orthonormality.\n\nNote that orthonormality is useful in subspace learning because it allows for as much as possible to be learned by the different dimensions, without any overlap (which would mean redundant information).\n\n[RW2] proposes a 3-stage process to improve regularization of the learned subspace. The goal is not simply for an orthonormal basis of the subspace, but also for a more explainable and interpretable latent space, which is not our goal or focus in this work. Furthermore, we do not need to use multiple stages of training.\n\n[RW1] uses clustering to identify certain subspaces, and predicts which subspace the data belongs to. We focus on learning a single subspace.\n\nOverall, the mentioned papers do not allow for a fair comparison with our approach or our choice of subspace regularization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mXU9PCZRWi", "reviewer": "Reviewer_N4Fr", "summary": "This work introduces a simplified GAN framework to learn the subspace of a spiked covariance model. The authors are able to derive precise training dynamics given specific assumptions and show they correspond to numerical simulations. They also prove that convergence rate is often faster than previous work that used a further simplified discriminator (single vector vs. matrix parameter). Authors extend convergence results to the case where exact subspace dimension is not known a priori.  Lastly MNIST results are conducted that show steady state feature alignment for this GAN method is higher than a standard method (Oja's method).", "questions": "Is Fig 2. meant to include GROUSE comparisons?\n\nDoes the generator need to be changed when moving from the spiked covariance dataset to MNIST one?\n\nDoes Fig 2 suggest that making the discriminator learning rate as small as possible the best course of action for feature learning?", "limitations": "While the authors state their assumptions explicitly, I would like to see further discussion on which assumptions could possibly be relaxed in the future.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Clarity:** The authors introduce the spiked covariance model and their GAN method+ assumptions clearly\n\n**Theoretical Results:** The authors use a variety of tools to characterize their system dynamics, and extend their results to differing dimensionalities of the generator and true distribution subspace", "weaknesses": "**Clarity:** While most sections are well written, a few components made it more difficult to understand the key contributions of the paper\n* The introduction doesn't clearly define single feature vs. multi-feature discriminator learning on lines 34/35\n* I'd like to see more discussion of the interpretation of the main theorem (4.3), especially since the ODE dynamics are fairly complex\n* Similarly in section 4.2 I'd like a brief discussion on how these microscopic dynamics are useful to analysis\n* In Fig. 1 It is unclear the what the difference between blue and yellow lines is (same for red and green)\n\n**Comparison to Previous Work**\n* While GROUSE and Oja's method are introduced as alternative subspace methods, it appears as if only Oja's method is used as an experimental baseline\n* I'd like to see one or two citations of other GAN stability analysis papers (e.g. something like Which Training Methods for GANs do actually Converge?) and their limitations as opposed to this works analysis\n\n**Experiments:**\n* Fig. 2 appears to show steady state feature alignment for GAN and Oja's method, but I would be interested in a comparison of the convergence rates as well.\n\n**Significance:**\n* Many assumptions seem hard to extend to the full GAN setting, for example A3 assumes an uncommon discriminator non-linearity and regularization function. A6 assumes the discriminator matrix parameter is orthonormal. \n* Most of this work appears to be extending the results in [Wang et al. 2018] to a slightly more complex discriminator. This is probably my biggest concern, and would like to hear the authors opinion on this difference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the valuable feedback provided by the reviewer. \n- *The introduction doesn't clearly ...*\n\nWe define single-feature to be a discriminator which can only learn one dimension of the subspace at a time. This is quite a slow process, especially when the number of features is high. We seek to show the benefits of using a multi-feature discriminator, which is a discriminator that can learn multiple (or all) dimensions at once, yielding a stronger discriminator. We will include a definition of these terms in the introduction. \n    \n- *I'd like to see more discussion ...*\n\nWe will include a more detailed explanation of the ODE. However, we include the key details here:\n1) The key parameter that we care about is the $\\mathbf{P}$ macroscopic state, representing the similarity between the true and learned subspaces. We can see that it depends on the interaction between the discriminator and the true subspace, and the interaction between the generator and discriminator. If either of those states are the zero matrix, training will not be possible. However, we find that random initialization is sufficient for escaping the fixed point around 0.\n2) The discriminator-true subspace macroscopic state (i.e., $\\mathbf{Q}$) depends on both its current overlap with the true subspace, as well as how the generator and discriminator interact. \n3) All the states depend on the covariance matrix (either the true or fake covariances). As can be seen in Figure 1, the dimension which has the largest covariance (the red color) is the dimension which is able to learn the features better. However, there is also a balance. Using too high a value for the covariances will mean that the generator samples appear unlike the true samples, preventing training.\n    \n- *Similarly in section 4.2 ...*\n\nWe will include such a discussion. The main use of the microscopic dynamics is to derive the PDE, from which the ODE is derived using the relevant test functions (cosine similarity).\n    \n- *In Fig. 1 It is ...*\n\nThe red and green lines represent the two dimensions of the generator, while the blue and yellow lines are the two dimensions of the discriminator.\n    \n- *While GROUSE and Oja's method ...*\n\nWe primarily used Oja’s method due to the asymptotic equivalence of Oja’s and GROUSE, and since the datasets used are sufficiently high dimensionality.\n    \n- *I'd like to see one or two ...*\n\nWe will be adding the following citations:\n\n[1] Mescheder, L.M., Geiger, A., Nowozin, S. (2018). Which Training Methods for GANs do actually Converge? \n\n[2] Fedus, W., Rosca, M., et al.. Many Paths to Equilibrium: GANs Do Not Need to Decrease a Divergence At Every Step.\n\nThe first paper is focused on analyzing the different methods of GAN regularization and how those impact training. Specifically, they focus on gradient penalties.\nThe second paper suggests analyzing GAN training from the perspective of Nash Equilibria, and while focused on true datasets, does not provide a way of analyzing the training and its modes, instead seeking to understand how different choices such as a gradient penalty impact training.\n\nIf there are any additional papers that you believe will be important for us to contrast against, we would be happy to take your recommendations.\n\n- *Fig. 2 appears to show ...*\n\nFor convergence rates, please see the attached PDF for a comparison of convergence rates on a real dataset (as measured by Grassmann Distance).\n\n- *Many assumptions seem hard ...*\n\nWhile we focus on the linear case, we find these results interesting and suggests future exploration into whether such a trend exists in the nonlinear case as well. Specifically, instead of the usual trend in training GANs of using a weak discriminator compared to the generator, it seems that it is possible to use a discriminator of similar strength.\n\nWhile this is true, we chose these in order to provide a fair comparison between single-feature (sequential) learning (found in the original paper Wang et al. 2018) and multi-feature learning. Furthermore, there are ways to to constrain the optimization of a matrix to be orthonormal (by modifying the gradients to live on the Grassmannian manifold). \n\n\n- *Most of this work ...*\n\nThe importance of our work comes not just from using a more complex discriminator, but in showing (both theoretically and empirically) that contrary to the standard assumptions, it is in fact possible to train GAN models with more powerful discriminators. Furthermore, training speed is faster, and the model actually learns more of the true subspace compared to a weaker discriminator. Empirically, we can see from Figure 5 the key differences in the learned features, even when providing the sequential discriminator 5 times as much training time. While we focus on a linear discriminator and generator, we believe that these results are important and open up important avenues for future research into extending these results into practical models.\n\n- *Is Fig 2. meant to include GROUSE comparisons?*\n\nFigure 2 is using the derived ODEs for both the GAN and Oja’s method, since [3] shows that Oja’s method and GROUSE have the same asymptotic trajectory in high dimensions, and so the same ODEs. However, we will make this more clear.\n\n[3] Chuang Wang, Yonina C. Eldar, and Yue M. Lu. Subspace estimation from incomplete observations: A high-dimensional analysis.\n\n- *Does the generator need ...*\n\nNo, there is no change in the generator.\n\n- *Does Fig 2 suggest ...*\n\nFor the situation in Figure 2, we use a warm initialization, avoiding the fixed point around 0. In such a case, if the goal is to attain the best learning of the subspace regardless of training time, smaller learning rates are better. In Figure 2, we consider the steady-state values after 5000 timesteps, to ensure that all the approaches have converged. However, in reality, training time is an important factor to consider, and so it will be important to balance these when picking learning rates.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chairs,

We appreciate the valuable feedback provided by the reviewers. 
We are encouraged that the reviewers found our paper provides a novel perspective on exploring the precise dynamics of the single-layer GAN model on subspace learning problems. We are also glad to see that the reviewers agreed that our experiments back up our claim that multi-feature discriminators enable improved training performance by jointly learning features in a non-sequential way unlike the prior results. However, we understand that they may have some concerns on the level of the novelty, which we address in our response to all reviewers.  
The following points clarify the novelty of the paper with some updates:

- The key contribution is the analysis of multi-feature vs single-feature (sequential) discriminators when it comes to training GANs. By using a multi-feature discriminator, not only is training much faster, but in fact it is possible to learn the true subspace much better. Figure 4 in the Appendix shows the gains made by switching to a multi-feature discriminator in terms of cosine similarity with the true subspace. Additionally, Figure 6 shows this more concretely with a real dataset (MNIST), where we can see not only the much faster convergence, but also the better steady-state results.

- We also introduce a new method for analysis in the cases where the true subspace feature dimension is not known. The common assumption of knowing the exact number of features is restrictive and does not match real-world datasets or scenarios. Our uplifting technique allows for more broad analysis in future works, where it will be possible to analyze learning outcomes depending on whether the number of fake features is smaller or greater than the number of true features. Additionally, our ODEs work using this method, which allows for not just the two extremes of learning a single feature at a time, or all at once, but any possible number of features in between.

- We provide additional results (as requested) in the attached PDF on another dataset, the well-known face dataset Olivetti Faces. The results emphasize our point more clearly, as the top 16 features learned by the GAN model are much more diverse and representative of the entire dataset compared to the features learned by Oja's method. We see that Oja's method is faster in terms of convergence, but all the features learned are similar. Eventually, after approximately 50 timesteps of training, the GAN model outperforms Oja's method in terms of Grassmann distance.

- We position the GAN model within the other subspace learning algorithms, showing how it compares, and highlighting how the learned features are more semantically diverse and meaningful compared to subspace learning algorithms such as Oja's method, and in fact does eventually outperform those methods for a range of learning rates. We use the word meaningful to state that it is more representative of the dataset visually. We see this as the GAN model being forced to learn a basis that matches the dataset better, because it has to learn how to trick the discriminator. 

As such, we believe presenting our work at NeurIPS 2024 will significantly contribute to the discussions at the conference.

Meta-review: **Summary of the paper:**
The paper builds on Wang et al’s work [6] to use single-layer GANs for subspace learning and extend it to a possible use of multi-feature discriminator with an unknown number of subspace dimensions. It further analyzes the improved convergence speed of such multi-feature discriminator compared to a sequential learning of a single feature discriminator. It demonstrates improved results over single-feature discriminators both in the quality of obtained bases and the empirical convergence time on several toy as well as real-world datasets.

**Summary of the reviews:**
Some reviewers appreciated the novelties in studying single-layer GANs dynamics for subspace learning. They further appreciated the empirically improved convergence speed and more successful modeling of more complex datasets compared to the baseline work [6], thanks to the multi-feature discriminator. They generally appreciated the clarity of the presentation.

On the other hand, they had concerns regarding (1) the technical novelty in light of Wang et al’s work [6] from which the formal framework of this work is borrowed, (2) motivation of working with single-layer GANs when alternatives such as VAEs and deeper GANs have been used for non-linear subspace or latent feature learning, (3) lack of more complicated datasets and absence of a proper quantitative analysis of the resulting bases.

**Summary of the rebuttal and discussions:**
The authors tried to clarify several points including (1) the formal novelties on top of [6], (2) how to preserve the orthonormality of the bases during SGD optimization, (3) the focus of the work and lack of comparison with non-linear VAE and GAN subspace learning. They further provided additional experiments on a face dataset.

The reviewers did acknowledge the theoretical novelties, on top of [6], after the discussion but not all were convinced about the significance of them. However, they all appreciated the empirical improvements, specifically for the new face dataset. 

**Consolidation report:**
This work formally extends the study of Wang et al. [6] to the case of non-sequential learning of multiple (unknown number of) features and for an arbitrarily large number of dimensions in the discriminator which can potentially improve its expressivity (and convergence time) for more complicated datasets. On the empirical side, despite the lack of an objective quantitative analysis (save for Grassmann distance), the qualitative bases visualization for the extension (at relatively early epochs) look more convincing or at least visually pleasing compared to Oja’s and fundamentally better than (non-converged) single-feature GANs. They have validated their method on real-world datasets that was not done in [6]. The real-world results further show the empirical significance of the extension in the learning of the complex bases of a face dataset as well as its convergence time. 
While the reviewers eventually did not reach a point to confidently vouch for the acceptance and one reviewer remained leaning towards rejection, all reviewers eventually acknowledged the theoretical novelty and empirical improvement over the prior work of [6]. The question was more a matter of the significance of these two contributions.

The AC believes the paper has taken significant steps from [6] and despite the simplicity of the setup (single-layer GAN) the line brings about detailed formalisms that can lead to further works on both feature learning and analysis of GANs training dynamics; both of which are important fields. 

**Recommendation:**
The AC sides with the majority of reviewers and leans towards acceptance. In a camera ready version, the authors are encouraged to:

- shorten the background of [6] so that they are able to more clearly, and at more length, present and discuss the details of their own formal contributions.
- incorporate the importance of the line of work in the context of other works on studying the training dynamics of GANs, the linear and non-linear subspace learning with GANs and other types of generative networks such as VAEs as well as score-based diffusion models.
- include the new face dataset possibly with a quantitative analysis of their identified bases.
Decision: Accept (poster)