Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning

Authors: andrew  bond
zafer  dogan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:12:39

Abstract: Subspace learning is a critical endeavor in contemporary machine learning, particularly given the vast dimensions of modern datasets. In this study, we delve into the training dynamics of a single-layer GAN model from the perspective of subspace learning, framing these GANs as a novel approach to this fundamental task. Through a rigorous scaling limit analysis, we offer insights into the behavior of this model. Extending beyond prior research that primarily focused on sequential feature learning, we investigate the non-sequential scenario, emphasizing the pivotal role of inter-feature interactions in expediting training and enhancing performance, particularly with an uninformed initialization strategy. Our investigation encompasses both synthetic and real-world datasets, such as MNIST and Olivetti Faces, demonstrating the robustness and applicability of our findings to practical scenarios. By bridging our analysis to the realm of subspace learning, we systematically compare the efficacy of GAN-based methods against conventional approaches, both theoretically and empirically. Notably, our results unveil that while all methodologies successfully capture the underlying subspace, GANs exhibit a remarkable capability to acquire a more informative basis, owing to their intrinsic ability to generate new data samples. This elucidates the unique advantage of GAN-based approaches in subspace learning tasks.

Main paper: 
 introduction: 
 Subspace learning is a widely explored task, especially with the growth of dimensionality in modern datasets. It is important to identify meaningful subspaces within the data, such as those determined by principal component analysis (PCA). However, due to the high dimensionality of the data, it is common to employ online methods such as Oja's method [1] and GROUSE [2] . Meanwhile, Generative Adversarial Networks (GANs) [3] , primarily used as generative models, have also demonstrated the ability to learn meaningful representations of data [4, 5] . Inspired by this, we explore how single-layer GAN models can be viewed as a form of subspace learning.We seek to improve the understanding of GAN training by relaxing some common assumptions made in previous analysis of GANs [6] . Specifically, we focus on the training dynamics of the gradient-based learning algorithms, which can be converted into a continuous-time stochastic process characterized by an ordinary differential equation (ODE). Furthermore, the dynamics of the model weights form a stochastic process modeled by a stochastic differential equation (SDE). Understanding these two equations provides the relevant information to understand convergence behaviour of the 
subspace learning: 
 Subspace learning is a heavily explored field, with many algorithms. However, when the noise of the data has a non-zero variance, most approaches fail, and the general technique used to solve the problem is some type of online PCA-based method. In Wang et al. [13] , a similar analysis of dynamics through ODEs is performed for multiple algorithms which learn subspaces with non-zero variance of noise. This analysis allows for steady-state and phase transition analysis of these algorithms. Balzano et al. [14] presents a survey of different online PCA algorithms used for subspace learning, in the case where only some of the data is visible at each timestep, and discuss how it is possible to find a unique subspace of a given rank which matches all the provided data.3 Background and problem formulationQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We make clear the assumptions we require for our analysis, and exactly what setting our analysis is performed in. We also discuss the areas we are unable to extend our analysis towards.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
true data model: 
 Our data y k is drawn from the following generative model, known as a spiked covariance model [15] :EQUATIONHere, U ∈ R n×d is the true subspace we wish to learn represented as an orthonormal basis, c k ∈ R d is a zero-mean random vector with covariance matrix Λ, a k ∈ R n is a standard Gaussian vector, and η T represents the noise level.The spiked covariance model is very widely studied, due to the non-triviality of learning U whenever η T > 0. The key property of this model is that the top d eigenvectors of the data covariance E[y k y T k ] are given by the columns of U. If there exists a strict eigengap between the top d corresponding eigenvalues and the other eigenvalues, then the reconstruction loss function is proven to have U as a global minima [16] .Subspace learning is a very important task in machine learning, most commonly performed by algorithms such as PCA or ICA. However, these approaches involve costly operations such as calculating covariance matrices or calculating matrix inverses, infeasible in high dimensions. Therefore, it is very common to use an online version of these algorithms, processing samples one at a time.Online subspace learning algorithms typically fall into two categories: algebraic methods and geometric methods. Algebraic methods are based on computing the top eigenvectors of some representation of a sample covariance matrix. Assuming a strict eigengap, the top eigenvectors will yield the true subspace. Meanwhile, geometric methods optimize a certain loss function over some geometric space (Euclidean space or a Grassmannian manifold). We review two subspace learning algorithms here, Oja's Method [1] and GROUSE [2] . While GROUSE was introduced for the missing data case, it can be used for full data too. For further details about these algorithms and their categorization, we direct the reader to [17] .However, we suggest a third category of online subspace learning algorithms, which we call the generative methods. Such methods, including single-layer GANs, do not have information about the specific task, and instead aim to learn the data simply by seeing the data and attempting to generate data from the same distribution.Oja's method [1] is a classical algebraic approach to online subspace learning. Given an orthonormalized initial matrix X 0 , we perform the following update at every timestep given a data sample y k :X k+1 = Π X k + τ y k y T k X k (2)Here, Π is an orthonormalization operator, and τ is the learning rate.GROUSE [2] performs gradient descent on the Grassmannian manifold, which guarantees orthonormality of the updates. In the full data case, we again start with an orthonormal initial matrix X 0 , and at each timestep, given a data sample y k , our update is:EQUATIONHere,θ k = τ ||r k ||||p k ||, w k = arg min w ||y k -X k w|| 2 2 , p k = X k w k , r k = y k -p k ,and τ is our learning rate.Here, we focus specifically on GANs. A GAN model seeks to learn a representation of the underlying subspace through the use of two components: a generator and a discriminator. The generator learns the subspace by trying to generate new samples from the subspace, while the discriminator acts as a classifier, attempting to distinguish data from the true subspace from data produced by the generator.Note that measuring performance through cosine similarity can actually be viewed as a way to measure the generalization performance of the generator, as it doesn't depend on any specific instance of generated data and instead provides a concrete measure of how similar the generated data will be.We assume that the generator also follows a spiked covariance model:EQUATIONHowever, we do not assume that η G = η T , or that the covariance of ck , Λ, is the same as Λ. The goal of the generator is to learn V k .The learning in the GAN model critically depends on the choice of discriminator, which aims to separate the data from the true and generated subspaces.The most common approach when training GANs is to use a discriminator that is weaker than the generator. If the discriminator is too strong, then it will easily learn to distinguish between true and generated samples, leading to vanishing gradients for the generator and thus preventing learning. However, a weak discriminator results in sequential learning, where the generator is only able to learn a subset of the features at a time. In multi-feature cases, this will lead to very slow learning.Motivated by this, we seek to analyze a model in which the discriminator has the same strength as the generator. Thus, we let W ∈ R n×d , and define the discriminator asEQUATIONwhere D : R n → R is some function (see the assumptions below). Since this discriminator is able to focus on all the features at once, this means the generator is also able to learn every feature at once. This is in contrast to the single-feature case (where W ∈ R n ) analyzed previously. While this is a strong assumption on the discriminator, we show below how this assumption can be relaxed.GAN training is modeled as a two-player minimax game, where the discriminator attempts to maximize some loss function and the generator attempts to minimize it. This is used as a way to learn a "surrogate" subspace which represents the true subspace. Therefore, the GAN model can be seen as a form of subspace learning, except that the focus is on generating new samples from the subspace.Specifically, let L(y, ỹ; W) be a loss function depending on the discriminator weights, and true and fake samples. If G denotes the true distribution and G denotes the generator distribution, the minimax game can be represented as minV max W E y∼G E ỹ∼ G L(y, ỹ; W).Following the approach of Wang et al. [6] , and in order to compare the sequential and multi-feature cases, we use the following loss function:EQUATIONHere, F, F are functions affecting the outputs of the discriminator, H is an element-wise function used for regularizing the weights of the generator and discriminator, and λ > 0 controls the strength of the regularization. As λ → ∞, the matrices V, W will become orthonormal.The standard approach to solve this minimax game is using stochastic gradient descent (SGD). At timestep k, given a sample y k from the true subspace and a sample ỹk from the generator subspace, we perform the following updates:EQUATIONHere, τ denotes the learning rate of the discriminator, and τ denotes the learning rate of the generator. Note that while it is common to use a batch of data at a time when using SGD, we focus on a single element at a time in order to simplify all the analysis.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
development of ode: 
 Similar to [6] , we make the following definitions: Definition 4.1. X k := [U, V k , W k ] ∈ R n×3d is, Q k , R k , S k , Z k } is called the macroscopic state of X k at time k, where P k := U T k V k , Q k := U T W k , R k := V T k W k , S k := V T k V k , and Z k := W T k W k .The macroscopic state can be written in matrix notation as M k = X T k X k , in which we getEQUATIONTo analyze the macroscopic dynamics, we reduce to a special case, which leads to a slightly modified set of the assumptions from Wang et al. [6] .(A.1) The sequences c k , ck are i.i.d. random variables with bounded moments of all orders, and {c k } is independent of {c k }.(A.2) The sequences {a k }, {ã k } are both independent Gaussian vectors with zero mean and covariance matrix I n .(A.3) H(A) = log cosh A -I, D(x) = ||x||, and F (x) = F (x) = x 2 2 . We note that the first derivative of H exists, the first four derivatives of F ( D(•)), F ( D(•)) exist, and all the derivatives are uniformly bounded. Thus, our choices satisfy the conditions of assumption (A.3) from Wang et al. [6] .(A.4) Let [U, V 0 , W 0 ] be the initial microscopic state. For i = 1, • • • , n, we have E[ d l=1 ([U] 4 i,l + [V 0 ] 4 i,l + [W 0 ] 4 i,l )] ≤ C/n 2, where C is some constant not depending on n.(A.5) The initial macroscopic state M 0 satisfies E||M 0 -M * 0 || ≤ C/ √ n,where M * 0 is a deterministic matrix and C is some constant not depending on n.(A.6) The columns of the discriminator matrix W are orthonormal, so that W T W = I d . Assumptions (A1) and (A2) are the usual i.i.d assumptions common in machine learning. (A3) is important for deriving the update equations. (A4) and (A5) are used to guarantee that the macroscopic state can converge. Our assumption (A6) of orthonormal discriminator matrix allows us to simplify the equations since the Z matrix of the macroscopic state is always just I d . Under these assumptions, as well as letting λ → ∞, we obtain a modified Theorem 1 from Wang et al. [6] , specifically considering the reduced case of equation (13) . Note that our choice of F, F , D means that our equations become an arbitrary-dimensional version of the original equations. EQUATIONwhere C(T ) is some constant depending on T but not n, and M(t) : R + ∪ {0} → R 3d×3d is a deterministic function. Moreover, M(t) is the unique solution of the following ODE:EQUATIONwith the initial condition M(0) = M * 0 , whereEQUATIONA sketch of the proof of this theorem can be found in Appendix B. The proof closely mirrors the proof of the original theorem in [6] .The microscopic dynamics are concerned with how the terms U, V, W change over time. Following previous work, we consider the empirical measureEQUATION)where δ is the delta measure. This is a discrete-time stochastic process, which can be embedded in continuous time as µ(n) t = µ k , with k = ⌊nt⌋.Then, as n → ∞, this process converges to a deterministic process µ t , which is the measure of the solution of the SDEEQUATION)where A is some diffusion term, negligable due to our assumption on the discriminator (A.6).From this equation and the convergence of the measure, we can obtain the following weak PDEEQUATIONwhere φ is a bounded, smooth test function. The ODE in the main theorem can be derived from this weak PDE. 
simulations: 
 In order to demonstrate that the ODE properly represents the training dynamics of the GAN model, we first perform simulations and show that the empirical results match the ODE, seen in Figure 1 .To understand how the training dynamics change based on the generator learning rate, we fix the discriminator learning rate as τ = 0.2 and fix the generator learning rate τ = 0.04. We show the results on 4 different noise levels. In all cases, we letEQUATIONWe set P 0 = Q 0 = 0.1 * I, and we ensure that the empirical setup is initialized with exactly matching P and Q values. We note that the ODE will never learn when the initialization is exactly 0, and so we must provide some level of similarity to start training. However, this is not very restrictive, as our experiments show that even random matrices will have approximately 0.001 * I for both P and Q, which is sufficient to escape the fixed point around 0. The columns represent η G = η T = 2, 1, 3, 4 respectively. At η = 5 or higher, the generator is unable to learn anything. In all cases, the green and red represent the two diagonals of P, and the blue and yellow represent the two diagonals of Q. We see that the simulations do match the predicted ODE results.A key insight found from the multi-feature discriminator is that the interaction between different features can help learning. When the macroscopic states are initialized to non-diagonal matrices, we see that the dimension with smaller covariance is actually able to attain better results and reach a similar cosine similarity to the dimension with higher covariance. Such an outcome is not possible in the sequential learning regime, due to the lack of interaction between features. In sequential learning, features are learned one at a time, and once a feature has been learned, the training will focus on a different feature instead. This phenomenon can be seen in Figure 2 , showing that the off-diagonal initialization allows for not only faster training (which also happens in the diagonal initialization case), but also higher steady-state values compared to the sequential learning case. We are unable to provide a detailed characterization of these fixed points, as a neat closed-form solution cannot be obtained.Figure 2 : ODE results when initialized with off-diagonal entries. We focus on the case η G = η T = 2, as that noise level is seen above to be ideal for learning. Additionally, in all cases, τ = 0.04, τ = 0.2.The solid lines are with our approach, while the dashed lines are using the discriminator in Wang et al. [6] . From left to right, we use an initialization of 0.1, 0.01, 0.001, 0.0001 for each component of the macroscopic states. It can be seen that our approach outperforms the single-feature discriminator in every case, with the gap becoming larger as the initialization approaches 0.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] We provide a link to our code in the Abstract, from which all of our experiments can be reproduced.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
unknown number of features: 
 While this type of analysis can provide interesting insights, it has a very restrictive assumption that we know the number of features d. This is done so that the macroscopic states are well-defined. However, we now seek to extend this analysis to the case where the true subspace has d features, the generator subspace has p features, and the discriminator learns q features, where we do not assume that d = p = q. While this analysis can be performed under any assumptions on the relative size of d, p, and q, we focus on the single case d ≤ q ≤ p ≪ n.To simplify the demonstration of this approach, we make the assumption that U = I d 0 , so that U contains the first d standard basis vectors. We introduce the idea of uplifting (inspired by the work in [18] ) the matrices U, W to the dimensionality of V. We use the same hyperparameters as all previous experiments, measured with respect to a full PCA decomposition which acts as a surrogate for the true subspace.First, since U is an orthonormal matrix, it lives in the Grassmannian Gr(d, n) of d-dimensional subspaces of R n . Similarly, W ∈ Gr(q, n). Our goal is to embed U and W into Gr(p, n). Once we do this, we can again calculate the macroscopic states we are interested in. To do this, we use the following map:EQUATIONThis produces a new matrix Ū ∈ Gr(p, n). We can perform a similar trick with W to obtain a matrix W . The important details about this uplifting trick are the following: (1) Due to the construction, we preserve orthonormality of all the matrices, (2) the subspaces of interest are found as the first d columns of the matrix Ū and the first q columns of the matrix W, and (3) the analysis of the diagonal case is unchanged under this uplifting (In the diagonal case, there is no interaction between the different dimensions, so we ignore the other dimensions. In the non-diagonal case, these additional dimensions only provide minor noise, and so don't affect the training at all).Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We are careful about specifying choice of hyperparameters, training algorithms, and information about the data. We analyze a wide range of hyperparameters in our paper.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
real image subspace learning: 
 In order to demonstrate the practicality of this analysis, we test our approach on the MNIST [19] and Olivetti Faces [20] dataset, and compare our approach with the single-feature discriminator from Wang et al. [6] . Here, we include some qualitative results regarding the learned features, and provide a quantitative analysis on the performance differences between the multi-feature and single-feature discriminators. We include the Olivetti Faces results in Figure 4 , and the MNIST results can be found in Appendix A.To perform these visualizations and measure performance, we first perform PCA on the entire dataset and extract the top K (16 or 36) features. We then use this as an approximation of the true subspace U, which allows us to compare the distances. We then track the Grassmann distance between the true and learned subspaces for both the multi-feature and single-feature approaches. The Grassmann distance between two d-dimensional subspaces of an n-dimensional space is given byEQUATION)where the θ i are the principal angles between the subspaces. Here, a lower distance means a better similarity between the subspaces. If the two matrices are orthonormal, the principal angles are the singular values of the cosine similarity matrix, explicitly connected with the macroscopic states.Figure 3 shows the Grassmann distances for the sequential and multi-feature learning cases on the Olivetti Faces dataset. This provides empirical justification on a real dataset, showing first that the phenomenon of faster training identified by the ODE in Figure 1 applies to practical settings as well. Furthermore, due to having no restrictions on off-diagonal entries of the macroscopic states, we see that the results in Figure 2 also apply to practical datasets, since our multi-feature discriminator attains better performance even in less time.8 GANs as a subspace learning algorithmIn the linear setting, GANs attempt to perform subspace learning. However, GANs do not fall into either of the categories introduced earlier. The other subspace learning algorithms all seek to minimize the following loss functionEQUATIONknown as the reconstruction error. This is because the global optima of this loss function is the true subspace itself, and so, we can view this as a prior included in the subspace learning algorithms.GANs do not have such information, and instead seeks to learn the subspace simply through seeing the datapoints. Therefore, we can consider GANs to be a third type of subspace learning algorithm, which we call the generative algorithms. We seek to understand how well the GAN model is able to learn a subspace compared to the existing subspace learning algorithms. We compare both analytically using the derived ODEs, as well as empirically on synthetic and the MNIST dataset, in order to see under what circumstances GANs learn a subspace at a comparable rate.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Much of our analysis is performed theoretically, based on ODEs that are deterministic. Some of the experiments are meant for producing visual results such as learned features, and for these, it does not make sense to consider statistical significance. Furthermore, the premise of our paper deals with convergence in high dimensionalities to a deterministic result. Therefore, for the experimental results we do provide, due to the high dimensionalities involved, we can expect results very close to the deterministic result. Because of this, error bars would not provide meaningful information and instead would simply clutter our charts.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: All of our experiments do not need any compute setup, and is doable using a single computer with a single CPU. Therefore, it is not important for us to provide these specific details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
learned features: 
 Figure 6 in the Appendix compares the features learned by the GAN model to the features learned by Oja's method. Both models are initialized to exactly the same weights, and trained on the same data at the same time, for a single epoch. For the GAN model, we use the same hyperparameters as the previous experiments above. For Oja's method, we used a learning rate of 0.1, which experimentally we found to produce the best results. We can clearly see that the features learned by the GAN model are more meaningful and more clearly resemble the true data, while most of the features that Oja's method learns aren't very interpretable. This suggests that because the GAN needs to be able to generate the images, this acts as a form of regularization on what types of features are learned. 
conclusion: 
 Our investigation into single-layer GAN models through the lenses of online subspace learning and scaling limit analysis has provided valuable insights into their data subspace learning dynamics. By extending our analysis to include multi-feature discriminators, we've unearthed novel phenomena pertaining to the interactions among different features, significantly enhancing learning efficiency. This advantage is particularly pronounced in scenarios of near-zero initialization, where the generator achieves higher maximum and steady-state performances compared to the sequential discriminator. Moreover, the interaction between dimensions enables the generator to closely match variances across dimensions, a feat unattainable in the sequential scenario. In the context of subspace learning, we see that in higher noise levels, the GAN is able to more consistently outperform Oja's method on a wide range of generator, discriminator, and Oja learning rates.Introducing an uplifting method for analysis in arbitrary dimensionalities enables us to better model uncertainties inherent in real-world subspace modeling. Practical validation on the MNIST and Olivetti Faces datasets reaffirms the applicability of our theoretical findings, underscoring the superiority of overparametrization in single-layer GANs over data availability. This prompts intriguing avenues for research in multi-layer GANs, probing whether similar phenomena persist in more complex architectures. Exploring these directions holds promise for further advancements in the field. Finally, we observe that GAN models excel in acquiring a more meaningful feature basis compared to Oja's method when applied to the real-world datasets, which we attribute to their ability to generate new data samples.Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: All data and models discussed in this paper do not have any high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes]  
 
 Appendix: A.1 Comparisons with Single-Feature Discriminator and Oja's MethodIn order to demonstrate that our approach works in more practical settings, we train our model on the MNIST dataset. Then, in order to understand what the model has learned about the dataset, we compute the SVD of the generator weights V, and plot the left singular vectors. Each of these vectors correspond to a single feature learned by the model, and so viewing these will help understand the model performance. Finally, we perform the same tests using the single-feature discriminator, to demonstrate the effects of sequential vs non-sequential learning of features.Our theory and development in the paper operated under the assumption that we knew everything about the true subspace. While this is not possible for these image datasets (since we cannot determine the true subspace U or the distribution of c), we can still use the same assumptions and model structure. Therefore, the generator still samples a c from a standard Gaussian distribution, and the choice of covariance and noise levels are determined through testing.The dataset is flattened into a 1 × 784 vector, so our ambient dimension n = 784. For our multi-feature model, we train for a single epoch. For the sequential discriminator, we train for 5 epochs. We focus on the d = 36 case, although it can be further scaled up as necessary. Through testing, we fix the covariance matrix Λ = 5 * I d and η G = 1. We use a generator learning rate of τ = 0.04 and a discriminator learning rate of τ = 0.2. While the multi-feature discriminator is able to learn good representations of all 36 basis elements as seen in Figure 5 , the sequential discriminator is unable to learn even half of them in the 5 epochs. As can be seen, the last 18 basis elements are just noise.This scaling becomes very problematic as the number of features increases. Even with just 36 features, a small amount given modern datasets, such a model requires significantly more training and is still unable to perform as well as the multi-feature model.Finally, we provide a comparison of the GAN learned features with the Oja's learned features in Figure 6 . It can be seen that most of the GAN features are more visually representative of the dataset compared to Oja's method. Figure 7 contains a comparison of the Grassmann distances for the multi-feature and single-feature cases. Even after 5 epochs, the sequential discriminator still has a much higher Grassmann distance than the multi-feature model, even though it has seen 5 times as much data. Specifically, after one epoch of training, the multifeature discriminator has a distance of 2.46, while the single-feature Figure 6 : After training both GAN and Oja's method on the MNIST dataset with 16 features, we use SVD to extract their learned features, and compare them here. We can see that despite Oja's method learning quicker than the GAN model (seen in previous analysis), the GAN features learned are a better representation of the data, while most of Oja's features do not resemble the data. Note that Oja learning an 8 as the first feature is due to the order of training samples seen.discriminator finishes with a distance of 3.17, showing a significant gap. We tested with up to 20 epochs, but saw no improvements for the sequential discriminator past 5 epochs. We also see an example of the training outcomes predicted by the ODE in Figure 7 . Specifically, our choice of learning rates τ = 0.04, τ = 0.2 and noise level η = 1 is seen in Row 1, Column 2 of Figure 1 , and we see the expected result of the generator oscillating around its steady state.Figure 7 : In the first figure, using PCA on the MNIST dataset, we obtain the top 16 features of the data, and use these features as an approximation for the true subspace U. We then track the Grassmann distance between the learned subspace and this approximation, for both our model trained for 1 epoch, and the sequential discriminator trained for 1 and 5 epochs. It is clearly seen that our discriminator learns much faster than the sequential discriminator, while at the same time obtaining a much lower distance, even when the sequential discriminator has sees 5 times as much data.This proof mirrors the proof of Theorem 1 in [6] . However, for completeness, we re-state the key results and sketch the proof here.The proof relies on the following result, found in [13] :Lemma B.1. Consider a sequence of stochastic processes {x(n) k , k = 0, 1, 2, • • • , ⌊nT ⌋} n=1,2,••• with some constant T > 0. If x (n)k can be decomposed into three partsEQUATIONsuch that (C.1) The processk k ′ =0 ρ (n)k ′ is a martingale, and E||ρ(n) k || 2 ≤ C(T )/n 1+ϵ1 for some ϵ 1 > 0; (C.2) E||δ (n) k || ≤ C(T )/n 1+ϵ2 for some ϵ 2 > 0; (C.3) ϕ(x) is a Lipschitz function, i.e., ||ϕ(x) -ϕ(x) ≤ C||x -x||; (C.4) E||x (n) k || 2 ≤ C for all k ≤ ⌊nT ⌋; (C.5) E||x (n) 0 -x * 0 || ≤ C/n ϵ3for some ϵ 3 > 0 and deterministic vector x * 0 , then we have ||xEQUATIONwhere x(t) is the solution of the ODEEQUATION)The relevant stochastic process is the macroscopic states introduced in Section 4. The macroscopic states are decomposed asEQUATIONNote that our macroscopic state can just be written as an n 2 dimensional vector, and it is equivalent to using the Frobenius norm in the conditions above. We seek to show that this decomposition satisfies the conditions (C.1) -(C.5).Immediately, Condition (C.5) is satisfied by the assumption (A.5) for the theorem. Additionally, (C.3) is satisfied by assumption (A.3) and Lemma 4 in the supplementary material of [6] .Next, we slightly modify Lemmas 2 and 7 in the supplementary material of [6] for our case. Lemma B.2 (Lemma 7 of [6] ). Under the assumptions (A.1) -(A.6), given T > 0, we haveEQUATION)The proof of this follows exactly from Lemma 7 of [6] . Lemma B.3 (Lemma 2 of [6] ). Under the assumptions (A.1) -(A.6), given T > 0, we haveEQUATION)Proof. We show that this holds for a fixed i.First, we know thatEQUATION)whenever γ = 2, 3, 4, due to boundedness of h, f, f . Additionally, we can writeEQUATION)Combining both of these, we get thatEQUATION)which follows from assumption (A.4). Similarly, we getEQUATION)Combining these for both terms and iteratively applying it, we getEQUATION)Then, due to assumption (A.4), we get the required result. Proof. We show that the expected norm squared of each macroscopic state is less than some C(T ).The cases of P k and S k are proven in Lemma 3 of [6] , and require no changes. Additionally, by our assumption (A.6) that the matrix W k is orthonormalized, we know that Z k = I d , and so the requirement is trivially satisfied for Z k . Thus, it remains to show this for Q k and R k . We show this for Q k , and R k follows similarly.EQUATIONas required.Lemma B.5. Condition (C.2) above is satisfied, meaning that for all k = 0, 1, • • • , ⌊nT ⌋, and for a given T > 0, we haveEQUATION)Proof. To prove this, we can split it into five parts, one for each of the macroscopic states. For the macroscopic state Z k , this just requires showing thatEQUATION)But the left side is just zero, since Z k = I d for all k. Thus, this is trivially satisfied.For the macroscopic state P k , we want to show thatEQUATION)However, from the gradient of our update equation for V , averaging over ck , ãk we see thatEQUATIONMultiplying both sides by U T on the left, we getEQUATIONBut then, the left side of the equation we wanted to show is just zero, and so the inequality is satisfied. Applying a similar process to the update equation for W k , we want to show thatEQUATIONand by averaging over ck , ãk , c k , a k and multiplying by U T on the left, we getEQUATIONAgain, this results in the left side of the expression we want to show just being zero. Finally, we show the result for S k . The case for R k follows similarly to the previous results.Using the property thatEQUATIONand averaging over ck , ãk , we getEQUATIONThe second term in the sum above has expected normEQUATIONThis concludes the proof.For condition (C.1), the requirement of being a martingale is automatically satisfied by construction. To show that the remainder of condition (C.1) is satisfied, it suffices to prove thatEQUATIONEQUATION)Proof. We can break this up into each of the 5 macroscopic states separately. As before, doing this for Z k is trivial. We show this for P k and Q k , and the rest follow similarly. For P k , we getE||P k+1 -P k || 2 ≤ Cn -2 E ||Q k ||E k ||c 2 2k + ||P k || 2 , ≤ Cn -2 E 1 + ||Q k || 2 + ||P k || 2 , ≤ C(T )n -2 .(42) This finishes the proof for P k . For Q k , we haveEQUATION)where in the last line, we used the previously calculated values for E||Z k || 2 and E||S k || 2 . The values f k and f2k are the values of f = F ′ and f = F ′ evaluated on the corresponding inputs.The conditions for the rest of the macroscopic states can be shown in the same way.Given the previous lemmas, the proof of the theorem then follows immediately from Lemma B.1.1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our claims made in the abstract reflect our analysis, and figures throughout our text support our claims. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: Our work focuses on theoretical results and analysis. We use our own data model as well two freely available and widely used datasets (MNIST and Olivetti Faces). Our work does not involve new data or interactions with humans. Additionally, there are no legal implications for our work.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: While models such as GANs do have important societal impacts, due to the theoretical nature of our work and the toy GAN model used, we do not feel that any of these societal impacts are directly relevant to our current work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: **Summary of the paper:**
The paper builds on Wang et al’s work [6] to use single-layer GANs for subspace learning and extend it to a possible use of multi-feature discriminator with an unknown number of subspace dimensions. It further analyzes the improved convergence speed of such multi-feature discriminator compared to a sequential learning of a single feature discriminator. It demonstrates improved results over single-feature discriminators both in the quality of obtained bases and the empirical convergence time on several toy as well as real-world datasets.

**Summary of the reviews:**
Some reviewers appreciated the novelties in studying single-layer GANs dynamics for subspace learning. They further appreciated the empirically improved convergence speed and more successful modeling of more complex datasets compared to the baseline work [6], thanks to the multi-feature discriminator. They generally appreciated the clarity of the presentation.

On the other hand, they had concerns regarding (1) the technical novelty in light of Wang et al’s work [6] from which the formal framework of this work is borrowed, (2) motivation of working with single-layer GANs when alternatives such as VAEs and deeper GANs have been used for non-linear subspace or latent feature learning, (3) lack of more complicated datasets and absence of a proper quantitative analysis of the resulting bases.

**Summary of the rebuttal and discussions:**
The authors tried to clarify several points including (1) the formal novelties on top of [6], (2) how to preserve the orthonormality of the bases during SGD optimization, (3) the focus of the work and lack of comparison with non-linear VAE and GAN subspace learning. They further provided additional experiments on a face dataset.

The reviewers did acknowledge the theoretical novelties, on top of [6], after the discussion but not all were convinced about the significance of them. However, they all appreciated the empirical improvements, specifically for the new face dataset. 

**Consolidation report:**
This work formally extends the study of Wang et al. [6] to the case of non-sequential learning of multiple (unknown number of) features and for an arbitrarily large number of dimensions in the discriminator which can potentially improve its expressivity (and convergence time) for more complicated datasets. On the empirical side, despite the lack of an objective quantitative analysis (save for Grassmann distance), the qualitative bases visualization for the extension (at relatively early epochs) look more convincing or at least visually pleasing compared to Oja’s and fundamentally better than (non-converged) single-feature GANs. They have validated their method on real-world datasets that was not done in [6]. The real-world results further show the empirical significance of the extension in the learning of the complex bases of a face dataset as well as its convergence time. 
While the reviewers eventually did not reach a point to confidently vouch for the acceptance and one reviewer remained leaning towards rejection, all reviewers eventually acknowledged the theoretical novelty and empirical improvement over the prior work of [6]. The question was more a matter of the significance of these two contributions.

The AC believes the paper has taken significant steps from [6] and despite the simplicity of the setup (single-layer GAN) the line brings about detailed formalisms that can lead to further works on both feature learning and analysis of GANs training dynamics; both of which are important fields. 

**Recommendation:**
The AC sides with the majority of reviewers and leans towards acceptance. In a camera ready version, the authors are encouraged to:

- shorten the background of [6] so that they are able to more clearly, and at more length, present and discuss the details of their own formal contributions.
- incorporate the importance of the line of work in the context of other works on studying the training dynamics of GANs, the linear and non-linear subspace learning with GANs and other types of generative networks such as VAEs as well as score-based diffusion models.
- include the new face dataset possibly with a quantitative analysis of their identified bases.

Reviews: review: paper_summary: paper_summary: This work explores the training dynamics of a single-layer GAN model, especially for high-dimensional subspace learning, presented as a novel approach. By connecting the GAN models with analysis to subspace learning, this work compares the effectiveness of GAN-based methods with former approaches e.g., Oja’s method and GROUSE, both theoretically and empirically. The findings in this work reveal that GANs demonstrate a remarkable ability to acquire a more informative basis due to their inherent capacity to generate new data samples or in this case subspaces. The experiments in this work demonstrate subspace learning tasks in a more efficient way compared to the counterparts.
strength_weakness: strength_weakness: - Because the subspaces have to be orthonormal, we need to ensure that Eq. 7 maintains this property when performing gradient descents. How can this property be achieved? Or is this not necessary using the approach in this work? Commonly, this desired property can be achieved by geometry aware constraints applied to gradient steps.
- This work borrows the idea of GAN with a minimax objective between the generator and discriminator. One common issue in GAN is the learning instability (e.g., mode collapse) in the learning stage. This work does not discuss this issue in depth. Is this problem not the case for subspace learning?
- The efficacy of the proposed method is demonstrated on MNIST with promising performance. However, the dataset used in this work is relatively small with limited variances. The proposed method would be more insightful to work on a larger scale dataset e.g., a face dataset (MORPH).
-The claim that the learned features are better than Oja’s method in P9 cannot be justified only from the depicted figure. This claim of learned features has not been quantified in what kinds of tasks and for what purposes.
- The proposed method is limited to the linear setting. Would it be possible to use the GAN design for subspace learning in this method for non-linear subspace learning with more complex types of data? This work would be more insightful to provide relationships with the choice of non-linear functions as well.
questions: questions: Please answer the questions and concerns in weaknesses.
limitations: limitations: There is no special section for limitations. It is hard to view direct negative impacts of this work as this is a core machine learning problem that can be applied to many applications.
paper_summary: paper_summary: This paper focuses on the training dynamics of the gradient-based learning algorithms, and converted into a continuous-time stochastic process characterized by an Ordinary Differential Equation (ODE). Empirical evidence demonstrates the correctness of the proposed method.
strength_weakness: strength_weakness: I am afraid I am not an expert in this ODE area. But still, as a machine learning scientist, I feel the empirical section is lacking the supportive evidence and ablation studies. Maybe adding more comparisons with other GAN-related generative modeling can help better support the assumptions in this paper.
questions: questions: Please could you think of other GAN baselines and evaluations metrics that can be used to strengthen the empirical evidence in this paper? For example the FID score is a popular metric usually used in this generative modeling scenario. Maybe demonstrating that the ODE can lead to similar FID score in order to demonstrates the equivalence between this new model and GAN?
limitations: limitations: It seems to me there is no discussion on limitations.
paper_summary: paper_summary: This paper proposes to learn the subspace from the observations by the GAN model. Taking the one-layer GAN model as the starting point, this paper provides a theoretical analysis from the perspective of training dynamics. Specifically, from the technical side, the proposed method trains both the generator and discriminator by using adversarial loss and subspace regularization.  Different from the original GAN dynamic training method, this paper further proposes an assumption that "the columns of the discriminator matrix W are orthonormal"
The proposed method is evaluated on both synthetic and real-world datasets.
strength_weakness: strength_weakness: I have a big concern about the contribution of this paper.  

1) The reason for using GAN for subspace learning, instead of other frameworks (such as VAE), is unclear. The advantages and drawbacks of this choice should be discussed. Please refer to the questions for the details.

2) This paper provides a theoretical analysis from the perspective of training dynamics (Wang et al. 2018). However, most equations and theoretical results are similar as (Wang et al. 2018), such as equations [1,5,6,7,8,9 10].  Only the assumption that "the columns of the discriminator matrix W are orthonormal" is newly added. However, given that the loss function in (Wang et al. 2018) had already constrained this. 

3) Technically, some GAN-based empirical methods [RW1,RW2] tried to add subspace regularization into the GAN model.  Some methods apply the disentanglement constraints to achieve the identification of the latent space, such as [RW3]. 
Technically,  what is the advantage of the proposed subspace regularization $ tr(H(W^TW))$. 

4) In the experiments, the comparison with VAE-based methods [RQ1,RQ2], the GAN-based empirical methods [RW1,RW2] should also be involved. 

[RW1] Liang, Jie, et al. "Sub-GAN: An unsupervised generative model via subspaces." Proceedings of the European Conference on Computer Vision (ECCV). 2018.
[RW2] Jiang, Hongxiang, et al. "Orthogonal Subspace Representation for Generative Adversarial Networks." IEEE Transactions on Neural Networks and Learning Systems (2024).
[RW3] Xie, Shaoan, et al. "Multi-domain image generation and translation with identifiability guarantees." The Eleventh International Conference on Learning Representations. 2023.
questions: questions: 1) What is the unique advantage of the GAN model for subspace learning? Why not use the VAE-based framework, such as iVAE [RQ1] and PCL[RQ2]. In my understanding, subspace learning is a representation learning method, VAE-based methods explicitly model a mapping from observation to latent variables (encoder), which may be more efficient to use in subspace learning.

2) What is the detailed difference between this paper and (Wang et al. 2018) on the theoretical side? 

3) What is the advantage of the proposed subspace regularization $ tr(H(W^TW))$ over other regularization methods [RW1,RW2] .

[RQ1] Khemakhem, Ilyes, et al. "Variational autoencoders and nonlinear ica: A unifying framework." International conference on artificial intelligence and statistics. PMLR, 2020.
[RQ2] Hyvarinen, Aapo, and Hiroshi Morioka. "Nonlinear ICA of temporally dependent stationary sources." Artificial Intelligence and Statistics. PMLR, 2017.
limitations: limitations: No clear negative societal impact needs to be listed.
paper_summary: paper_summary: This work introduces a simplified GAN framework to learn the subspace of a spiked covariance model. The authors are able to derive precise training dynamics given specific assumptions and show they correspond to numerical simulations. They also prove that convergence rate is often faster than previous work that used a further simplified discriminator (single vector vs. matrix parameter). Authors extend convergence results to the case where exact subspace dimension is not known a priori.  Lastly MNIST results are conducted that show steady state feature alignment for this GAN method is higher than a standard method (Oja's method).
strength_weakness: strength_weakness: **Clarity:** While most sections are well written, a few components made it more difficult to understand the key contributions of the paper
* The introduction doesn't clearly define single feature vs. multi-feature discriminator learning on lines 34/35
* I'd like to see more discussion of the interpretation of the main theorem (4.3), especially since the ODE dynamics are fairly complex
* Similarly in section 4.2 I'd like a brief discussion on how these microscopic dynamics are useful to analysis
* In Fig. 1 It is unclear the what the difference between blue and yellow lines is (same for red and green)

**Comparison to Previous Work**
* While GROUSE and Oja's method are introduced as alternative subspace methods, it appears as if only Oja's method is used as an experimental baseline
* I'd like to see one or two citations of other GAN stability analysis papers (e.g. something like Which Training Methods for GANs do actually Converge?) and their limitations as opposed to this works analysis

**Experiments:**
* Fig. 2 appears to show steady state feature alignment for GAN and Oja's method, but I would be interested in a comparison of the convergence rates as well.

**Significance:**
* Many assumptions seem hard to extend to the full GAN setting, for example A3 assumes an uncommon discriminator non-linearity and regularization function. A6 assumes the discriminator matrix parameter is orthonormal. 
* Most of this work appears to be extending the results in [Wang et al. 2018] to a slightly more complex discriminator. This is probably my biggest concern, and would like to hear the authors opinion on this difference.
questions: questions: Is Fig 2. meant to include GROUSE comparisons?

Does the generator need to be changed when moving from the spiked covariance dataset to MNIST one?

Does Fig 2 suggest that making the discriminator learning rate as small as possible the best course of action for feature learning?
limitations: limitations: While the authors state their assumptions explicitly, I would like to see further discussion on which assumptions could possibly be relaxed in the future.

Comments: title: Acknowledgement of Rebuttal
Thanks for your rebuttal
Acknowledgement of Rebuttal
comment: Dear Reviewers and Area Chairs,

We sincerely thank the reviewers for the constructive feedback provided during the rebuttal period. Your insights have been invaluable in refining and improving our work. We are confident that presenting our work at NeurIPS 2024 will spark valuable and engaging discussions, paving the way for future research into the application of multi-feature discriminators in more complex models.

Authors
Thank you for the clarification on the multi-feature discriminator. Your explanation makes sense, especially in how the multi-feature approach extends Wang et al.'s work in the single-layer case. I'm curious to know whether this concept could enhance current GAN models like StyleGAN or CycleGAN. This is just a thought out of curiosity, not a critique of your method.

On a related note, I believe your work stands out from the original methods and makes a unique contribution. I will adjust my score to reflect this.

However, I still feel that a more thorough discussion of VAE-based methods, along with some revisions to the writing, would further improve the quality of this work.
We thank the reviewer for their valuable feedback. While we acknowledge and respect their concerns regarding the perceived lack of novelty and the structure of the manuscript, we would like to provide some clarification and context for the work.

- Firstly, we appreciate the recognition that our additional experiments demonstrate the significance of the multi-feature discriminator over the single-feature discriminator. The faster convergence of our proposed method compared to Oja's method and the single-feature discriminator on challenging datasets is a noteworthy contribution that we believe advances the state of the art in this area.

- Regarding the concerns about the novelty of our work in relation to Wang et al., we would like to emphasize that our intention was not merely to replicate their findings but to build upon them in a meaningful way. Specifically, our work extends Wang et al. by exploring the application of multi-feature discriminators in GANs and addressing arbitrary dimensions—a significant extension that, in our view, goes beyond incremental improvement. The adoption of equations and sections from Wang et al. was necessary to provide a clear foundation for our contributions and to ensure that our work was accessible to readers who may not be familiar with the previous work. Additionally, we are the first to explore how these theoretical developments can be uncovered in real-world datasets.

- We acknowledge the suggestions made by reviewer regarding the structure of the manuscript. We agree that the essentials of our proposed work could be more prominently highlighted in the main paper, and we are open to restructuring the manuscript to move some of the background information to the Appendix. This would allow us to better emphasize the novel aspects of our work in the main text.

- Finally, we understand that the perceived incremental nature of the work may affect the overall assessment of the reviewer. However, we believe that the contributions of this work—particularly in extending the multi-feature discriminator to arbitrary dimensions and demonstrating its effectiveness on challenging datasets—are both significant and novel. We hope that with the proposed restructuring and the clarifications provided, the reviewer might reconsider their evaluation on the novelty and impact of this work.

Thank you once again for your constructive feedback. We value your insights and hope to address them satisfactorily in our revised manuscript.
I would thank the authors for providing the responses to all concerns.
In particular, additional experiments do show the significance of the multi-feature discriminator over the single feature disciminator. Also, we can observe that compared to the Oja's method and single feature discriminator, the proposed method could lead to faster convergence on the challenging dataset.

I agree to Reviewer 7DBz and N4Fr that the novelty in this work is somewhat incremental against Wang et al. Many equations and sections are borrowed from this source, even though they are explicitly mentioned in the paper. The contribution in this work about multi-feature discriminator GAN is not highlighted due to major statements and equations adoption from Wang et al. in the main paper. I agree with Reviewer 7DBz suggestions that the repetitive and adopted information should be included in Appendix, and the authors could keep the essentials of the proposed work in the main paper. Unfortunately, in the current version, insightful and distinctive insights and analyses are located in Appendix e.g., Off-diagonal simulation, comparison results between single and multi-features GAN.  

The position of this work is clear that the adoption from Wang et al. is not meant to reinvent the wheel but to provide backgrounds and further analysis of the proposed method. Specifically, this work extends the work of Wang et al. to multi-feature cases and arbitrary dimensions. Considering that the work is fairly incremental and the manuscript needs improvements, I would be lukewarm about this work and keep my current rating.
We thank the reviewer for the response, but we respectfully disagree with the assessment that the submission lacks novelty.

The reviewer argues that the technical extension of the multi-feature discriminator to be incremental and lists multiple works that propose to use multiple discriminators in practical use. In contrast, our work introduces a new perspective on the use of single-layer GAN model by leveraging a powerful multi-feature discriminator on subspace learning problem. There is a distinction between multi-feature discriminators and multiple discriminators. We believe that there may be a misunderstanding in regards to what a multi-feature discriminator is, which we would like to clarify. The former is a single discriminator which learns multiple features of the true subspace at once, while the latter has multiple discriminators, each of which could be single-feature or multi-feature, with each discriminator focusing on a different part of the learning task. The listed works have nothing in common with our multi-feature discriminator approach.   

We stated our perspective on the VAE models before. While it is possible to view them from a subspace learning point of view, our goal is to understand how training dynamics of GANs is  compared to other traditional subspace learning methods. We do not simply want to compare based on empirical results and specific datasets, but also providing a theoretical comparison using the derived ODEs characterizing training dynamics. As mentioned in our previous response, the only such ODE existing for any VAE models, found in (Ichikawa et al. 2023) , is far too complicated to realistically compare with Oja's and the GAN models.

Finally, we would like to restate our contributions as follows:

1. We frame GANs as a new type of subspace learning algorithm, and systematically compare that to traditional methods like Oja’s and GROUSE based on training dynamics.
2. We demonstrate the speed and accuracy benefits of a multi-feature discriminator over a single-feature one, particularly in learning the true subspace.
3. We introduce an uplifting method to analyze cases where the true subspace's dimensionality is unknown, making the analysis more applicable to real-world scenarios.

Overall, we believe our work aligns with the high standards of NeurIPS, and we hope the reviewer will reconsider the novelty and significance of our contributions in this light.
Thanks for the further clarifications from the authors. I do believe your analysis allows to gain insights into the GAN model using ODEs and contributes to the theoretical support of GAN models. I believe these theoretical contribution can further inspire more thoughts around generative training. I therefore decide to increase my score.
Thank you for taking the time to carefully consider my concerns. I apologize for the delayed response.

I agree with Reviewer N4Fr that the technical extension of the multi-feature discriminator appears incremental. There are studies that support the use of multiple discriminators to enhance GAN models, such as [1] and [2]. Additionally, employing different discriminators for extra guidance has been explored in works like [3].

Additionally, I recognize that the theoretical subspace analysis introduces a new point over Wang et al. 2018. My concern here lies in the distinct difference this approach has compared to VAE-based frameworks. The authors argue that VAE-based methods focus on non-linear models, which they suggest is an unfair comparison. However, I believe that the linear case is simply a special instance of non-linear functions. Analyzing non-linear settings is indeed more challenging, and single-layer or linear cases may not fully reflect real-world applications.

Minor suggestion: If all equations are identical to those in previous work, it would be beneficial to remove them and simply provide a reference. This could help draw more attention to the unique contributions of the paper.

Since these concerns remain open, I will maintain my current score.


[1] Choi, Jinyoung, and Bohyung Han. "Mcl-gan: Generative adversarial networks with multiple specialized discriminators." Advances in Neural Information Processing Systems 35 (2022): 29597-29609.
[2] Cai, Zhipeng, et al. "Generative adversarial networks: A survey toward private and secure applications." ACM Computing Surveys (CSUR) 54.6 (2021): 1-38.
[3] Ma, Cheng, et al. "Structure-preserving super resolution with gradient guidance." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
I thank the authors for their detailed response. In particular, I appreciate the addition of experiments focusing on convergence rate, and clarification of grouse/oja's method baselines. Adding the author's response about the Figure 2 learning rates to the experimental discussion would also be useful. 

I see the other reviewers were also concerned about novelty compared to the Wang et al. 2018 work. While I still think the multi-feature discriminator is a fairly incremental change over a single feature, I agree the authors' analysis for the case of unknown subspace dimension is enough to distinguish it.

Given the promised changes I am willing to increase my score by a point. I am interested to see reviewer 7DBz's thoughts on these changes as well.