On the Consistency of Maximum Likelihood Estimation of Probabilistic Principal Component Analysis
Authors: Arghya Datta
Sayak Chakrabarty
Venue: NeurIPS 2023 poster (2023)
Keywords: maximum likelihood estimate
non-identifiability
Redner approach
quotient topological spaces
consistency

Abstract: Probabilistic principal component analysis (PPCA) is currently one of the most used statistical tools to reduce the ambient dimension of the data. From multidimensional scaling to the imputation of missing data, PPCA has a broad spectrum of applications ranging from science and engineering to quantitative finance.\\

Despite this wide applicability in various fields, hardly any theoretical guarantees exist to justify the soundness of the maximal likelihood (ML) solution for this model. In fact, it is well known that the maximum likelihood estimation (MLE) can only recover the true model parameters up to a rotation. The main obstruction is posed by the inherent identifiability nature of the PPCA model resulting from the rotational symmetry of the parameterization. To resolve this ambiguity, we propose a novel approach using quotient topological spaces and in particular, we show that the maximum likelihood solution is consistent in an appropriate quotient Euclidean space. Furthermore, our consistency results encompass a more general class of estimators beyond the MLE. Strong consistency of the ML estimate and consequently strong covariance estimation of the PPCA model have also been established under a compactness assumption.

On the Consistency of Maximum Likelihood Estimation of Probabilistic Principal Component
Analysis
Arghya Datta"
Sayak Chakrabarty Northwestern University Department of Computer Science
Université de Montréal
Department of Mathematics and Statistics
Abstract
Probabilistic principal component analysis (PPCA) is currently one of the most used statistical tools to reduce the ambient dimension of the data. From multidimensional scaling to the imputation of missing data, PPCA has broad spectrum of applications ranging from science and engineering to quantitative finance.
Despite this wide applicability in various fields, hardly any theoretical guarantees exist justify the soundness of the maximum likelihood (ML. solution for this model. fact, is well known that the maximum likelihood estimation (MLE) can only recover the true model parameters up rotation. The main obstruction is posed by the inherent identifiability nature of the PPCA model resulting from the rotational symmetry of the parameterization. To resolve this ambiguity, we propose a novel approach using quotient topological spaces and in particular, we show that the maximum likelihood solution is consistent an appropriate quotient Euclidean space. Furthermore, our consistency results encompass more general class of estimators beyond the MLE. Strong consistency of the ML estimate and consequently strong covariance estimation of the PPCA model have also been established under a compactness assumption.
Introduction
In the era of big data, principal component analysis (PCA) is a standard dimension reduction tool frequently used in exploratory data analysis. PCA is however not statistical model, implying that uncertainty quantification is not available. Uncertainty quantification is desirable in some situations, like when principal components have an interpretation. Probabilistic principal component analysis (PPCA) is a model introduced by Tipping and Bishop [1999] to remedy the situation. It consists of an additive model with a normal noise term. Let p > q be two given positive integers. The PPCA RR denotes random vector that is observed after
model can be written as J. = Wz + e, where a linear transformation W is applied to a latent (hidden) variable E RR9 up to gaussian noise (0 o² Ip). We assume that data is centered, meaning has zero mean. In practice, the latent variable is not observed and is conventionally assumed to be normally distributed. Given a set of n data points observations {II, ... Zn,) C R2, the two unknown parameters W and o² are estimated by maximum likelihood estimation (MLE).
PPCA is very wnol- and pervasive technique used in dimension reduction of high-dimensional data and used in figuring out the relevant dimensions in large-scale data mining. It must be said that
sayakchaarrbbarty2255ee n.northwestem.cch
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the primary goal of PPCA is not always to perform better than traditional PCA but open up series of possibilities for future extensions for further analysis. PPCA also enables comparison with other probabilistic techniques. Namely, it is possible to quantify the uncertainty in parameters W and x² through a Bayesian reformulation [TTppiigg and Bishop, 1999]. One can also consider modelling the error terms using heavy-tailed distributions (such as Student-t) to achieve robustness [Gai et al., 2008J. These models have appealing properties; in particular, the quality of the information retained from the original data is similar to that of the original PCA with added flexibility.
Despite the wide applicability, nothing is known about the consistency of the maximum likelihood (ML) estimation of the PPCA. The main obstruction is posed by the identifiability of the PPCA model over the parameter space (W,o²) M(p.q) x R+, where Mip, q) and R_ denote the set of all p x q matrices and the set of positive real numbers respectively. One can restrict the attention only to rank q matrices, but as we shall see, doing so will not significantly make our goal any harder (or easier) to achieve as the MI. estimation of W has rank q. It can be readily seen that our model induces a Gaussian distribution on the data points I ~ NO, WW + I which is be used for the maximum likelihood estimation. From a frequentist point of view, we now assume that the unknown parameters (W, o²) has a true value which is equal to (Wo, og). Additionally, we observe that the marginal distribution of the data points (00, ww + o21)) used for the ML estimation remains invariant under any translation of W by an orthogonal matrix R since (WR) (WR) ww As can be seen in the literature it is only possible to recover the maximum likelihood estimate of W up to rotations [Tipping and Bishop, 1999]. This lack of identifiability puts an immediate roadblock to directly applying Wald's consistency theorems [Wald, 1949], a well-known method to guarantee the consistency of maximum likelihood estimate.
turns out that the quotients of the Euclidean spaces and in particular, the quotient of the space M(p.g) q) x R2+ C TRxx x R is the natural topological object to consider to talk about the consistency results of the PPCA model. In a simplified situation when a² is known, heuristically, the quotient space treats all the points Wo and its rotational translates WoR as a single entity by labelling them equivalent and therefore throwing all of them into a single equivalence class. Therefore, the parameter space for W, i.e. Σ Míp, q) becomes a set of equivalence classes ww" or equivalently W = WR, for some orthogonal matrix R. In this setting, we thus get rid of the identifiability issue by passing through the quotient of Rxx4 x IR via the proposed equivalence relation. We then study the consistency of the maximum likelihood in this space. The formal details about the definition of quotient space by a given equivalence relation can be found in Section 4.
E { W Mi[p, q)} through an equivalence relation W' E [W] iff W'w"
The above approach is inspired by Redner [1981] where the author outlines way to extend Wald's consistency theorems [Wald, 1949] under the non-identifiability assumption. We would like to mention that, even though passing through the quotient topological space eliminates the rotational ambiguity of the ML estimate by definition, it introduces many further technical issues. To the best of our knowledge, all the subtleties have not been properly addressed in Redner's work even though the author applies the proposed methodology to analyze the consistency of ML estimates of mixture models. One of the key things that is not explicitly mentioned in their work is that the underlying metric space structure of the quotient topological space can be substantially different from the natural topological structure of the quotient space itself and the topology generated by the open balls with the induced metric in quotient space is in general, not same as that of the usual quotient topology. Thus related geometric properties, for instance, the notion of convergence and other regularity properties of the parameter space become more delicate to analyze. We address all these underlying issues carefully later in section
To the best of our knowledge, no previous research work has attempted to justify the wide usage of the ML estimate of the PPCA model because of the inherent identifiability nature of the model parameters. To resolve this challenge we propose novel framework to see the PPCA model through the lens of an appropriate quotient of an Euclidean space. This perspective not only allows us to formalize the problem precise mathematical terms but also helps us resolve the dispute caused by the non-identifiability.
Considering from the point of novelty, the contributions of this paper are two-fold: we prove that the maximum likelihood estimates (W, .²) are consistent, i.e., (W, 0²) + (Wo, a0²) in probability

---

in the appropriate quotient Euclidean space. In fact, our consisteney results are valid for a general family of estimators that in particular, contains the maximum likelihood estimator. Furthermore, strong consistency, i.e., an almost sure convergence of the MI. estimates to the true parameter is also deduced when the parameter space is compact.
The rest of the paper is organized as follows: section 2 outlines a brief description of the necessary background followed by comprehensive discussion of relevant research works. In section 3, we formally introduce the PPCA model, and the ML estimates of the parameters of interest and talk about the main question we are interested in. In section 4, we provide several basic definitions and mention a few key results from the theory of quotient topological spaces in a self-contained fashion. Section 5 contains all our contributions followed by their detailed proofs in section 6. In section 7, we state Wald's conditions and interpret them within the quotient parameter space. Finally, in section 8, we talk about the impact of our work and put forward few open questions before concluding in section 9.
2 Background and Previous Work
The abstract formalism concerning quotient topological spaces is not foreign to the theory of applied statistics. For in-depth discussion about the application of quotient space into various statistical models, see McCullagh [1999]. Even though, the necessary tools and definition of topological quotients introduced in section 4, an inquisitive reader is certainly encouraged to consult the classic Munkres. The main line of attack used in this paper has been previously introduced in Redner [1981], where the author proves the consistency of mixture models under the standard regularity conditions of Wald's consistency theorems [Wald, 1949]. It is worthwhile to mention that the theoretical justification of using the maximum likelihood estimates in practice has gained lot of recent interest. For instance, Chen [2017] has rigorous treatment of several consistency results with an application non-parametric mixture models.
In the previous section, have already mentioned that PPCA offers many advantages over traditional PCA, particularly in terms of model flexibility and the potential for generalization to accommodate various statistical reformulations. Building upon the recognition of its strength by Bengio et al. [2013], Goodfellow et al. [2016], and Ruff et al. [2021] as one of the most notable advancements in probabilistic models, subsequent research endeavours have leveraged PPCA to achieve remarkable results in various applications. Within the realm of cryo-electron microscopy (cryo-EM) in the field of biology, extensive discussions in the literature have explored its efficacy, as evidenced by the works of Heimowitz et al. [2018], Penczek et al. [2011], Tagare et al. [2015]. Similarly, in the domain of computer vision, the probabilistic model has proven its superiority, as demonstrated by Szeliski [2022] and others. While the precise reasons underlying its success in certain situations remain elusive, PPCA has found widespread applicability in diverse domains. It has found great applications in incremental learning for visual tracking [Lim et al., 2004], in the study of Gaussian processes [Bonilla al., 2007, Lawrence and Hyvärinen, 2005], orthogonal signal correction (OSC) [Lee et al., 2023], weighted nuclear norm minimization [Gu et al., 2017], and in outlier detection algorithms like [Domingues et al., 2018]. Recent advancements have showcased its potential in areas such as few-shot learning [Wang et al., 2020], data mining techniques [Witten and Frank, 2002], and most notably, in finance for exploiting market integration for pure alpha investments (Tzagkarakis et al., 2013]. The broad spectrum of applications where PPCA has exhibited remarkable performance underscores its versatility and its value as a powerful tool in numerous domains.
As we have already noted, the theoretical justification behind the ML estimates used for the PPCA model in practice not always straightforward and has recently attracted the attention of several authors. In a fairly recent work [Chérief-Abdellatif, 2019], the authors provide a consistent estimator of the quantity WWT assuming o² is known. Their work is primarily concerned with the model selection questions statisties using a variational approach. Even though in our work we assume the rank of the matrix rank(W) y to be known and is fixed by the user (this assumption can be removed as long as does not grow with the sample size n)), of independent interest to want to estimate q. In their work [Chérief-Ade Abddeef-,b,lllatif,, 2019], the authors constructed estimator of the

---

quantity to demonstrate how closely their estimated model recovers the true model corresponding the true value of W, they were also able estimate the marginal covariance of the data points and thereby came up with a consistent estimator of WW". One may notice that, in this case, estimating wW" instead of W eliminates the non-identifiabilityy issue. It important to mention that their work does not use the maximum likelihood approach and instead uses variational techniques to construct an estimator of ww"", In this regard, it is relevant to mention that another work [Bouveyron et al., 2011] was able to provide a consistent estimator of the parameter g using maximum likelihood estimation. Finally, we restate that no related previous work attempts to justify the commonly used ML estimates (W,+?) of the PPCA model, perhaps due to ambiguity caused by the inherent non non-identifiable parameter space. In the following section, we begin by presenting the PPCA model and concerned ML estimates originally developed in Tipping and Bishop [1999]
3 Problem Statement
Notation In what follows, we shall always let capital letters denote random variables and their deterministic realizations will be denoted by lowercase letters. Let X2... be a sequence of independent identically distributed random variables where the p.d.f. of the distribution of X, is known except a parameter 9 = (We, o3) in some parameter space θ C RPx0 x RR, for some matrix Wo and positive integers p. q- We assume the distribution of X1 is generated from true parameter Oo. For given parameter = (Wa, oz), we let f(x; 0) denote the normal p.d.f. N(:; O, Wo + p)). Henceforth, the notation [[W]] will mainly refer to the spectral norm, which is defined as the largest eigenvalue of the matrix WWT. is worth noting that in some rare instances, we have employed the Frobenius norm, but such usage has been explicitly mentioned. Unless otherwise stated, all the probabilities and expectations are taken with respect to the true parameter lg. Finally, for m x m matrix A, we denote its characteristic polynomial det nmm A) by (A(A).
The PPCA model: Let us consider that we have access a structured dataset taking the form of a matrix X whose lines are given by F1,..., Tn E Re with a assume that the columns of X are centered (meaning a mean of 0). We assume
large positive integer. We
Ij Wz, + Ess
(3.1)
where Z1y.... Rn, Ell...,,........ i.i.d. the realizations of independent random variables with = (0, Ig) and e ~ of and W is an unknown fixed x q matrix. > 0 being an unknown fixed scale parameter and L, and Lp being identity matrices of size q and p. respectively. The random variable = plays the role of a hidden (latent) variable and that of an error term. We recall that rank(W) q. To estimate the parameters W (often called the loading matrix) and the unknown variance x² of the homoscedastic error term c, Tipping and Bishop [1999] employs the maximum likelihood (ML) procedure and solutions are given as follows
P
W = UA, 1,) and 0²
1/2
Σ
65.
5=9+1
where R is any X 9 rotation matrix, the columns of U are given by the first q eigen vectors of the sample covarinnce matrix S, 1/n Σ nıaf and A, - diag(ō1....,.6) 5,) are the q eigenvalues of
Sg sorted in the descending order. We note that the solution W. in this case, is not unique since R could be any orthogonal matrix which in practice, is often set be Ly. For now, we redistrict our discussion assuming the rank q fixed. For this model, given the parameters (W, we have the following marginal density for the generated data {
pa W o) = f p(ale, W,
= O WW + ,T1)),
where we used the fact that the latent variable a is distributed as ~ NOO, Thus for our problem, the parameter space is
θ == (0:0- 0 (W E R2 x R+}.

---

In the above definition there is slight abuse of notation as the parameter @ is self-referencing to itself. Nonetheless. we retain the notation for the sake of clarity and the case of exposition. We assume the data points { are generated from a true parameter do (Wo, oy), i.e., they are i.i.d. samples from normal distribution whose p.d.f. is given by I [(; 0o). In this work, we aim to prove the consistency of maximum likelihood estimates, i.e. we want to discuss whether (W, - (Wo, oj) holds Pao almost surely or in probability.
Nearly all consistency proofs of well-known statistical models take identifiabiiity for granted. Unfortunately, the PPCA model is not identifiable, since there may be matrices Wa and Wa such that Wo W = W w in which case (a; 0) f(z; Φ) if o.. It is not difficult to see in such cases, W will a rotational translate of the matrix We. i.c., W W for some orthogonal matrix R'. This rotational symmetry in parameterization results in lack of identifiability. In the current work we adopt a similar approach as outlined in Redner [198] ] and subsequently work in quotient topological space of O. Next, rigorously introduce this quotient space framework and develop analytical tools to prove our consistency results.
4 A primer on quotient topological spaces
In this section, we give the definition of a quotient of topological space by a given equivalence relation and discuss several important results pertinent
work.
Definition 4.1 (Equivalence relation). Let S be set. A binary relation on S is said to be an cquivalence iff it is (i) reflexive: z ~ z, Vx E S, (ii) symmetric: x ~ + y ~ x, Vx, y E S and (ii) transitive:
V, y ~ = is evident that an equivalence relation on set induces a partition of the set and vice-versa. The disjoint sets in this partition are called the equivalence classes. If S is given, then we write [s] == fy E S y ~ s} for the equivalence class that contains s and finally the set of equivalent classes (S/ ~) [[s] E S} called the quotient of the set S by the equivalence relation
Vz, y, S.
4.1 Quotient topological spaces and its metrizability
Given a space X and an equivalence relation ~ on X, the set-theoretic quotient X/ ~ (the set of equivalence classes) inherits topology from X, called the quotient topology [Munkres]. It is well known that the surjective map X + X/ ~ defined as x - [a]] is continuous. We now record a useful result from Munkres, which will be important in proving our main results in the forthcoming sections.
Theorem 4.2. If X/X/ ~ are topological spaces stated as above, then for another topological space Y and for continuous map f X - Y with the additional property that ~ x! implies f(x) = fxx), there exists unique continuous map f (X/ ~) Y such that
X
Y
π
X
Figure 1: Illustration of the above theorem using a commutative diagram.
Throughout this paper, we have Y = IR and X is a subset of R2 for some positive integer p. which is a complete metric space and Euclidean distance serves as natural metric on R.. The key idea is to extend this metric from X to X/ ~ and this job is accomplished through an intermediate state, by introducing suitable pseudometric on X. A pseudometric real numbers and the only difference between a pseudometric and metric is that 0(z,, y) 0 need not imply x = y for x, E X. To construct the most commonly used metric on the quotient space X/ we first introduce:
a function on X
to non-negative

---

let X C R be a complete subspace of the Euclidean space and d be the standard Euclidean distance on R.. For z, y E X, we let
d y) == inf (nf(d(11,))) v1) +... + d(zn, yn) | x ~ Fi, yi ~ 11+1(1 < n - 1); Un ~ y)- Then it can be checked that d1 defines pseudometric on X. We now declare x ~ y whenever di (x. y) = 0, and is not difficult to verify the relation is an equivalence. Then we can endow the set-theoretic quotient X/ R with the metric d, which we define as [x), di (z, y) where [z], [v] are the classes of x, y in X/ R. Furthermore, we let X to be the completion of the above space under the metric d, meaning all the Cauchy sequences converge.
As the reader may have noticed already, there are two equivalence relations at play on X, namely ~ and R. Although the construction of the space X/ Re is dependant on the equivalence ~- through the definition of d1, it is not straightforward to identify the relationship between the two abstract quotient spaces X/ ~ and X/ Re. Now is a good time to pause and unveil the primary reason to justify the development of these technical tools. Our initial goal is metrize the topological quotient X/ for a given equivalence relation ~ so that we could extend Wald's conditions / [Wald, 1949] in X/ ~ prove consistency results in this space. The procedure outlined in the previous paragraph provides a recipe for constructing metric d on the space X/ R. The next crucial result, Lemma 4.3 shows that for certain nice equivalence relations ~i, the two spaces X/ and X/ are identical. This result is a key fact that allows us to maneuver the metric construction into X/ ~ We now state the result from Weaver [2018]. The proof of this result is skipped as it is deemed to be technical and it serves a little purpose to our main goal. For a thorough treatment of metrizatiblity of quotient topologiaal spaces, we refer the inquisitive reader to the comprehensive book on Lipschitz algebras [Weaver, 2018].
Lemma 4.3. C is a closed subset of a complete metric space X. Let X/C denote the quotient space of X by the equivalence defined by e if either x y x. y E C then the underlying set Xx and the quotient X/C are the same. The metric on X/C in this case, will take the form:
d([z), [y]) = min(d(æ, y), d(z,, C) + +dy,C") for any [x), bl E EX/C.
In the above statement, d(x,, C") denotes the distance between the point x and the closed subset C defined by d(x, C) == inf d(z; c) and likewise for d(y, C.. An immediate follow-up question of the above result would be: what is the space X and the closed subset C' we use in our context? We now recall two things from the previous section, first the parameter space for the PPCA model which is θ C RPM+1 and second the issue of identifiability, which arose because there could exist two parameters 0, d E θ such that their corresponding densities are equal, which equivalent to WoWT + T W WT + Ip. p- the closed subset defined by
In what follows, using Redner's 's idea [Redner, 1981] we consider
C E WoW/ + = W. + + C
which allows us get rid of identifiability since in the quotient topological space O/C, f(;; [0]) = fr; [oo]) if and only if [0] = [0o] or equivalently 0 ~ 0g by construction. We can now hope for checking Wald's consistency conditions in 0/C to prove (0] converges to [0o] in probability, where θ = (W, 2² denotes the MLE.
In the introduction, we pointed out that despite the extension of Wald's work [Wald, 1949] to quotient topological spaces by Redner [1981], an inadequate amount of attention was paid to the comprehensive understanding of the interplay between the quotient topology and the topology generated by the metric in that treatment. For instance, a priori it is, by no means obvious how one would interpret their assumption in the quotient space O/C without invoking Lemma 4.3. Also, their assumption 5 does not clearly mention what mode of convergence has been used since there are two available notions of convergence, one coming from the quotient topology and another coming from the metric topology generated by d, and those two topologies are not same in general. nontrivial fact that the latter is contained in the former (for a discussion on this topic see James [1990]). However, in most statistical applications, for example in the applications to mixture distributions and clustering [Redner, 1981], we tend to quotient the parameter space by a suitable closed subspace depending on the context, and subsequently the quotient metric takes an easier form as given Lemma 4.3. Therefore, several technical constructions get greatly simplified in these applications.

---

Theoretical Results
In this section, we state our main results. Let {ai IR be sequence of data points that are generated from latent observations [= E IR via the PPCA model defined in (3.1) and recall from section 3 that the data points are assumed be generated from the parameter 0g (Wo, o E θ of the underlying model. Our data points 2; can therefore be viewed as sequence of realizations of i..d. random variables whose true marginal density is given by f(:; fo) = (a; O, W. Wo + og We note a crucial point that for two parameters Φ θ, f (z; 0) f (a; Φ) holds if and only [0] = [o] in 9/C. Therefore, we write fz; 0) for simplicity, instead of cumbersome f(r; [0] when stating results in the quotient space O/C. the spirit of Wald [1949] and Redner [1981], two of our main results are the following:
Theorem 5.1. Let S is any closed subset of θ not intersecting C' then
0z [x; 0o)
P
lup II
=
#ES i=1
Theorem 5.2. Let (2 In) ) be any measurable muasurabllfuuncionn of the observations 21,... Un such that
((x;0a)) ==1 (0);
II
> ffr all n,
1, that is On converges Og in 0/C almost surely, and therefore in then P [lim[0.] [06)) probability.
The above results have elegant geometric interpretations. particular, Theorem 5.1 implies that if Uc is any open neighbourhood containing C' then almost surely all but fimitely many terms of the sequence {0x] land inside Uc. As an immediate consequence of Theorem 5.2 when 1, we obtain:
Corollary 5.3 (Strong consistency the MLE in the quotient space). Let 0, denote the sequence of maximum likelihood estimates of the PPCA model. We then have that
TP [im = [0o] =
that is maximum likelihood estimates converge to the true parameter almost surely, and therefore in probability in the quotient space O/C.
Additionally, using the above corollary we obtain that the covariance for the PPCA process be consistently (strong) estimated under a compactness assumption.
Theorem 5.4. Let O, a compact subset of θ containing the point Og (Wo, a Let On (W, G²) denote the sequence of maximum likelihood estimates of the PPCA model. Then we have
P WW - 6-I, Wo + +oāIo) 1.
Now we talk about possible generalizations above results in light of Wolfowitz [1949]. where the observations {X, are assumed be identically distributed but need not be independent. As noted in Wolfwowitz's s work that Wald [1949] proves strong consistency (i.e. almost sure convergence) and his proof techniques can be extended for dependent random variables {X,] as long as the sequence {X,} satisfies the strong law of large numbers. In the spirit of Wald's work, Wolfowitz [1949] proves the consistency of MI. estimates (i.e. convergence in probability) only under the assumption that the sequence {X;) satisfies weak law of large numbers. The assumption that the sequence } satisfies the weak law of large numbers may seem technical but it offers great deal of flexibility in real life applications due to the following result of Bernstein [Cacoullos, 2012].
Theorem 5.5 (Bernstein). Let {X,} } be sequence of centered random variables. If there exists constant
0 such that for every E N we have Var(X.) & k and if the following condition is true:
lim Cov(X, X,) = 0, |
then the weak law of large numbers holds.

---

We may note that the above result could be extremely relevant in real life applications as intuitively it accommodates all statistical models which allows local dependence among observations, in the sense that X; can significantly correlate with X5 as long as remains sufficiently close to i (i.e., l - jl is small), but the correlation gradually vanishes as l j becomes large. Temporal stochastic processes or time series data with appropriate assumptions could be tailored into the category of those statistical models where Bernstein's result applies. We therefore state our next set of results on the consistency of maximum likelihood estimates of the PPCA model but only under weak law assumption. The following result is an immediate consequence Wolfowitz [1949].
Theorem 5.6. Let the data points {
realizations of identically distributed random variables {X,R whose density is given by fæ; 0o). Let the sequence of random variables {X;} satisfy the weak law of large numbers. Given nj > 0 and a closed subset of θ not intersecting there exists quantity h(S) E (0, 1) which depends only on the closed set S and an integer N(n, S) such that
[(x;0) f(1;;0))
IP
sup BES
1((5) < for every 2 N),
Theorem 5.7. Let the data points {xi be realizations of identically distributed random variables {X; whose density given by f(x;; Let the sequence of random variables {X,} satisfy the weak law of large numbers. Let On(I,,...,.5 In) be any measurable function of the observations T1. Bn such that
II
> c for all n,
(xx 0o)
then [0.] E, [0o). that is 0n converges 6, in 0/C in probability.
Furthermore we also have the two following results in spirit of corollary 5.3 and Theorem 5.4.
Corollary 5.8 (Consistency of the MLE in the quotient space). Let the data points {ax {xx);=1 be realizations of identically distributed random variables (X,) whose density is given by f(x; 0o). Let the sequence of random variables {X,} satisfy the weak law of large numbers. Let 6, denote the sequence of maximum likelihood estimates the PPCA model. We then have that
[0.] - 10o),
that is maximum likelihood estimates converge to the true parameter 0g in probability in the quotient space O/C.
Additionally, using the above corollary we can obtain prove consistent covariance estimation is possible for the PPCA model under compactness assumption.
Theorem 5.9 (Consistent estimation of covariance). Let the data points {xi) be realizations of identically distributed random variables {X,)== whose density is given by (a; 0o)- Let the sequence of random variables {Xi) satisfy the weak law of large numbers. Let On be compact
subset of containing the point Oo = (Wa, og). Let On = -W (W,6²) denote the sequence of maximum likelihood estimates of the PPCA model. Then have
WW + P, WoW. + og
6 Proof of theoretical results
We only present the proofs of Theorem 5.7 and Theorem 5.9. Corollary 5.8 follows from Theorem
5.7. The proofs of Theorem 5.2, corollary 5.3 and Theorem 5.4 will be analogous.
Proof of Theorem 5.7. We fix two quantities Mi, > 0. We need to show that there exists No(m, 72) such that
P (dd((,)) (0.)) > m) < holds for all > No(m, m2). We consider the following closed subset of the parameter space θ which does not intersect
So (0 e d(10), (ol) > m)

---

Using Theorem 5.6 we select 0 < h(m2) < 1 such that
fax; 0) h" (ma) ((x);;0) 2=1
P
sup II PES0
< 1/2 for every > N ((,.)))
for some (m, (22). The dependence on ?/ comes from the choice of the set Su. Next, choose No such that h² (772) < e holds for every n > No. Let No(n, 72) = max No. N ((h. 72)).. Therefore for all No,,, 12)., have that
f 0)
P(d(((.)), > m P
sup
BESo
[mi 0o)
f 0) f(z;;0)
IP
sup
ht" (11)
0E50
1/2
To prove Theorem 5.9 we need the following lemma.
Lemma 6.1. Let Y be complete metric space, and let 1/ θ Y be Lipschitz map such that (0) = (f() whenever O ~ 0'. Then ip lifts an unique Lipschitz map from 0/C to
Proof. The proof of the statement follows from proposition 1.4.4 and proposition 1.4.3 in Weaver [2018].
Proof of Theorem 5.9. We consider the function t/ - RPXP defined by 2/ (WWW 0²) wwT o² L² Since 0g E int(Oo), can therefore consider the restriction vle Oo Rexp, which is Lipschitz function vi is C² and Og is compact. Therefore the lift vi 9o O Rexp is Lipschitz and hence continuous with respect to the topology generated by the metric d.
Invoking corollary 5.8, we have W E Woo Since is continuous with re-
spect to the topology generated by the quotient metric d from the previous lemma, using standard results from probability theory we infer that (
([W (W)
((o..of).
As we saw in the demonstration above, Lemma 6.1 plays a crucial role. We would like to emphasize that Theorem 4.2 would be sufficient ensure the continuity of the lift of & with respect the metric d as the quotient topology needs to be same that of the topology generated by the metric on the quotient space O/C (see the counterexamples subsection in the appendices).
7 Auxiliary Lemmas
Our goal for this section will be to state Wald's conditions [Wald, 1949] to establish the consistency of MLE of the PPCA model and interpret those in the quotient parameter space O/C as stated in Redner [1981]. Unless otherwise stated, throughout this section we will be working in the quotient parameter space O/C introduced in section 4. We recall that f (8,. denotes the unknown parameter such that the true marginal distribution the data points ) is given by the density (æ; 6o) = (0, Wo + 1p For given > O, we let N, NN)( denote the closed ball of radius around [0] in @/C. Following Redner [1981] we begin by introducing the quantities:
sup [(x; Φ) and f" 0,r) max (1, f(x, Or)), f(x, (d))))(((A)
M(x,8) 8) sup [(x; (b) and h" (x, 8) max h(x,
((e))
We are now in position to state Wald's conditions as series of following lemmas whose proofs will imply Theorem 5.1 and Theorem 5.6 as a consequence of the results stated in Wald [1949] and Wolfowitz [1949], respectively.

---

10
Lemma 7.1. The parameter space (0/C,d) and bounded subset of 0/C compact.
d) is metric space with the property that every closed
Lemma 7.2. For each parameter [0] 0/C and for sufficiently small and sufficiently large s, (,) is measurable and the following expectations are bounded
Es, llog (x, 0, r)] < oo and (log (I,, s)] < 00.
Lemma 7.3. Let [0..) C 0/C be sequence. If [f,] [9o]) + do then f(x; 0.) + 0 except on foll set which does not depend on the sequence {[o]]]
Lemma 7.4. For each ([0), (ol) E 0/C x 0/C We have
Eøll [1; < co.
Lemma 7.5. [0:] 00) in 0/C then [x; 0:) (a;; 0) except on Pao Poo-null which does depend on the sequence [[:]]]
The above conditions fairly technical and primarily deal with the geometry of the quotient parameter space O/C. One also needs to be careful as there are two different topologies available in this quotient space which need not be the same. For instance, in Lemma 7.5, the convergence under the if condition [0:] + [0] is required to hold in the topology generated by the metric d which is a stronger requirement than merely asking for convergence the quotient topology. The proofs of the above results are deferred to the relevant subsection in the appendices.
8 Limitations and Broader Impact
In this work, we were primarily focused to establish the consistency of maximum likelihood estimates of the PPCA model in quotient Euclidean space. However, were neither able to provide rate of convergence nor we could say something definitive about the asymptotic distribution of the MLE, and this opens up two avenues for future research. Nonetheless, it is worth noting that research does not present any discernibee negative societal implications.
Conclusion
In this paper, we proposed a novel topological framework provide rigorous justification for the theoretical validity of the maximum likelihood estimation of the Probabilistic Principal Component Analysis (PPCA) model. Although our consistency results hold within quotient space, implying that the true parameter is recoverable that this represents the optimal achievable outcome due to the challenges caused by identifiability inherent in the underlying model. Our result is stated in terms of the abstract framework of quotient topological spaces, but an immediate (and concrete) consequence of our work (strong) consistent covariance estimation of the PPCA process through maximum likelihood estimates, which wasn't 't known before. Our rigorous framework opens doors for applications in many other statistical models where rotational ambiguity (or more generally ambiguity due to closed space or symmetric parametrization) is present. One such example is the matrix factorization problem where one interested in the estimation of two matrices A. B such that the data matrix X AB + noise is one such problem. Our methodology is highly flexible in the sense that does not depend on the statistical model much long there are some regularities in the geometry of the parameter space θ, which makes it even more usable, even for non-linear models such as nonlinear independent component analysis where the observed data is generated vector and is a parameter and the function & is non linear. The methodology we developed in our work for the PPCA model could be readily applied in this case as quotient space construction barely depends on the fact that È is linear for PPCA. The primary focus for us was to build a strong theoretical foundation which was missing the relevant literature. Furthermore, our study offers concise and comprehensive treatment of the theory of quotient topological spaces, with an eye on statistical applications, building upon and expanding upon previous works with enhanced rigour and clarity.
to a closed subset of the parameter space, our results establish
{(2 (== noise, where 2 is latent random

---

10 Acknowledgements
The first author would like thank his supervisors, Prof. Philippe Gagnon and Prof. Florian Maire, for many helpful conversations and continuous support by providing generous financial assistance and excellent working conditions during this project. The authors would like express their gratitude to the anonymous reviewers and area chairs of NeurlPS 2023 for their careful review of the paper and their valuable comments and suggestions, which have greatly contributed to improving earlier versions of the manuscript. The authors would also like to thank Prof. Nik Weaver in the mathematics department at Washington University in St. Louis for an insightful private communication. The first author received financial support from his supervisors and fellowships provided by the Faculté des études supérieures et postdoctorales (FESP) and the Bourse d' exemption at the Université de Montréal during the course of this work. The first author is sincerely thankful to his family and remains deeply indebted to his special friend who goes by the nickname 1915131 for providing support of all kinds unconditionally throughout the duration of the work.
The second author would like to acknowledge the financial assistance and excellent work- ing condition provided by Northwestern University.
References
Y. Bengio, A. Courville, and P. Vincent. Representation learning: review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8):1798 1990))888,888 2013.
D. Bertsekas and S. E. Shreve. Stochastic optimal control: the discrete time case. 1978.
E. V. Bonilla, K. Chai, and C. Williams. Multi-task gaussian process prediction. Advances in neural information processing systems, 20, 2007.
C. Bouveyron, G. Ccleux, and S. Girard. Intrinsic dimension estimation by maximum likelihood in isotropic probabilistic PCA. Pattern Recognition Letters, 1706-1:770611133 2011.
T. Cacoullos. Exercises in probability. Springer Science & Business Media, 2012.
J. Chen. Consistency of the mle under mixture models. STATISTICAL pages 47-63, 2017.
B.-E. Chérief-Abdelle Consisteney of elbo maximization for model selection. In Symposium Advances in Approximate Bayesian Inference, pages 11-31. PMLR, 2019.
R. Domingues, M. Filippone, P. Michiardi, and Zouaoui. A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern recognition, 74:406-421, 2018.
Gai, Y. Li, and R. L. Stevenson. Robust bayesian PCA with student's t-distribution: the variational inference approach. In 2008 15th IEEE International Conference on Image Processing, pages 1340-1343. IEEE, 2008
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
S. Gu, Q. Xie, D. Meng, W. Zuo, X. Feng, and I.. Zhang. Weighted nuclear norm minimization and its applications to low level vision. International journal of computer vision, 21:183-208, 2017.
A. Heimowitz, J. Andén, and A. Singer. Apple picker: Automatic particle picking, low-effort cryo-em framework. Journal of structural biology. 204()):215-2) 2018.
I. M. James. Introduction to uniform spaces, volume 1 144. Cambridge University Press, 1990.
N. Lawrence and A. Hyvärinen. Probabilistic non-linear principal component analysis with gaussian process latent variable models. Journal of machine learning research, 6(1 1), 2005.
G. Lee, E. Sim, Y. Yoon, and K. Lee. Probabilistic orthogonaal-ignaalloorrecc principal component analysis. Knowledge-Bast Based Systems, 268: 10473, 2023.
Lim, D. Ross, R.-S. Lin, and M.-H. Yang. Incremental learning for visual tracking. Advances in neural information processing systems, 17, 2004.

---

12
P. McCullagh. Quotient spaces and statistical models. Canadian Journal of Statistics, 27(3):447-456, 1999.
Munkres. Topology, second edition.
A. Penczek, M. Kimmel, and C. M. Spahn. Identifying conformational states of macromolecules
by eigen-analysis of resampled eryo-em images. Structure, 19(1 (011111552-1599) 2011.
R. Redner. Note the consistency of the maximum likelihood estimate for nonidentifiable distribu- tions. The Annals of Statistics, pages 225-228, 1981.
L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A unifying review of deep and shallow anomaly detection, Proceedings of the IEEE, 109(5):756-795, 2021.
R. Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
H. D. Tagare, A. Kucukelbir, F. J. Sigworth, H. Wang, and M. Rao. Directly reconstructing principal components heterogeneous particles from cryo-em images. Journal of structural biology, 191 (2)2245-662, 2015.
M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3):111-622 1999.
G. Tzagkarakis, J. Caicedo-Llano, and T. Dionysopoulos. Exploiting market integration for pure alpha investments via probabilistic principal factors analysis. 2013.
A. Wald. Note on the consistency the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4):595-601, 1949.
Y. Wang, C. Xu, C. Liu, L. Zhang, and Y. Fu. Instance credibility inference for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12836 12845, 2020.
N. Weaver. Lipschitz algebras. World Scientific, 2018.
I. H. Witten and Frank. Data mining: practical machine learning tools and techniques with java implementations. Acm Sigmod Record, 31(1):76-71) 2002.
J. Wolfowitz. On Wald's proof of the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4):601-602 1949.

---

13
11 Appendix
11.1
Counterexample: Quotient topology and the metric topology on the quotient parameter space are different.
It was pointed out in the introduction that special care is needed when talking about continuity and convergence in probability within the quotient parameter space 0/C as there are two different topologies at play. In fact, a substantial challenge of our work was to meticulously address the technical issues that arise from the interplay between these two topologies. In Redner [1981], was claimed that the MLE converges to the true parameter in the quotient topological space and this follows from the theory of quotient spaces which not true in general, as it untrue that the quotient metric behaves in a similar fashion to that of the original Euclidean metric and therefore it would take a great deal of effort to unambiguously define the right notion of convergence to be used for assumption 5 (and 3) in Redner [1981]. I]. The interaction between the metric topology and the quotient topology depends highly on the equivalence relation on the parameter space O Here is a counter-example highlighting situation where the quotient topology is nontrivial but the topology generated by the quotient metric d degenerate:
Consider the space X = {(z, y) : z.y > 0} - {0,0)) with the usual Euclidean norm, i.., the first quadrant with axes except the origin and define the equivalence relation on X given by: Pı ~ P2 3 Ph= = Xp2 for some À > 0, i.e., the points pI, P2 are equivalent if they are on the same ray. The projection map here is X + X/ ~ is given by (p) [p). Geometrically, it is helpful to visualize the space X/ ~ as a north-east part of the unit circle S which is a subspace of full circle. Under this equivalence relation it is true that di ([x], [v]) = 0 (from the definition of pseudo metric di in section 4) for any two distinct [x], [v] E X/ ~ which can be easily argued by taking x = x/nny/ 2/1 = y/n with N. so that, I ~ I. y1 ~ y and letting n - xc. Therefore the usual metric structure totally breaks down in the quotient space as any two different equivalence class has distance zero between them while in this case the quotient topology non-trivial and we can still talk about convergence with regards to the quotient topology. But to extend Wald's 's condition in the quotient, the space X/ must be endowed with a non-trivial metric space structure, which is impossible in situation like this.
We also note that the quotient space had to pass through pseudometric achieve the desired metric space structure in order to set up the ground for Wald's results to extend. In general, it is very hard to interpret something meaningful out of this intangible metric from a statistical point of view. However, when the quotient is done with respect to closed set, one can provide a concrete tangible geometrical description of the quotient metric space through Lemma 4.3, which in our work has been done in the context of PPCA. The model PPCA is not important here, rather the key part is that: the quotient has to be with respect to a nice enough equivalence (for instance with respect to a closed set, which is the case in Redner's work, our work and numerous other statistical models where identifiablity arises). This key technical but unavoidable construction seems to have gone unnoticed in earlier works. Note that, it was imperative come up with an explicit and tangible description of the metric in the quotient space as that plays a crucial role towards the proof of assumption (Lemma 7.5 in work) in Redner [1981], whose proof was assumed straightforward.
11.2 Counterexample: Theorem 4.2 not sufficient to ensure the continuity of the lift with respect to the quotient metric d.
We pointed out in section 6 that Lemma 6.1 1 is not sufficient to guarantee the continuity of the lift of the map & used in the deduction of Theorem 5.9. Indeed it is possible that continuous (and not Lipschitz) function respect to the metric topology generated by d on the space O/C. Here a counterexample which highlights the situation:
θ + Y lifts to a function ğ O/C + Y which is no longer continuous with
Let X be a subset of the unit square 1]2 consisting of the origin and the points a, = =G.
(), 0). b, = (,.1 n 1), and e, = (2, 1) for > 2. We endow X with the Eu-
clidean metric inherited from [0. 1J2. We declare the following equivalence relation ~ on X: we let an ~ bn for all n, and let y= X + R be the function such that g(0) g(an) = g(h.)) = 0. and g(c,)) = In the quotient space X/ ~ the sequence ([c,]) converges the origin, so the lift g isn't continuous, {(c..) = 1 for every n and g0)) 0. One may note that the function g is not

---

14
Lipschitz in this example since the points } and {crt J will become arbitrarily close as n + co while (g(b,,) g(c,)l ) I. Even though the domain of the function g, in this case, is discrete, one can extend y continuously to [0, 1J², using the Tietz extension theorem (see Munkres) and provide counterexample where the domain an Euclidean space.
We note that & (used in the deduction Theorem 5.9) is not Lipschitz in the parameter space θ due to the presence of quadratic term in W. One way to ensure that 1 is Lipschitz is to assume the existence compact subset O which contains the point Og its interior.
11.3 Proofs of auxiliary lemmas
Proof of Lemma 7.1. This is a direct consequence of the properties discussed in Section 4 and in particular, the fact that the topology generated from the metric d is weaker than the standard quotient topology on O/C. Let K be a closed and bounded set in (0/C, , d) and (Ua JaeA be an open cover for K. Since the set π (K) is closed (by the continuity of quotient map) and bounded in θ, it is compact. Therefore the open cover { (Ua))aA can reduced to a finite sub-cover of π (K) in O. Using the surjectivity of the quotient map, we then see that there is a finite subset F C A such that
K = ক (K)) - (U=
= U aef
aEE
Proof of Lemma 7.2. To ensure the measurability conditions so that the above integrals are well defined we include all the Ps, null sets to the associated Borel sigma algebra as discussed in Bertsekas and Shreve [1978] (p.167, Corollary 7.42.1).
Given a parameter o (W og)) E O, and r > O, we note the function Φ + -p (lr))((()/22)) log der(de W W + +//1)/2 is continuous functions on If ~ d', then the function evaluates to the same value for á and d'. Therefore the function factors through the quotient topological space 0/C by Theorem 4.2. In this case, the map & + -p log(2))/2 - log det ( w + 7) can be realized as a continuous function on the quotient space 0/C. For [0] E O/C fixed, we consider the ball (10). which is compact. Thus by the virtue of continuous functions, the map will have finite maximum in ([0).. We call these maximum M1 (r). The lig-liieldhood log f (x; () is given by
p p2og(2T)) log det W WT Tg 5 T (W WT + -' x,
log / (2; Φ)
2
and thus we have that
Eg, llog f", 0, [.rr]=
(I.
/ log max
1,
sup (d)EN-(10)
f(z; f(1:00 dPoo(o)
{ sup ((O))
log (o)
JR
JRP Mi(r)dPa,
inf xJ (WoW" + - (l))
Mr(r (since WT + T) -1 is positive definite)
Bounding ht (I. s) is a similar task. First we note there exists M > 0 such that f(x; σ) < 1 (using
Wald's assumption 3 Wald [1949] or the next lemma) whenever di d([Ø)), 0) [[]]]| M Pick
sufficiently large such that (o) $ N, ((,)) implies [|[]]]] > M (can be ensured using triangle
inequality [| > [[]6,] [oll [[[o]]]) and thus in this case h (æ, s) and Ellogh h" (z,
evaluates to 0.
Proof of Lemma 7.3. Since d(((.), (0o]) + oc in 0/C implies d(O,, 0o) + oo in (by definition of d in Section 4), we have ||0:]| + 2o by triangle inequality. We aim to show log fa; 0t) -c0. The

---

15
log-likelihood, therefore, is bounded by
p plog((TT)
1 xx" (We W + tb 1p) log Wo W + 1. 1p)
log f(x; 0:)
1):-)) -xx
2
2
2
log det Wo, Wo, (p)
log P_ Wo, w., (o).)
-Wo,
(P-wo, w7, is the characteristic polynomial
log(og, + +)ig)
(Dig) of Wo, W
log o + W
We observe that if ||0:| + co, either [Wa,ll or | (or both) must become arbitrarily large, which in turn implies the required result.
Proof of Lemma 7.4.
Esll log fx;;))
llog
JR" (2lo((2T)
log det dey + +zg1) dPo(s)
+
RD
+ 2
Es (Wo W oz
< (0) + 2 E (W + -
< (0) + (Wo + oz - | |0
Proof of Lemma 7.5. Suppose, C. We observe d( [8) - in O/C implies d(0,, C) - in
O. Since C' is closed set and θ is complete, therefore the sequence {0:) converges to some 0g E C. As - (2;; 0) continuous, we have f(x; 0.) - [z; 0g) [=; 0), this case.
If 0 & C, then there is a neighbourhood Up C θ of 0 such that Un n C = 0 and consequently all but finitely many terms the sequence {0,) belong to Ug. This implies, in this case, d((0.), (0) = d(0,, 0)
for all but finitely many and the conclusion readily follows.


Reviews:
[{"review_id": "iZo4O62INp", "reviewer": "Reviewer_ZSYo", "summary": "paper addresses \"Probabilistic PCA\" which stands for a setup where the vector of p observations x can be written as x = Wz + eps where eps stands for an additive (Gaussian, centered, iid) noise, the matrix W \\in R^{p x q} is an unknown and z in R^q is the other unknown . Only the value of q is known in advance and the goal is to recover W and z and remove the noise. \nThe main challenge is to address the fact that the model is invariant w.r.t. applying a rotation R to W and z: Wz = (W R) (R' z) . Note that  (W R) & (R' z) have the same euclidean norms as W & z. The contribution is to propose an analysis in the quotient space of the vector space by the rotations. ", "questions": "1. There is a broader set of problems where rotation invariance causes trouble, and that set of problem directly related to the PPCA problem considered: all problems where the observations / measurements are of the form <u_i, u_j> or <u_i, v_j>. This comprises PPCA, but also matrix factorization and embedding problems. Can authors discuss how their method can expand to those problems?\n2. Is there a relation between rotation invariance and computational complexity? Can a rotation be a distractor in the gradient steps convergence of a first order method? ", "limitations": "The focus seems to be relatively narrow and implications of the proven results not sufficiently emphasized. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper contains a rigorous consistency analysis of the PPCA problem with respect to rotation invariance", "weaknesses": "The main focus of the paper on the rotation invariance does not seem to be the biggest concern with PPCA or other related (see Questions)  methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{We addressed all the questions and weaknesses raised by the reviewer}$.\n\nThank you for your careful review and valuable assessment. First, we address your questions.\n1) You are correct that `There is a broader set of problems where rotation invariance causes trouble' and we agree that PPCA is just a part of that broad set. Our methodology and framework are readily applicable even in those cases. To continue with your question, let us assume that (for matrix factorization) we are interested in the estimation of two matrices $\\textbf{A},\\textbf{B}$ such that the data matrix $\\textbf{X}=\\textbf{A}\\textbf{B}+\\text{noise}.$\nAssume that $X\\in\\mathbb{R}^{n\\times p},A\\in\\mathbb{R}^{n\\times m},B\\in\\mathbb{R}^{m\\times p}$. Observe that if $(A, B)$ is a solution to $X=AB$ then so is $(AR, R^TB)$, where $R$ is any $m\\times m$ orthogonal matrix. Let $A_0,B_0$ are the true values of $A$ and $B$, respectively. We can try to check Wald conditions on the space $\\mathbb{R}^{n\\times m}/\\sim_{A_0}\\times \\mathbb{R}^{m\\times p}/\\sim_{B_0}$, where $\\sim_{A_0},\\sim_{B_0}$ are equivalence relations defined by $\\sim_{A_0}:=\\{ A' :A'A'^T=A_0A_0^T \\}$  and $\\sim_{B_0}=\\{B': B'^TB'=B_0^TB_0\\}$. Note that $\\sim_{A_0},\\sim_{B_0}$ are both closed subspaces of Euclidean space. Geometrically we are in a nice situation here. In fact, it will work out as long as the likelihood function for this matrix factorization model has some regularities. More generally, if the parameters $u_i,v_j$ are of your interest and the likelihood contains something like $\\langle u_i,v_j\\rangle$, you can always do the above construction.\n2) Authors are unsure how this question is related to our work as we have a `closed form' expression of the Maximum likelihoods in our case stated on page 4 in the description of the PPCA model. We usually apply numerical schemes (like gradient descent) where closed forms are $\\textit{not}$ available. \n\nHaving said that, we respond to the question. If rotational invariance (or more generally model identifiability) is present then that will lead to too many (possibly uncountable) local minima which is bad news. Too many local minima might cause trouble especially if their cost is higher than the global minima of the objective function. But, it seems that even in this case, it is possible to treat those large number of local minimas equivalent with respect to their costs and do a similar kind of analysis but that would mean trying to understand the dynamics of gradient steps convergence of a first-order method on a quotient space (which is non-Euclidean).  It is possible though, to use our framework even in this case. But usually, to understand the dynamics, we need some sort of a smoothness (or a notion of differentiability) assumption beyond continuity which seems difficult on the quotient spaces. It is possible to do differential calculus on quotient spaces if they are manifolds and admit smooth structures. However, we can prove that those quotient spaces are not manifolds. So, standard methodologies from differential geometry do not apply. This seems like an excellent (but considerably harder due to reasons we mentioned) follow-up line of research in the future but for now is not very related to the work we have done.\n\nLastly, we respectfully disagree with the $\\textit{weakness}$ pointed out by the reviewer for the following reasons:\n\n1) Authors understand that their result is stated in terms of abstract topological spaces but it readily implies consistent covariance estimation is possible. Though the PPCA community does not care much about rotational invariance they do care about consistent covariance estimation which was not known before. For instance our result implies $\\mathbb{P}(\\widehat{W}\\widehat{W}^T+\\widehat{\\sigma^2}I\\to W_0W_0^T+\\sigma_0^2I)=1$ This follows from a standard fact in probability theory that $\\mathbb{P}(X_n\\to c)=1\\implies \\mathbb{P}(f(X_n)\\to f(c))=1$ if $f$ is nice (continuous). In our case, the map $[(W,\\sigma^2)]\\to WW^T+\\sigma^2I_p$ can be seen as a continuous map from the quotient space $X/C$ to the space $R^{p\\times p}$ by Theorem 4.2.\n2) As you already observed in the answer to the first question our methodology is highly flexible in the sense that it does $\\textit{not}$ depend on the statistical model much as long as there are some regularities in likelihood function and in the geometry of the parameter space $\\Theta$. This makes it even more usable, even for non-linear models like nonlinear independent component analysis where the observed data is generated as $x=f(z|\\theta)+\\text{noise}$, where $z$ is a latent random vector and $\\theta$ is a parameter and the function $f$ is $\\textit{non-linear}.$ The methodology we developed in our work for the PPCA model could be readily applied in this case as our quotient space construction barely depends on the fact that $f$ is linear for PPCA. It would be an interesting line of work to investigate the connection between the methodologies discussed in our work to that of Zheng et al `On the Identifiability of Nonlinear ICA: Sparsity and Beyond', NeurIPS 2022, where authors recover the latent sources up to an equivalence but using other techniques. \nThere are numerous possible ways of research directions that can come out of this work. The primary focus for us was to build a strong theoretical foundation (that was missing in the literature, Redner(1981)), This allows seamless and flexible applications. Finally, we hope to make abstract topological ideas more mainstream and practical. For your assessment of us not sufficiently emphasising the implication of our work, please refer to the last paragraph of reviewer gPBN's response where we discuss this point. We warmly welcome your suggestions and comments to improve our final version. \n\n$\\textit{We hope we have addressed all comments satisfactorily and kindly request you to revise your score.}$"}, {"review_id": "AIYRWKfzYX", "reviewer": "Reviewer_gPBN", "summary": "The paper discusses consistency of the maximum likelihood (ML) estimation in probabilistic principal component analysis (PPCA). Despite its wide applicability, proving ML estimation consistency in PPCA has been a challenging task because of the non-identifiability of the problem. The author(s) extend the quotient space topology idea presented in Redner (1981) to a more general setting where ML estimator is one such estimator. Also, strong consistency of the ML estimator is proved under a compactness assumption.", "questions": "1. What would be $X_{\\equiv}$ of Lemma 4.3 in the case of PPCA and in what sense $X_{\\equiv}$ and the quotient $X\\C$ are \"identical\" in that lemma? The two metric defined on those two topological spaces are not same- right?\n\n2. I was wondering whether it's not possible to have the consistency result for the MLE in PPCA as a corollary of the results proved in the following\n\nVan der Vaart, A. W., & Wellner, J. A. (1992). Existence and consistency of maximum likelihood in upgraded mixture models. Journal of multivariate analysis, 43(1), 133-146.\n\nI understand the above may be for mixture models but can't we have a corollary for one mixing component which would be the marginal distribution of the data given W and $\\sigma$ in your case.", "limitations": "Yes, the author(s) have adequately addressed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Detailed description of the quotient space topology and the associated metric. The constructive approach was clear and concise.\n2. With standard assumptions on the quotient space, the author(s) derived both weak and strong consistency of the ML estimator in the PPCA model.\n3. Verification of the Wald's conditions [Wald, 1949]  in the quotient space topology with sufficient technical details.", "weaknesses": "1. Some remarks on the Wald's conditions on the quotient space to clarify how those conditions could be interpreted in the quotient space would be useful.\n2. One or two examples to clarify the result in Theorem 4.2 would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{We addressed all the questions raised by the reviewer.}$\n\nThank you for your careful review and meticulous reading and assessment. We are very happy that you asked two excellent questions which we would like to address below:\n\nLet us clarify the picture in depth before answering the question, we recall a few things.\n\ni) We have a topological (and metric) space $X$ which we want to quotient wrt. an equivalence relation $\\sim$.\n\nii) Problem is there is no nice metric structure (in general) on $X/\\sim$, which seems to have gone unnoticed in the previous work by Redner (1981). There is however, a well-known pseudometric (defined after line 208) and the topology this pseudometric generates can be very different from the quotient topology (please see the example provided in reviewer QWak's response). So we have to be very careful when saying things like convergence, and continuity as those concepts crucially rely on which topology we are using.\n\niii) To extend Wald's condition we need a metric space as written towards the end of Wald's original paper. Therefore $X/\\sim$ with a pseudometric is not enough. However this pseudometric can be turned into a metric through another equivalence relation $\\approx$. (which basically treats two points `same' if their distance under this pseudometric is 0 and thus turns it into a metric). \n\niv) Now there are two equivalene relations on $X$, namely $\\sim$ and $\\approx$ ($\\approx$ and $\\sim$ are interrelated as we mentioned in line 216). Therefore there are two possible quotient spaces $X/\\sim$ endowed with a pseudometric and $X/\\approx$ endowed with a metric. Of course, those spaces are related as $\\sim$ and $\\approx$ are related. But we want to work with $X/\\approx$ since that has a metric space structure and we have a hope to check Wald's condition there.\n\nv) Notice $X/\\sim$ (which is the same as $X/C$) is our parameter space where we should be trying to extend Wald's condition but in the previous point, we said we want to work in $X/\\approx$ because $X/\\sim$ might fail to have the desired metric space structure.\n\nvi) It is a non-trivial fact (please see p18, Lipschitz algebras, Nik Weaver) that for some `nice' (in our case which is a quotient by a closed set $C$) equivalence relations $\\sim$, it is true that $\\sim=\\approx$ and the equality here means, they partition $X$ into same equivalence classes. The great news now is that: $X/\\approx=X/\\sim$, implies that the pseudometric on $X/\\sim$ is actually a metric that we crucially exploit.\n\n$\\textit{Now to answer your question}$,\n\n1) The space $X_{\\approx}$ is abstract and does not have a very clear picture in general but in our paper, it is essentially $X/C$ which has a clear picture. You are absolutely right that in general the metric structures are not the same but in our case it is, which is one of the key things for Wald's idea to work. Lastly, we mention that for most statistical applications (like Redner(1981)) we quotient by a closed space, and then we should be in a good situation. However, the reference we provided in point(vi) above outlines a fairly general framework beyond just quotient wrt a closed set, under which $\\sim=\\approx$ holds, and also gives solid counterexamples when any of those conditions are violated which again explains why care is needed to build a correct foundation.\n\n2) Authors are aware of the work: Van der Vaart, A. W., \\& Wellner, J. A. (1992). Our work won't be a corollary of their proven theorems as they assume model identifiability (see their assumption (3.3) in section 3, consistency).\n\nFinally, we cordially thank you again for reading our work in-depth and we will include an example or two along with a brief discussion on how Wald's condition could be interpreted in the Quotient space in the final version. Due to the space constraint of this conference, we had to make a choice to present the topics. With regards to the previous works along this line the authors strongly felt it is highly important to keep things very transparent and build a solid foundation. This is the reason why we kept every detailed argument as a part of the presentation. We had to sacrifice a bit on possible implications of our work, one of which is notably a (strong) consistent covariance estimation of the PPCA model (please see the second last paragraph of reviewer ZSYo's response). However, the argument can get slightly technical and we can elaborate more if you are interested but overall it is an immediate implication of the convergence in the quotient space. Lastly, we very much welcome your further suggestion to improve our final version.\n\n$\\textit{We hope we have addressed all your questions satisfactorily and kindly request you to revise your score.}$"}, {"review_id": "R2Lfl3EPpO", "reviewer": "Reviewer_MhxU", "summary": "In this work, the authors propose a novel topological framework and show that the maximum likelihood (ML) solution of probabilistic principal component analysis (PPCA) is consistent in an appropriate quotient Euclidean space. The consistency results encompass more estimators beyond the ML solution. In addition, the ML solution has been shown to achieve strong consistency when the parameter space is compact.", "questions": "I am not familiar with the techniques used in this work. Does the consistency in the appropriate quotient Euclidean space means that the ML estimate $(\\hat{\\mathbf{W}},\\hat{\\sigma}^2)$ converges to the true parameters $(\\mathbf{W}_0, \\sigma^2_0)$ in probability up to rotation (i.e., $\\hat{\\mathbf{W}}$ converges to $\\mathbf{W}_0 \\mathbf{R}$ with $\\mathbf{R}$ being orthogonal)?", "limitations": "NA", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work seems to be the first work to establish the (strong) consistency result of the ML solution of PPCA. ", "weaknesses": "It would be ideal to remove the assumption that the latent dimension $q$ is known.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{We have addressed all the questions raised by the reviewer}.$\nThank you for your careful review and assessment. \n\n$\\textit{You are correct}$. If we assume $\\sigma^2=\\sigma_0^2$ is known then our result does imply that $\\widehat{W}$ converges to $W_0R$ for some orthogonal matrix $R$, which is to say $\\mathbb{P}([\\widehat{W}]\\to [{W}_0])=1$. However, it is not just an abstract and technical result. In this case, using standard results from probability theory it is possible to argue that: $\\mathbb{P}(\\widehat{W}\\widehat{W}^T\\to\\widehat{W}_0\\widehat{W}_0^T)=1$, as the function $W\\to WW^T$ is continuous. This is a concrete result (and can be stated without referring to the quotient space) which says that the true covariance matrix $W_0W_0^T+\\sigma_0^2I$ can be consistently estimated through the maximum likelihood estimates $\\widehat{W}\\widehat{W}^T+\\sigma_0^2I$. \n\nOn a slightly more technical side, note that our quotient space construction was necessary as an intermediate step in the previous argument, in fact, we crucially used the continuous function $W\\to WW^T$ factors through the quotient space using Theorem 4.2 in our paper, we can give more details if you want but skip for now to keep the discussion simpler. Lastly, the known $\\sigma^2$ assumption was only for illustration purposes and can of course be removed, and in that case, the parameter $(\\widehat{W},\\widehat{\\sigma}^2)$ recovers $(W_0,\\sigma_0^2)$ up to a closed set $C$ defined in our paper just above the line 229, and here $\\widehat{W}\\widehat{W}^T+\\widehat{\\sigma^2}I$ consistently estimates the covariance of the PPCA process, a theoretical guarantee with respect to the maximum likelihood estimate that was not known before. \n\nFinally, $\\textit{regarding the weakness}$ you pointed out, thank you for raising this concern. We can remove the assumption that: $q$ is known. Our proof techniques work as long as $1\\leqslant q\\leqslant p$ is a fixed integer, i.e. they do not grow with the sample size $n$. \nUsually, it is a different line of interest to want to estimate $q$ (related to model selection problems), also $q$ can be estimated consistently through MLE; We discussed all these relevant works in lines 134-148 in our work, which motivated us not to focus too much on $q$. The authors warmly welcome further questions, suggestions to improve the current manuscript, or if there is anything that the reviewer would want to see in the final version. Due to the page constraint, we had to make a choice on the content to present and our main focus was to judiciously build a rigorous framework to make it available for the community which was only partly addressed before by the previous work, Redner (1981).\n\n$\\textit{We hope we have addressed all your comments satisfactorily and kindly request you to revise your score.}$\n"}, {"review_id": "Z2GsdqzWZC", "reviewer": "Reviewer_QWak", "summary": "This work studies the consistency of the maximum likelihood estimates of probabilistic PCA. These estimates are unique up to a rotation, so they use the quotient space and claim in their Lemma 6.1, ..., 6.5 stated in Redner[1981] are verified and apply the consistency results available in this book.", "questions": "Could you clarify the contributions of the paper ? Is there something I missed ?", "limitations": "Not applicable.", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "This paper is overall well written, with a thorough introduction to equivalence classes and quotient topological spaces.", "weaknesses": "Compared to Redner [1981] and Wald [1949], the novelty is not enough. I see the contribution as mainly an application of the result of Wald to the case of probabilistic PCA. The general idea is rather straightforward: using the quotient space instead of $R^d$ is the first idea that comes to mind. I agree that the precise formulation is not trivial but alone, and although the writing is excellent, it is not enough for a publication in my opinion. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Respectful Disagreement}$\n\nThank you for your careful review. \nWe have clarified the details of our theoretical contributions and we respectfully disagree with the assessment that the contribution is mainly an application of the result of Wald and that there is inadequate reproducibility Not only are there significant technical novelties where our work differs from Redner's (1981), but our research definitely impacts future research}\n\n$\\textit{Novel Contributions}$\n\nWe agree that the quotient of $\\mathbb{R}^d$ is the first that comes to mind but it takes a significant amount of technical novelty to rigorously build a framework that is amenable to an extension of Wald's work. The model PPCA has been around since 1999. In light of the widely used ML estimates it is highly unlikely that nobody tried to justify them through an argument based on Wald's criteria. The main difficulty was to interpret Wald's condition in a quotient space framework which partly has been introduced in Redner's work but that is neither precise nor complete.\n\nWhile Redner's work is the first one to $\\textit{attempt}$ to address this issue with the help of quotient topology, the treatment developed by Redner is rather misleading as highlighted in our work (l67-78, l234-247). We humbly request the reviewer to glance through Redner's paper where he states Thereom 4. Unfortunately, it is assumptions 5 (and 3) that Redner takes for granted. For instance, it is simply untrue that the quotient metric behaves in a similar fashion to that of the original Euclidean metric and therefore it would take a great deal of effort to unambiguously define the right notion of convergence to be used for assumption 5 (and 3), which Redner seems to have missed. Redner mentions that the MLE converges to the true parameter in this topological space and this follows from the theory of quotient spaces which is not true. In general, metric topology and quotient topology are two very different things and their interaction depends highly on the equivalence relation $\\sim$. Here is a counter-example: Consider the space $X=\\{(x,y):x,y\\geqslant 0\\} - \\{(0,0)\\} $ with Euclidean norm, and define the equivalence relation $\\sim$ on $X$ given by: $p_1\\sim p_2\\iff p_1=\\lambda p_2$ for some $\\lambda>0$, i.e., the points $p_1,p_2$ are equivalent if they are on the same ray. The projection map here is $\\pi:X\\to X/\\sim$ is given by $\\pi(p)=[p]$. Under this equivalence relation it is true that $d_1([x],[y])=0$ (from the definition of pseudo metric $d_1$ after l208 in paper) for any two distinct $[x],[y]\\in X/\\sim$, which can be easily argued by taking $x_1=x/n,y_1=y/n$ with $n\\in\\mathbb{N}$, so that, $x_1\\sim x,y_1\\sim y$ and letting $n\\to \\infty $.  Therefore the usual metric structure totally breaks down as any two different equivalence classes have a distance zero between them. But quotient topology is non-trivial and we can still talk about convergence there.\n\nTo extend Wald's condition, the space $X$ must be endowed with a non-trivial metric structure which Redner has missed. For instance, it is impossible in the above example. The quotient space has to pass through a pseudometric to achieve the desired metric space structure in order to build set up the ground for Wald's results to extend. In general, it has been very hard to interpret something meaningful out of the intangible metric structure from a statistical point of view. However, when the quotient is done with respect to a closed set one can provide a concrete tangible geometrical description of the metric space through Lemma 4.3, which in our work has been done in the context of PPCA.\n\nThe model PPCA is $\\textit{not}$ important here, rather the $\\textit{key}$ part is that the quotient has to be with respect to a $\\textit {nice enough equivalence}$. This key technical but unavoidable construction seems to have gone unnoticed by Redner. Not only our contribution is building this connection that requires a significant amount of bridging between different equivalences that are governing the quotient space, but coming up with an explicit description of the metric in the quotient space is a significant contribution that was only ``existential\" prior to us. This explicit description plays a crucial role in the proof of assumption 5 (our Lemma 6.5) which again was assumed trivial in Redner's work (see the proof of Theorem 5). Also, note that in that proof, it was taken for granted that $g^{*}(x,\\gamma,r)$ is measurable, which is not true generally. It is only upper semianalytic (see proposition 7.47, p179 in the reference we gave in proof of Lemma 6.2). \n\n$\\textbf{Impact and reproducibility for future research:}$\n\nAs an immediate consequence of our work is (strong) consistent covariance estimation of the PPCA process which wasn't known before. (please see the second last paragraph of reviewer ZSYo's response) Our rigorous framework opens doors for many other general statistical models where rotational ambiguity (or more generally ambiguity due to a closed space or symmetric parametrization) is present. For instance, Matrix factorization problem where one is interested in the estimation of two matrices $\\textbf{A},\\textbf{B}$ such that the data matrix $\\textbf{X}=\\textbf{A}\\textbf{B}+\\text{noise}$ is one such problem, thanks to the Reviewer 1.\n\nOur methodology is highly flexible in the sense that it does $\\textit{not}$ depend on the statistical model much as long as there are some regularities in the geometry of the parameter space $\\Theta$. (please see the last paragraph of reviewer ZSYo's response)\n\nOur contribution includes an explicit formulation for smooth enough equivalence relations that was very much needed to make this flexible topological toolkit widely accessible to future researchers. Finally, we welcome your further suggestions and comments to improve the final manuscript.\n\n$\\textit{We hope we have addressed all comments satisfactorily and kindly request you to revise your score.}$"}]

Author rebuttal:

Meta-review: While it was pointed out that the contribution in techniques for obtaining the results may be limited, the reviewers agreed that the paper addresses something that has been overlooked in the literature, and the results constitute a useful contribution to the PPCA problem. Please incorporate the reviewers' comments when preparing the camera-ready version.
Decision: Accept (poster)