SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Arghya Datta*
Sayak Chakrabarty†
Probabilistic principal component analysis (PPCA) is currently one of the most used statistical tools to reduce the ambient dimension of the data. From multidimensional scaling to the imputation of missing data, PPCA has a broad spectrum of applications ranging from science and engineering to quantitative finance.
Despite this wide applicability in various fields, hardly any theoretical guarantees exist to justify the soundness of the maximum likelihood (ML) solution for this model. In fact, it is well known that the maximum likelihood estimation (MLE) can only recover the true model parameters up to a rotation. The main obstruction is posed by the inherent identifiability nature of the PPCA model resulting from the rotational symmetry of the parameterization. To resolve this ambiguity, we propose a novel approach using quotient topological spaces and in particular, we show that the maximum likelihood solution is consistent in an appropriate quotient Euclidean space. Furthermore, our consistency results encompass a more general class of estimators beyond the MLE. Strong consistency of the ML estimate and consequently strong covariance estimation of the PPCA model have also been established under a compactness assumption.
In the era of big data, principal component analysis (PCA) is a standard dimension reduction tool frequently used in exploratory data analysis. PCA is however not a statistical model, implying that uncertainty quantification is not available. Uncertainty quantification is desirable in some situations, like when principal components have an interpretation. Probabilistic principal component analysis (PPCA) is a model introduced by Tipping and Bishop [1999] to remedy the situation. It consists of an additive model with a normal noise term. Let  $p > q$  be two given positive integers. The PPCA model can be written as  $x = \mathbf{W}z + \varepsilon$ , where  $x \in \mathbb{R}^p$  denotes a random vector that is observed after a linear transformation  $\mathbf{W}$  is applied to a latent (hidden) variable  $z \in \mathbb{R}^q$  up to a gaussian noise  $\varepsilon \sim \mathcal{N}(0, \sigma^2 I_p)$ . We assume that our data is centered, meaning  $x$  has zero mean. In practice, the latent variable  $z$  is not observed and is conventionally assumed to be normally distributed. Given a set of  $n$  data points or observations  $\{x_1, x_2, \ldots, x_n\} \subset \mathbb{R}^p$ , the two unknown parameters  $\mathbf{W}$  and  $\sigma^2$  are estimated by maximum likelihood estimation (MLE).
PPCA is a very well-known and pervasive technique used in dimension reduction of high-dimensional data and used in figuring out the relevant dimensions in large-scale data mining. It must be said that
\*arghya.datta@umontreal.ca
$\dagger$  sayakchakrabarty2025@u.northwestern.edu
the primary goal of PPCA is not always to perform better than traditional PCA but open up a series of possibilities for future extensions for further analysis. PPCA also enables comparison with other probabilistic techniques. Namely, it is possible to quantify the uncertainty in parameters  $\mathbf{W}$  and  $\sigma^2$  through a Bayesian reformulation [Tipping and Bishop, 1999]. One can also consider modelling the error terms using heavy-tailed distributions (such as Student-t) to achieve robustness [Gai et al., 2008]. These models have appealing properties; in particular, the quality of the information retained from the original data is similar to that of the original PCA with added flexibility.
Despite the wide applicability, nothing is known about the consistency of the maximum likelihood (ML) estimation of the PPCA. The main obstruction is posed by the identifiability of the PPCA model over the parameter space  $(\mathbf{W},\sigma^2)\in \mathbb{M}(p,q)\times \mathbb{R}_+$ , where  $\mathbb{M}(p,q)$  and  $\mathbb{R}_+$  denote the set of all  $p\times q$  matrices and the set of positive real numbers respectively. One can restrict the attention only to rank  $q$  matrices, but as we shall see, doing so will not significantly make our goal any harder (or easier) to achieve as the ML estimation of  $\mathbf{W}$  has rank  $q$ . It can be readily seen that our model induces a Gaussian distribution on the data points  $x\sim \mathcal{N}(0,\mathbf{WW}^T +\sigma^2 I_p)$ , which is to be used for the maximum likelihood estimation. From a frequentist point of view, we now assume that the unknown parameters  $(\mathbf{W},\sigma^2)$  has a true value which is equal to  $(\mathbf{W}_0,\sigma_0^2)$ . Additionally, we observe that the marginal distribution of the data points  $\mathcal{N}(0,\mathbf{WW}^T +\sigma^2 I_p)$  used for the ML estimation remains invariant under any translation of  $\mathbf{W}$  by an orthogonal matrix  $\mathbf{R}$  since  $(\mathbf{WR})(\mathbf{WR})^T = \mathbf{WW}^T$ . As can be seen in the literature it is only possible to recover the maximum likelihood estimate of  $\mathbf{W}$  up to rotations [Tipping and Bishop, 1999]. This lack of identifiability puts an immediate roadblock to directly applying Wald's consistency theorems [Wald, 1949], a well-known method to guarantee the consistency of maximum likelihood estimate.
It turns out that the quotients of the Euclidean spaces and in particular, the quotient of the space  $\mathbb{M}(p,q)\times \mathbb{R}_+ \subset \mathbb{R}^{p\times q}\times \mathbb{R}$  is the natural topological object to consider to talk about the consistency results of the PPCA model. In a simplified situation when  $\sigma^2$  is known, heuristically, the quotient space treats all the points  $\mathbf{W}_0$  and its rotational translates  $\mathbf{W}_0\mathbf{R}$  as a single entity by labelling them equivalent and therefore throwing all of them into a single equivalence class. Therefore, the parameter space for  $\mathbf{W}$ , i.e.  $\Sigma = \mathbb{M}(p,q)$  becomes a set of equivalence classes  $\tilde{\Sigma} = \{[\mathbf{W}]:\mathbf{W}\in \mathbb{M}(p,q)\}$  through an equivalence relation  $\mathbf{W}'\in [\mathbf{W}]$  iff  $\mathbf{W}'\mathbf{W}'' = \mathbf{W}\mathbf{W}^T$  or equivalently  $\mathbf{W}' = \mathbf{WR}$ , for some orthogonal matrix  $\mathbf{R}$ . In this setting, we thus get rid of the identifiability issue by passing through the quotient of  $\mathbb{R}^{p\times q}\times \mathbb{R}$  via the proposed equivalence relation. We then study the consistency of the maximum likelihood in this space. The formal details about the definition of quotient space by a given equivalence relation can be found in Section 4.
The above approach is inspired by Redner [1981] where the author outlines a way to extend Wald's consistency theorems [Wald, 1949] under the non-identifiability assumption. We would like to mention that, even though passing through the quotient topological space eliminates the rotational ambiguity of the ML estimate by definition, it introduces many further technical issues. To the best of our knowledge, all the subtleties have not been properly addressed in Redner's work even though the author applies the proposed methodology to analyze the consistency of ML estimates of mixture models. One of the key things that is not explicitly mentioned in their work is that the underlying metric space structure of the quotient topological space can be substantially different from the natural topological structure of the quotient space itself and the topology generated by the open balls with the induced metric in quotient space is in general, not same as that of the usual quotient topology. Thus related geometric properties, for instance, the notion of convergence and other regularity properties of the parameter space become more delicate to analyze. We address all these underlying issues carefully later in section 4.
To the best of our knowledge, no previous research work has attempted to justify the wide usage of the ML estimate of the PPCA model because of the inherent identifiability nature of the model parameters. To resolve this challenge we propose a novel framework to see the PPCA model through the lens of an appropriate quotient of an Euclidean space. This perspective not only allows us to formalize the problem in precise mathematical terms but also helps us resolve the dispute caused by the non-identifiability.
Considering from the point of novelty, the contributions of this paper are two-fold: we prove that the maximum likelihood estimates  $(\hat{\mathbf{W}},\hat{\sigma}^2)$  are consistent, i.e.,  $(\hat{\mathbf{W}},\hat{\sigma}^{2})\to (\mathbf{W}_{0},\sigma_{0}^{2})$  in probability
in the appropriate quotient Euclidean space. In fact, our consistency results are valid for a general family of estimators that in particular, contains the maximum likelihood estimator. Furthermore, strong consistency, i.e., an almost sure convergence of the ML estimates to the true parameter is also deduced when the parameter space is compact.
The rest of the paper is organized as follows: section 2 outlines a brief description of the necessary background followed by a comprehensive discussion of relevant research works. In section 3, we formally introduce the PPCA model, and the ML estimates of the parameters of interest and talk about the main question we are interested in. In section 4, we provide several basic definitions and mention a few key results from the theory of quotient topological spaces in a self-contained fashion. Section 5 contains all our contributions followed by their detailed proofs in section 6. In section 7, we state Wald's conditions and interpret them within the quotient parameter space. Finally, in section 8, we talk about the impact of our work and put forward a few open questions before concluding in section 9.
The abstract formalism concerning quotient topological spaces is not foreign to the theory of applied statistics. For an in-depth discussion about the application of quotient space into various statistical models, see McCullagh [1999]. Even though, the necessary tools and definition of topological quotients are introduced in section 4, an inquisitive reader is certainly encouraged to consult the classic Munkres. The main line of attack used in this paper has been previously introduced in Redner [1981], where the author proves the consistency of mixture models under the standard regularity conditions of Wald's consistency theorems [Wald, 1949]. It is worthwhile to mention that the theoretical justification of using the maximum likelihood estimates in practice has gained a lot of recent interest. For instance, Chen [2017] has a rigorous treatment of several consistency results with an application to non-parametric mixture models.
In the previous section, we have already mentioned that PPCA offers many advantages over traditional PCA, particularly in terms of model flexibility and the potential for generalization to accommodate various statistical reformulations. Building upon the recognition of its strength by Bengio et al. [2013], Goodfellow et al. [2016], and Ruff et al. [2021] as one of the most notable advancements in probabilistic models, subsequent research endeavours have leveraged PPCA to achieve remarkable results in various applications. Within the realm of cryo-electron microscopy (cryo-EM) in the field of biology, extensive discussions in the literature have explored its efficacy, as evidenced by the works of Heimowitz et al. [2018], Penczek et al. [2011], Tagare et al. [2015]. Similarly, in the domain of computer vision, the probabilistic model has proven its superiority, as demonstrated by Szeliski [2022] and others. While the precise reasons underlying its success in certain situations remain elusive, PPCA has found widespread applicability in diverse domains. It has found great applications in incremental learning for visual tracking [Lim et al., 2004], in the study of Gaussian processes [Bonilla et al., 2007, Lawrence and Hyvarinen, 2005], orthogonal signal correction (OSC) [Lee et al., 2023], weighted nuclear norm minimization [Gu et al., 2017], and in outlier detection algorithms like [Domingues et al., 2018]. Recent advancements have showcased its potential in areas such as few-shot learning [Wang et al., 2020], data mining techniques [Witten and Frank, 2002], and most notably, in finance for exploiting market integration for pure alpha investments [Tzagkarakis et al., 2013]. The broad spectrum of applications where PPCA has exhibited remarkable performance underscores its versatility and its value as a powerful tool in numerous domains.
As we have already noted, the theoretical justification behind the ML estimates used for the PPCA model in practice is not always straightforward and has recently attracted the attention of several authors. In a fairly recent work [Chérief-Abdellatif, 2019], the authors provide a consistent estimator of the quantity  $\mathbf{W}\mathbf{W}^T$  assuming  $\sigma^2$  is known. Their work is primarily concerned with the model selection questions in statistics using a variational approach. Even though in our work we assume the rank of the matrix  $\mathrm{rank}(\mathbf{W}) = q$  to be known and is fixed by the user (this assumption can be removed as long as  $q$  does not grow with the sample size  $n$ ), it is of independent interest to want to estimate  $q$ . In their work [Chérief-Abdellatif, 2019], the authors constructed an estimator of the
quantity  $q$  to demonstrate how closely their estimated model recovers the true model corresponding to the true value of  $\mathbf{W}$ , they were also able to estimate the marginal covariance of the data points and thereby came up with a consistent estimator of  $\mathbf{W}\mathbf{W}^T$ . One may notice that, in this case, estimating  $\mathbf{W}\mathbf{W}^T$  instead of  $\mathbf{W}$  eliminates the non-identifiability issue. It is important to mention that their work does not use the maximum likelihood approach and instead uses variational techniques to construct an estimator of  $\mathbf{W}\mathbf{W}^T$ . In this regard, it is relevant to mention that another work [Bouveyron et al., 2011] was able to provide a consistent estimator of the parameter  $q$  using maximum likelihood estimation. Finally, we restate that no related previous work attempts to justify the commonly used ML estimates  $(\hat{\mathbf{W}},\hat{\sigma}^2)$  of the PPCA model, perhaps due to ambiguity caused by the inherent non-identifiable parameter space. In the following section, we begin by presenting the PPCA model and concerned ML estimates originally developed in Tipping and Bishop [1999]
Notation In what follows, we shall always let capital letters denote random variables and their deterministic realizations will be denoted by lowercase letters. Let  $X_{1}, X_{2}, \ldots$  be a sequence of independent identically distributed random variables where the p.d.f. of the distribution of  $X_{1}$  is known except a parameter  $\theta = (\mathbf{W}_{\theta}, \sigma_{\theta}^{2})$  in some parameter space  $\Theta \subset \mathbb{R}^{p \times q} \times \mathbb{R}_{+}$  for some matrix  $\mathbf{W}_{\theta}$  and positive integers  $p, q$ . We assume the distribution of  $X_{1}$  is generated from a true parameter  $\theta_{0}$ . For a given parameter  $\theta = (\mathbf{W}_{\theta}, \sigma_{\theta}^{2})$ , we let  $f(x; \theta)$  denote the normal p.d.f.  $\mathcal{N}(x; 0, \mathbf{W}_{\theta} \mathbf{W}_{\theta}^{T} + \sigma_{\theta}^{2} I_{p})$ . Henceforth, the notation  $\| \mathbf{W} \|$  will mainly refer to the spectral norm, which is defined as the largest eigenvalue of the matrix  $\mathbf{WW}^T$ . It is worth noting that in some rare instances, we have employed the Frobenius norm, but such usage has been explicitly mentioned. Unless otherwise stated, all the probabilities and expectations are taken with respect to the true parameter  $\theta_{0}$ . Finally, for a  $m \times m$  matrix  $\mathbf{A}$ , we denote its characteristic polynomial  $\det(\lambda I_m - A)$  by  $\mathcal{P}_A(\lambda)$ .
The PPCA model: Let us consider that we have access to a structured dataset taking the form of a matrix  $\mathbf{X}$  whose lines are given by  $x_{1},\ldots ,x_{n}\in \mathbb{R}^{p}$  with  $p$  a large positive integer. We assume that the columns of  $\mathbf{X}$  are centered (meaning a mean of 0). We assume
$$
x _ {i} = \mathbf {W} z _ {i} + \varepsilon_ {i}, \tag {3.1}
$$
where  $z_{1},\ldots ,z_{n},\varepsilon_{1},\ldots ,\varepsilon_{n}$  are i.i.d. the realizations of independent random variables with  $z\sim \mathcal{N}(0,I_q)$  and  $\varepsilon \sim \mathcal{N}(0,\sigma^2 I_p)$ , and  $\mathbf{W}$  is an unknown fixed  $p\times q$  matrix,  $\sigma >0$  being an unknown fixed scale parameter and  $I_{q}$  and  $I_{p}$  being identity matrices of size  $q$  and  $p$ , respectively. The random variable  $z$  plays the role of a hidden (latent) variable and  $\varepsilon$  that of an error term. We recall that  $\mathrm{rank}(\mathbf{W}) = q$ . To estimate the parameters  $\mathbf{W}$  (often called the loading matrix) and the unknown variance  $\sigma^2$  of the homoscedastic error term  $\epsilon$ , Tipping and Bishop [1999] employs the maximum likelihood (ML) procedure and solutions are given as follows
$$
\widehat {\mathbf {W}} = \mathbf {U} \left(\Delta_ {q} - \sigma^ {2} I _ {q}\right) ^ {1 / 2} \mathbf {R} \mathrm {a n d} \widehat {\boldsymbol {\sigma}} ^ {2} = \frac {1}{p - q} \sum_ {j = q + 1} ^ {p} \delta_ {j},
$$
where  $\mathbf{R}$  is any  $q\times q$  rotation matrix, the columns of  $\mathbf{U}$  are given by the first  $q$  eigen vectors of the sample covariance matrix  $S_{x} = 1 / n\sum_{i = 1}^{n}x_{i}x_{i}^{T}$  and  $\Delta_q = \mathrm{diag}(\delta_1,\ldots ,\delta_q)$  are the  $q$  eigenvalues of  $S_{x}$  sorted in the descending order. We note that the solution  $\widehat{\mathbf{W}}$ , in this case, is not unique since  $\mathbf{R}$  could be any orthogonal matrix which in practice, is often set to be  $I_{q}$ . For now, we redistrict our discussion assuming the rank  $q$  is fixed. For this model, given the parameters  $(\mathbf{W},\sigma^2)$ , we have the following marginal density for the generated data  $\{x_i\}$
$$
p (x | \mathbf {W}, \sigma) = \int p (x | z, \mathbf {W}, \sigma) p (z) d z = \mathcal {N} (x; 0, \mathbf {W} \mathbf {W} ^ {T} + \sigma^ {2} I _ {p}),
$$
where we used the fact that the latent variable  $z$  is distributed as  $\sim \mathcal{N}(0, I_q)$ . Thus for our problem, the parameter space is
$$
\Theta := \left\{\theta : \theta = \left(\mathbf {W} _ {\theta}, \sigma_ {\theta} ^ {2}\right) \in \mathbb {R} ^ {p \times q} \times \mathbb {R} _ {+} \right\}.
$$
In the above definition there is a slight abuse of notation as the parameter  $\theta$  is self-referencing to itself. Nonetheless, we retain the notation for the sake of clarity and the ease of exposition. We assume the data points  $\{x_{i}\}$  are generated from a true parameter  $\theta_0 = (\mathbf{W}_0,\sigma_0^2)$ , i.e., they are i.i.d. samples from normal distribution whose p.d.f. is given by  $f(x;\theta_0)$ . In this work, we aim to prove the consistency of maximum likelihood estimates, i.e. we want to discuss whether  $(\widehat{\mathbf{W}},\widehat{\sigma}^{2})\to (\mathbf{W}_{0},\sigma_{0}^{2})$  holds  $P_{\theta_0}$  almost surely or in probability.
Nearly all consistency proofs of well-known statistical models take identifiability for granted. Unfortunately, the PPCA model is not identifiable, since there may be matrices  $\mathbf{W}_{\theta}$  and  $\mathbf{W}_{\phi}$  such that  $\mathbf{W}_{\theta} \mathbf{W}_{\theta}^{T} = \mathbf{W}_{\phi} \mathbf{W}_{\phi}^{T}$ , in which case  $f(x; \theta) = f(x; \phi)$  if  $\sigma_{\theta} = \sigma_{\phi}$ . It is not difficult to see in such cases,  $\mathbf{W}_{\phi}$  will be a rotational translate of the matrix  $\mathbf{W}_{\theta}$ , i.e.,  $\mathbf{W}_{\phi} = \mathbf{W}_{\theta} \mathbf{R}'$  for some orthogonal matrix  $\mathbf{R}'$ . This rotational symmetry in parameterization results in a lack of identifiability. In the current work we adopt a similar approach as outlined in Redner [1981] and subsequently work in a quotient topological space of  $\Theta$ . Next, we rigorously introduce this quotient space framework and develop analytical tools to prove our consistency results.
4 A primer on quotient topological spaces
In this section, we give the definition of a quotient of a topological space by a given equivalence relation and discuss several important results pertinent to our work.
Definition 4.1 (Equivalence relation). Let  $S$  be a set. A binary relation  $\sim$  on  $S$  is said to be an equivalence iff it is (i) reflexive:  $x \sim x, \forall x \in S$ , (ii) symmetric:  $x \sim y \iff y \sim x, \forall x, y \in S$  and (iii) transitive:  $x \sim y, y \sim z \iff x \sim z, \forall x, y, z \in S$ .
It is evident that an equivalence relation on a set induces a partition of the set and vice-versa. The disjoint sets in this partition are called the equivalence classes. If  $s \in S$  is given, then we write  $[s] := \{y \in S \mid y \sim s\}$  for the equivalence class that contains  $s$  and finally the set of equivalent classes  $(S / \sim) := \{[s] \mid s \in S\}$  are called the quotient of the set  $S$  by the equivalence relation  $\sim$ .
4.1 Quotient topological spaces and its metrizability
Given a space  $X$  and an equivalence relation  $\sim$  on  $X$ , the set-theoretic quotient  $X / \sim$  (the set of equivalence classes) inherits a topology from  $X$ , called the quotient topology [Munkres]. It is well known that the surjective map  $\pi : X \to X / \sim$  defined as  $x \to [x]$  is continuous. We now record a useful result from Munkres, which will be important in proving our main results in the forthcoming sections.
Theorem 4.2. If  $X, X / \sim$  are topological spaces stated as above, then for another topological space  $Y$  and for a continuous map  $f: X \to Y$  with the additional property that  $x \sim x'$  implies  $f(x) = f(x')$ , there exists a unique continuous map  $\tilde{f}: (X / \sim) \to Y$  such that  $f = \tilde{f} \circ \pi$
Figure 1: Illustration of the above theorem using a commutative diagram.
Throughout this paper, we have  $Y = \mathbb{R}$  and  $X$  is a subset of  $\mathbb{R}^p$  for some positive integer  $p$ , which is a complete metric space and Euclidean distance serves as a natural metric on  $\mathbb{R}^p$ . The key idea is to extend this metric from  $X$  to  $X / \sim$  and this job is accomplished through an intermediate state, by introducing a suitable pseudometric on  $X$ . A pseudometric  $\delta$  is a function on  $X \times X$  to non-negative real numbers and the only difference between a pseudometric and a metric is that  $\delta(x, y) = 0$  need not imply  $x = y$  for  $x, y \in X$ . To construct the most commonly used metric on the quotient space  $X / \sim$  we first introduce:
let  $X \subset \mathbb{R}^p$  be a complete subspace of the Euclidean space and  $d$  be the standard Euclidean distance on  $\mathbb{R}^p$ . For  $x, y \in X$ , we let
$$
d _ {1} (x, y) := \inf  \left\{d \left(x _ {1}, y _ {1}\right) + \dots + d \left(x _ {n}, y _ {n}\right) \mid x \sim x _ {1}, y _ {i} \sim x _ {i + 1} (1 \leqslant i \leqslant n - 1), y _ {n} \sim y \right\}.
$$
Then it can be checked that  $d_{1}$  defines a pseudometric on  $X$ . We now declare  $x \approx y$  whenever  $d_{1}(x,y) = 0$ , and it is not difficult to verify the relation  $\approx$  is an equivalence. Then we can endow the set-theoretic quotient  $X / \approx$  with the metric  $\bar{d}$ , which we define as  $\bar{d}([x],[y]) = d_{1}(x,y)$  where  $[x],[y]$  are the classes of  $x,y$  in  $X / \approx$ . Furthermore, we let  $X_{\approx}$  to be the completion of the above space under the metric  $\bar{d}$ , meaning all the Cauchy sequences converge.