SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
D. P. Bertsekas and S. E. Shreve. Stochastic optimal control: the discrete time case. 1978.
E. V. Bonilla, K. Chai, and C. Williams. Multi-task gaussian process prediction. Advances in neural information processing systems, 20, 2007.
C. Bouveyron, G. Celeux, and S. Girard. Intrinsic dimension estimation by maximum likelihood in isotropic probabilistic PCA. Pattern Recognition Letters, 32(14):1706-1713, 2011.
T. Cacoullos. Exercises in probability. Springer Science & Business Media, 2012.
J. Chen. Consistency of the mle under mixture models. STATISTICAL SCIENCE, pages 47-63, 2017.
B.-E. Chérief-Abdellatif. Consistency of elbo maximization for model selection. In Symposium on Advances in Approximate Bayesian Inference, pages 11-31. PMLR, 2019.
R. Domingues, M. Filippone, P. Michiardi, and J. Zouaoui. A comparative evaluation of outlier detection algorithms: Experiments and analyses. Pattern recognition, 74:406-421, 2018.
J. Gai, Y. Li, and R. L. Stevenson. Robust bayesian PCA with student's t-distribution: the variational inference approach. In 2008 15th IEEE International Conference on Image Processing, pages 1340-1343. IEEE, 2008.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT press, 2016.
S. Gu, Q. Xie, D. Meng, W. Zuo, X. Feng, and L. Zhang. Weighted nuclear norm minimization and its applications to low level vision. International journal of computer vision, 121:183-208, 2017.
A. Heimowitz, J. Andén, and A. Singer. Apple picker: Automatic particle picking, a low-effort cryo-em framework. Journal of structural biology, 204(2):215-227, 2018.
I. M. James. Introduction to uniform spaces, volume 144. Cambridge University Press, 1990.
N. Lawrence and A. Hyvarinen. Probabilistic non-linear principal component analysis with gaussian process latent variable models. Journal of machine learning research, 6(11), 2005.
G. Lee, E. Sim, Y. Yoon, and K. Lee. Probabilistic orthogonal-signal-corrected principal component analysis. Knowledge-Based Systems, 268:110473, 2023.
J. Lim, D. Ross, R.-S. Lin, and M.-H. Yang. Incremental learning for visual tracking. Advances in neural information processing systems, 17, 2004.
P. McCullagh. Quotient spaces and statistical models. Canadian Journal of Statistics, 27(3):447-456, 1999.
J. Munkres. Topology, second edition.
P. A. Penczek, M. Kimmel, and C. M. Spahn. Identifying conformational states of macromolecules by eigen-analysis of resampled cryo-em images. Structure, 19(11):1582-1590, 2011.
R. Redner. Note on the consistency of the maximum likelihood estimate for nonidentifiable distributions. The Annals of Statistics, pages 225-228, 1981.
L. Ruff, J. R. Kauffman, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-795, 2021.
R. Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
H. D. Tagare, A. Kucukelbir, F. J. Sigworth, H. Wang, and M. Rao. Directly reconstructing principal components of heterogeneous particles from cryo-em images. Journal of structural biology, 191 (2):245-262, 2015.
M. E. Tipping and C. M. Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3):611-622, 1999.
G. Tzagkarakis, J. Caicedo-Llano, and T. Dionysopoulos. Exploiting market integration for pure alpha investments via probabilistic principal factors analysis. 2013.
A. Wald. Note on the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4):595-601, 1949.
Y. Wang, C. Xu, C. Liu, L. Zhang, and Y. Fu. Instance credibility inference for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12836-12845, 2020.
N.Weaver.Lipschitzalgebras.World Scientific,2018.
I. H. Witten and E. Frank. Data mining: practical machine learning tools and techniques with java implementations. Acm Sigmoid Record, 31(1):76-77, 2002.
J. Wolfowitz. On Wald's proof of the consistency of the maximum likelihood estimate. The Annals of Mathematical Statistics, 20(4):601-602, 1949.
11 Appendix
11.1 Counterexample: Quotient topology and the metric topology on the quotient parameter space are different.
It was pointed out in the introduction that special care is needed when talking about continuity and convergence in probability within the quotient parameter space  $\Theta / C$  as there are two different topologies at play. In fact, a substantial challenge of our work was to meticulously address the technical issues that arise from the interplay between these two topologies. In Redner [1981], it was claimed that the MLE converges to the true parameter in the quotient topological space and this follows from the theory of quotient spaces which is not true in general, as it is untrue that the quotient metric behaves in a similar fashion to that of the original Euclidean metric and therefore it would take a great deal of effort to unambiguously define the right notion of convergence to be used for assumption 5 (and 3) in Redner [1981]. The interaction between the metric topology and the quotient topology depends highly on the equivalence relation  $\sim$  on the parameter space  $\Theta$ . Here is a counter-example highlighting a situation where the quotient topology is nontrivial but the topology generated by the quotient metric  $\bar{d}$  is degenerate:
Consider the space  $X = \{(x, y) : x, y \geqslant 0\} - \{(0, 0)\}$  with the usual Euclidean norm, i.e., the first quadrant with axes except the origin and define the equivalence relation  $\sim$  on  $X$  given by:  $p_1 \sim p_2 \iff p_1 = \lambda p_2$  for some  $\lambda > 0$ , i.e., the points  $p_1, p_2$  are equivalent if they are on the same ray. The projection map here is  $\pi : X \to X / \sim$  is given by  $\pi(p) = [p]$ . Geometrically, it is helpful to visualize the space  $X / \sim$  as a north-east part of the unit circle  $\mathbf{S}^1$ , which is a subspace of a full circle. Under this equivalence relation it is true that  $d_1([x], [y]) = 0$  (from the definition of pseudo metric  $d_1$  in section 4) for any two distinct  $[x], [y] \in X / \sim$ , which can be easily argued by taking  $x_1 = x / n, y_1 = y / n$  with  $n \in \mathbb{N}$ , so that,  $x_1 \sim x, y_1 \sim y$  and letting  $n \to \infty$ . Therefore the usual metric structure totally breaks down in the quotient space as any two different equivalence classes has a distance zero between them while in this case the quotient topology is non-trivial and we can still talk about convergence with regards to the quotient topology. But to extend Wald's condition in the quotient, the space  $X / \sim$  must be endowed with a non-trivial metric space structure, which is impossible in a situation like this.
We also note that the quotient space had to pass through a pseudometric to achieve the desired metric space structure in order to set up the ground for Wald's results to extend. In general, it is very hard to interpret something meaningful out of this intangible metric from a statistical point of view. However, when the quotient is done with respect to a closed set, one can provide a concrete tangible geometrical description of the quotient metric space through Lemma 4.3, which in our work has been done in the context of PPCA. The model PPCA is not important here, rather the key part is that: the quotient has to be with respect to a nice enough equivalence (for instance with respect to a closed set, which is the case in Redner's work, our work and numerous other statistical models where identifiability arises). This key technical but unavoidable construction seems to have gone unnoticed in earlier works. Note that, it was imperative to come up with an explicit and tangible description of the metric in the quotient space as that plays a crucial role towards the proof of assumption 5 (Lemma 7.5 in our work) in Redner [1981], whose proof was assumed straightforward.
11.2 Counterexample: Theorem 4.2 is not sufficient to ensure the continuity of the lift with respect to the quotient metric  $\bar{d}$ .
We pointed out in section 6 that Lemma 6.1 is not sufficient to guarantee the continuity of the lift of the map  $\psi$  used in the deduction of Theorem 5.9. Indeed it is possible that a continuous (and not Lipschitz) function  $g:\Theta \to Y$  lifts to a function  $\tilde{g}:\Theta /C\to Y$  which is no longer continuous with respect to the metric topology generated by  $\bar{d}$  on the space  $\Theta /C$ . Here is a counterexample which highlights the situation:
Let  $X$  be a subset of the unit square  $[0,1]^2$  consisting of the origin and the points  $a_n = (\frac{1}{n}, 0), b_n = (\frac{1}{n}, 1 - \frac{1}{n})$ , and  $c_n = (\frac{1}{n}, 1)$  for  $n \geqslant 2$ . We endow  $X$  with the Euclidean metric inherited from  $[0,1]^2$ . We declare the following equivalence relation  $\sim$  on  $X$ : we let  $a_n \sim b_n$  for all  $n$ , and let  $g: X \to \mathbb{R}$  be the function such that  $g(0) = g(a_n) = g(b_n) = 0$ , and  $g(c_n) = 1$ . In the quotient space  $X / \sim$ , the sequence  $\{[c_n]\}$  converges to the origin, so the lift  $\tilde{g}$  isn't continuous, as  $\tilde{g}([c_n]) = 1$  for every  $n$  and  $g(0) = 0$ . One may note that the function  $g$  is not
Lipschitz in this example since the points  $\{b_n\}$  and  $\{c_n\}$  will become arbitrarily close as  $n \to \infty$  while  $|g(b_n) - g(c_n)| = 1$ . Even though the domain of the function  $g$ , in this case, is discrete, one can extend  $g$  continuously to  $[0,1]^2$ , using the Tietz extension theorem (see Munkres) and provide a counterexample where the domain is an Euclidean space.
We note that  $\psi$  (used in the deduction of Theorem 5.9) is not Lipschitz in the parameter space  $\Theta$  due to the presence of a quadratic term in  $\mathbf{W}$ . One way to ensure that  $\psi$  is Lipschitz is to assume the existence of a compact subset  $\Theta_0$  which contains the point  $\theta_0$  in its interior.
11.3 Proofs of auxiliary lemmas
Proof of Lemma 7.1. This is a direct consequence of the properties discussed in Section 4 and in particular, the fact that the topology generated from the metric  $\bar{d}$  is weaker than the standard quotient topology on  $\Theta /C$ . Let  $K$  be a closed and bounded set in  $(\Theta /C,\bar{d})$  and  $\{U_{\alpha}\}_{\alpha \in \Lambda}$  be an open cover for  $K$ . Since the set  $\pi^{-1}(K)$  is closed (by the continuity of quotient map) and bounded in  $\Theta$ , it is compact. Therefore the open cover  $\{\pi^{-1}(U_\alpha)\}_{\alpha \in \Lambda}$  can be reduced to a finite sub-cover of  $\pi^{-1}(K)$  in  $\Theta$ . Using the surjectivity of the quotient map, we then see that there is a finite subset  $F\subset \Lambda$  such that
$$
K = \pi (\pi^ {- 1} (K)) \subseteq \pi \left(\bigcup_ {\alpha \in F} \pi^ {- 1} (U _ {\alpha})\right) = \bigcup_ {\alpha \in F} U _ {\alpha}.
$$
Proof of Lemma 7.2. To ensure the measurability conditions so that the above integrals are well defined we include all the  $P_{\theta_0}$  null sets to the associated Borel sigma algebra as discussed in Bertsekas and Shreve [1978] (p.167, Corollary 7.42.1).
Given a parameter  $\phi = (\mathbf{W}_{\phi},\sigma_{\phi}^{2})\in \Theta$ , and  $r > 0$ , we note the function  $\phi \mapsto -p\log (2\pi) / 2 - \log \det (\mathbf{W}_{\phi}\mathbf{W}_{\phi}^{T} + \sigma_{\phi}^{2}I) / 2$  is continuous functions on  $\Theta$ . If  $\phi \sim \phi'$ , then the function evaluates to the same value for  $\phi$  and  $\phi'$ . Therefore the function factors through the quotient topological space  $\Theta / C$  by Theorem 4.2. In this case, the map  $\phi \mapsto -p\log (2\pi) / 2 - \log \det (\mathbf{W}_{\phi}\mathbf{W}_{\phi}^{T} + \sigma_{\phi}^{2}I) / 2$  can be realized as a continuous function on the quotient space  $\Theta / C$ . For  $[\theta ]\in \Theta /C$  fixed, we consider the ball  $N_{r}([\theta ])$ , which is compact. Thus by the virtue of continuous functions, the map will have finite maximum in  $N_{r}([\theta ])$ . We call these maximum  $M_{1}(r)$ . The log-likelihood  $\log f(x;\phi)$  is given by
$$
\log f (x; \phi) = - \frac {p \log (2 \pi)}{2} - \frac {1}{2} \log \det  (\mathbf {W} _ {\phi} \mathbf {W} _ {\phi} ^ {T} + \sigma_ {\phi} ^ {2} I _ {p}) - \frac {1}{2} x ^ {T} (\mathbf {W} _ {\phi} \mathbf {W} _ {\phi} ^ {T} + \sigma_ {\phi} ^ {2} I _ {p}) ^ {- 1} x,
$$
and thus we have that
$$
\begin{array}{l} \mathbb {E} _ {\theta_ {0}} \left[ \log f ^ {*} (x, \theta , r) \right] = \int_ {\mathbb {R} ^ {p}} \log \max  \left\{1, \sup  _ {[ \phi ] \in N _ {r} ([ \theta ])} f (x; \phi) \right\} d P _ {\theta_ {0}} (x) \\ \leqslant \int_ {\mathbb {R} ^ {p}} \sup  _ {[ \phi ] \in N _ {r} ([ \theta ])} \log f (x; \phi) d P _ {\theta_ {0}} (x) \\ = \int_ {\mathbb {R} ^ {p}} M _ {1} (r) d P _ {\theta_ {0}} (x) - \frac {1}{2} \mathbb {E} _ {\theta} \left[ \inf  _ {[ \phi ] \in N _ {r} ([ \theta ])} x ^ {T} \left(\mathbf {W} _ {\phi} \mathbf {W} _ {\phi} ^ {T} + \sigma_ {\phi} ^ {2} I\right) ^ {- 1} x \right] \\ \leqslant M _ {1} (r) \quad \left(\text {s i n c e} \left(\mathbf {W} _ {\phi} \mathbf {W} _ {\phi} ^ {T} + \sigma_ {\phi} ^ {2} I\right) ^ {- 1} \text {i s p o s i t i v e d e f i n i t e}\right). \\ \end{array}
$$
Bounding  $h^*(x, s)$  is a similar task. First we note there exists  $M^* > 0$  such that  $f(x; \phi) < 1$  (using Wald's assumption 3 Wald [1949] or the next lemma) whenever  $\bar{d}([\phi], 0) = \|[\phi]\| > M^*$ . Pick  $s$  sufficiently large such that  $[\phi] \notin N_s([\theta_0])$  implies  $\|[\phi]\| \geqslant M^*$  (can be ensured using triangle inequality  $\|[\phi]\| \geqslant \|[\theta_0] - [\phi]\| - \|[\theta_0]\|$ ) and thus in this case  $h^*(x, s) = 1$  and  $\mathbb{E}[\log h^*(x, s)]$  evaluates to 0.
Proof of Lemma 7.3. Since  $\bar{d}([\theta_i], [\theta_0]) \to \infty$  in  $\Theta / C$  implies  $d(\theta_i, \theta_0) \to \infty$  in  $\Theta$  (by definition of  $\bar{d}$  in Section 4), we have  $\| \theta_i \| \to \infty$  by triangle inequality. We aim to show  $\log f(x; \theta_i) \to -\infty$ . The
log-likelihood, therefore, is bounded by
$$
\begin{array}{l} \log f (x; \theta_ {i}) = - \frac {p \log (2 \pi)}{2} - \frac {1}{2} \log \det  (\mathbf {W} _ {\theta_ {i}} \mathbf {W} _ {\theta_ {i}} ^ {T} + \sigma_ {\theta_ {i}} ^ {2} I _ {p}) - \frac {1}{2} x ^ {T} (\mathbf {W} _ {\theta_ {i}} \mathbf {W} _ {\theta_ {i}} ^ {T} + \sigma_ {\theta_ {i}} ^ {2} I _ {p}) ^ {- 1} x \\ \leqslant - \frac {1}{2} \log \det  \left(\mathbf {W} _ {\theta_ {i}} \mathbf {W} _ {\theta_ {i}} ^ {T} + \sigma_ {\theta_ {i}} ^ {2} I _ {p}\right) \\ = - \frac {1}{2} \log \mathcal {P} _ {- \mathbf {W} _ {\theta_ {i}} \mathbf {W} _ {\theta_ {i}} ^ {T}} \left(\sigma_ {\theta_ {i}} ^ {2}\right) \quad \left(\mathcal {P} _ {- \mathbf {W} _ {\theta_ {i}} \mathbf {W} _ {\theta_ {i}} ^ {T}} \text {i s t h e c h a r a c t r i s t i c p o l y n o m i a l}\right) \\ = - \frac {1}{2} \sum_ {j = 1} ^ {p} \log \left(\sigma_ {\theta_ {i}} ^ {2} + \lambda_ {i j}\right) \quad \left(\left\{\lambda_ {i j} \right\} _ {j = 1} ^ {p} \text {e i g e n v a l u e s o f} \mathbf {W} _ {\theta_ {i}} \mathbf {W} _ {\theta_ {i}} ^ {T}\right) \\ \leqslant - \frac {1}{2} \log \left(\sigma_ {\theta_ {i}} ^ {2} + \| \mathbf {W} _ {\theta_ {i}} \|\right). \\ \end{array}
$$
We observe that if  $\| \theta_i\| \to \infty$ , either  $\| W_{\theta_i}\|$  or  $\| \sigma_{\theta_i}^2\|$  (or both) must become arbitrarily large, which in turn implies the required result.
Proof of Lemma 7.4.
$$
\begin{array}{l} \mathbb {E} _ {\phi} [ | \log f (x; \theta) | ] = \int_ {\mathbb {R} ^ {p}} | \log f (x; \theta) | d P _ {\phi} (x) \\ = \int_ {\mathbb {R} ^ {p}} \left(\frac {p \log (2 \pi)}{2} + \frac {1}{2} \log \det  \left(\mathbf {W} _ {\theta} \mathbf {W} _ {\theta} ^ {T} + \sigma_ {\theta} ^ {2} I\right)\right) d P _ {\phi} (x) \\ + \frac {1}{2} \mathbb {E} _ {\phi} \left[ x ^ {T} \left(\mathbf {W} _ {\theta} \mathbf {W} _ {\theta} ^ {T} + \sigma_ {\theta} ^ {2} I\right) ^ {- 1} x \right] \\ \leqslant c _ {1} (\theta) + \frac {1}{2} \mathbb {E} _ {\phi} [ \| x \| _ {2} \| \left(\mathbf {W} _ {\theta} \mathbf {W} _ {\theta} ^ {T} + \sigma_ {\theta} ^ {2} I\right) ^ {- 1} x \| _ {2} ] \\ \leqslant c _ {1} (\theta) + \left\| \left(\mathbf {W} _ {\theta} \mathbf {W} _ {\theta} ^ {T} + \sigma_ {\theta} ^ {2} I\right) ^ {- 1} \right\| \mathbb {E} _ {\phi} [ \| x \| _ {2} ^ {2} ]. \\ \end{array}
$$
Proof of Lemma 7.5. Suppose,  $\theta \in C$ . We observe  $\bar{d}([ \theta_i ], [\theta]) \to 0$  in  $\Theta / C$  implies  $d(\theta_i, C) \to 0$  in  $\Theta$ . Since  $C$  is a closed set and  $\Theta$  is complete, therefore the sequence  $\{\theta_i\}$  converges to some  $\theta_0 \in C$ . As  $\theta \mapsto f(x; \theta)$  is continuous, we have  $f(x; \theta_i) \to f(x; \theta_0) = f(x; \theta)$ , in this case.
If  $\theta \notin C$ , then there is a neighbourhood  $U_{\theta} \subset \Theta$  of  $\theta$  such that  $U_{\theta} \cap C = \emptyset$  and consequently all but finitely many terms in the sequence  $\{\theta_i\}$  belong to  $U_{\theta}$ . This implies, in this case,  $\bar{d}([\theta_i], [\theta]) = d(\theta_i, \theta)$  for all but finitely many  $i$  and the conclusion readily follows.