NeurIPS 2025

Rethinking PCA Through Duality
Jan Quan∗
ESAT-STADIUS & Leuven.AI
KU Leuven, Belgium
jan.quan@kuleuven.be
Johan Suykens
ESAT-STADIUS & Leuven.AI
KU Leuven, Belgium
johan.suykens@kuleuven.be
Panagiotis Patrinos
ESAT-STADIUS & Leuven.AI
KU Leuven, Belgium
panos.patrinos@kuleuven.be
Abstract
Motivated by the recently shown connection between self-attention and (kernel)
principal component analysis (PCA), we revisit the fundamentals of PCA. Using the
difference-of-convex (DC) framework, we present several novel formulations and
provide new theoretical insights. In particular, we show the kernelizability and out-
of-sample applicability for a PCA-like family of problems. Moreover, we uncover
that simultaneous iteration, which is connected to the classical QR algorithm, is an
instance of the difference-of-convex algorithm (DCA), offering an optimization
perspective on this longstanding method. Further, we describe new algorithms
for PCA and empirically compare them with state-of-the-art methods. Lastly,
we introduce a kernelizable dual formulation for a robust variant of PCA that
minimizes the l1-deviation of the reconstruction errors.
1
Introduction
Principal component analysis (PCA) is one of the most fundamental dimensionality reduction
techniques and has seen widespread use in various fields, including machine learning [20, 10, 38, 23],
signal processing [4, 24], and computer vision [52, 78, 82], due to its ability to efficiently extract
meaningful features from high-dimensional data. Its nonlinear extension, kernel PCA [51], extends
this method through a feature mapping into a high-dimensional space (possibly infinite-dimensional),
enabling the discovery of nonlinear patterns. Crucially, this is achieved without ever computing the
transformed data explicitly, using the kernel trick [11].
In recent years, (kernel) PCA has regained interest due to its connection to the self-attention mecha-
nism [79], which lies at the heart of the widely successful transformer archite...