Title: Identifiability and recoverability in self-supervised models
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: identifiability, self-supervised learning, disentanglement

Abstract: Self-supervised models exhibit a surprising stability in their internal representations. Whereas most prior work treats this stability as a single property, we formalize it as two distinct concepts: **identifiability** (consistency across runs) and 
**recoverability** (alignment with some unobserved ground truth). We prove an $\epsilon$-**near-identifiability** result for the representations of models with nonlinear decoders, generalizing existing identifiability theory beyond last-layer representations in e.g. generative pre-trained transformers (GPTs) to near-identifiability of the internal representations of a broad class of models including (masked) autoencoders (MAEs) and supervised learners. Although these weaker assumptions confer weaker identifiability, we show that independent components analysis (ICA) can resolve much of the remaining linear ambiguity for this class of models, and validate and measure our near-identifiability claims empirically. With additional assumptions on the data-generating process, identifiability extends to recoverability, yielding a simple and practical recipe for disentanglement: ICA post-processing of latent representations. On synthetic  benchmarks, this approach achieves state-of-the-art disentanglement with a vanilla autoencoder. With a foundation model-scale MAE for cell microscopy, it disentangles biological variation from technical batch effects, substantially improving downstream generalization.

TL;DR: We define a notion of near-identifiability, and show that a broad class of self-supervised learning models have near-identifiable internal representations.