Quantifying the similarity of information contained in probabilistic latent spaces

Authors: Kieran A. Murphy, Sam Dillavou, Danielle Bassett
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Information theory, representation learning, disentanglement

Abstract:
In contrast to point-based representation spaces, probabilistic representation spaces have a well-defined sense in which they compress information about a dataset.
When viewing representation spaces as communication channels, it becomes natural to ask about the similarity of information content of different representation spaces.
Starting with classic measures of similarity of hard clustering assignments, we propose a natural modification that generalizes to probabilistic representation spaces.
We also propose a practical route toward estimating the similarity measure based on fingerprinting a representation space with a sample of the dataset that is applicable when the transmitted information is only a handful of bits.
Equipped with the similarity measures, we build upon model centrality as a signature of unsupervised disentanglement by assessing ``channel centrality'' and finding information fragments that are repeatedly learned in VAE and InfoGAN ensembles.
Additionally, we evaluate the diversity of information content of the full latent space over the course of training for ensembles of models, and find a striking difference in homogeneity of information depending on the dataset.
Finally, we leverage the differentiability of the proposed method and perform ensemble learning with VAEs by boosting the information content of a set of weak learners incapable of representing the global structure of a dataset.

Decision: reject

Reviews and discussion:

Metareview: