Title: Learning Coarse-Grained Representations: An Exploration of Mutual Information via Hyperspherical Density
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: self-supevised learning, mutual information, kernel density estimation, von misses-fisher distribution, representation learning

Abstract: We revisit InfoMax for representation learning, using hyperspherical geometry with a non-parametric von Mises-Fisher kernel density estimator and differential entropy. This method is minimal with no asymmetry and trains stably. Results are competitive on smaller datasets such as CIFAR-10, STL-10 and LC25000, but lags behind modern baselines on ImageNet-1000. Experiments show that weakening the global entropy term consistently helps classification accuracy, suggesting that strict mutual information classification favors coarse grouping over fine discrimination.

TL;DR: We propose a novel SSL method using hyperspherical density estimation for MI maximization. It learns strong coarse-grained representations but struggles with fine-grained details, suggesting a gap in current MI theory.