Title: WHY TEACHER–STUDENT SELF-SUPERVISED LEARNING WORKS: A MUTUAL INFORMATION PERSPECTIVE
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Representation Learning, Non-Contrastive Learning, Mutual Information, Teacher-Student framework

Abstract: We study teacher-student (TS) self-supervised learning methods (e.g., BYOL, SimSiam), which learn strong representations without relying on negative samples but currently lack a clear information-theoretic explanation. Building on the InfoMax perspective that unifies many multi-view SSL families, we show that TS-SSL implicitly maximizes a variational lower bound on the mutual information \(I(Z_\theta; X)\) between inputs and the teacher representations \(Z_\theta\). Concretely, we prove that the predictor yields an implicit variational estimate of \(I(Z_\theta; (Z_\phi, X))\), the mutual information between teacher and student representations, and that the alternating optimization—student prediction (with stop-gradient) followed by teacher updates—implicitly maximizes a lower bound on \(I(Z_\theta; X)\). Then, we derive convergence results characterizing the evolution of the teacher representation’s entropy and alignment during training. Eventually, motivated by these theoretical insights, we introduce a simple mutual-information–based regularizer on the student latent space that enforces monotonic growth of \(I(Z_\theta; X)\) and yields consistent downstream improvements on both natural-image and medical-imaging benchmarks.

TL;DR: We show that teacher–student SSL implicitly maximizes mutual information, derive convergence results, and propose an MI-based regularizer that boosts BYOL and SimSiam performance on natural and medical imaging benchmarks.