To Compress or Not to Compress- Self-Supervised Learning and Information
  Theory: A Review
Authors: name: Ravid Shwartz-Ziv; name: Yann LeCun
Published: 2023-04-19 00:33:59

Abstract: Deep neural networks have demonstrated remarkable performance in supervised
learning tasks but require large amounts of labeled data. Self-supervised
learning offers an alternative paradigm, enabling the model to learn from data
without explicit labels. Information theory has been instrumental in
understanding and optimizing deep neural networks. Specifically, the
information bottleneck principle has been applied to optimize the trade-off
between compression and relevant information preservation in supervised
settings. However, the optimal information objective in self-supervised
learning remains unclear. In this paper, we review various approaches to
self-supervised learning from an information-theoretic standpoint and present a
unified framework that formalizes the self-supervised information-theoretic
learning problem. We integrate existing research into a coherent framework,
examine recent self-supervised methods, and identify research opportunities and
challenges. Moreover, we discuss empirical measurement of information-theoretic
quantities and their estimators. This paper offers a comprehensive review of
the intersection between information theory, self-supervised learning, and deep
neural networks.