SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present an application of max-slicing to generative modeling under the InfoGAN framework [56]. The InfoGAN learns a disentangled latent space by maximizing the mutual information between a latent code variable and the generated data. We revisit this architecture but replace the classical mutual information regularizer in the InfoGAN objective with mSMI. Our max-sliced InfoGAN is tested on the MNIST and Fashion-MNIST datasets. Figure 3 presents the generated samples for several projection dimensions. We consider 3 latent codes  $(C_1, C_2, C_3)$ , which automatically learn to encode different features of the data. We vary the values of  $C_1$ , which is a 10-state discrete variable, along the column (and consider random values of  $(C_2, C_3)$  along the rows). Evidently,  $C_1$  successfully disentangles the 10 class labels and the quality of generated samples is on par with past implementations [56, 26]. We stress that since mSMI relies on low-dimensional projections, the resulting InfoGAN mutual information estimator uses a reduced number of parameters (at the negligible cost of optimizing over linear projections). Additional details are given in Supplement C.
This paper proposed mSMI, an information theoretic generalization of CCA. mSMI captures the full dependence structure between two high dimensional random variables, while only requiring an optimized linear projection of the data. We showed that mSMI inherits important properties of Shannon's mutual information and that when the random variables are Gaussian, the mSMI optimal solutions coincide with classic  $k$ -dimensional CCA. Moving beyond Gaussian distributions, we present a neural estimator of mSMI and establish non-asymptotic error bounds.
Through several experiments we demonstrate the utility of mSMI for tasks spanning independence testing, multi-view representation learning, algorithmic fairness and generative modeling, showing it outperforms popular methodologies. Possible future directions include an investigation of an operational meaning of mSMI, either in information theoretic or physical terms, extension of the proposed formal guarantees to the nonlinear setting, and the extension of the neural estimation convergence guarantees to deeper networks. Additionally, mSMI can provide a mathematical foundation to mutual information-based representation learning, a popular area of self-supervised learning [10, 57].
In addition to the above, we plan to develop a rigorous theory for the choice of  $k$ , which is currently devised empirically and is treated as a hyperparameter. When the support of the distributions lies in some  $d' < d$  dimensional subspace, the choice of  $k = d'$  is sufficient to recover the classical mutual information, and therefore it characterizes the full dependence structure. Extrapolating from this point, we conjecture that the optimal value of  $k$  is related to the intrinsic dimension of the data distribution, even when it is not strictly supported on a low-dimensional subset.
<sup>5</sup>Routine per iteration not to exceed the runtime of Slice per iteration. We used an NVIDIA V100 GPU.
[54] Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Physical review E, 69(6):066138, 2004.
[55] Cédric Malherbe and Nicolas Vayatis. Global optimization of lipschitz functions. In International Conference on Machine Learning, pages 2314-2323. PMLR, 2017.
[56] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[57] Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress—self-supervised learning and information theory: A review. arXiv preprint arXiv:2304.09355, 2023.
[58] Yury Polyanskiy and Yihong Wu. Information Theory: From Coding to Learning. Cambridge University Press, 2022.
[59] Jan R Magnus and Heinz Neudecker. Matrix differential calculus with applications in statistics and econometrics. John Wiley & Sons, 2019.
[60] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289, 2015.
[61] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[62] Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. Advances in neural information processing systems, 27, 2014.
[63] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.