SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark 4 (Beyond Gaussian data). While the mSMI solution coincides with that of CCA in the Gaussian case, this is no longer expected to hold for non-Gaussian distributions. CCA is designed to maximize correlation, while mSMI has Shannon's mutual information between the projected variables as the optimization objective. Unlike correlation, mutual information captures higher order dependencies between the variables, and hence the optimal mSMI matrices will not generally coincide with  $(\mathrm{A}_{\mathrm{CCA}},\mathrm{B}_{\mathrm{CCA}})$ . Furthermore, the intricate dependencies captured by mutual information suggest that the optimal mSMI projections may yield representations that are more effective for downstream tasks. We empirically verify this observation in Section 5 on several tasks, including classification, multi-view representation learning, and algorithmic fairness.
Similarly to the above, the Gaussian max-sliced entropy is related to PCA [49, 14]. In Supplement A.3, we prove the following.
Proposition 3 (Gaussian max-sliced entropy). For a  $d$ -dimensional Gaussian variable  $X \sim \mathcal{N}(m, \Sigma)$ , we have  $\overline{\mathsf{sh}}_k(X) = \sup_{A \in \mathrm{St}(k,d)} \mathsf{h}(A^\top X) = \mathsf{h}(A_{\mathsf{PCA}}^\top X) = 0.5 \sum_{i=1}^k \log(2\pi e \lambda_i(\Sigma))$ , where  $\mathsf{A}_{\mathsf{PCA}}$  is optimal PCA matrix and  $\lambda_1(\Sigma), \ldots, \lambda_k(\Sigma)$  are the top  $k$  eigenvalues of  $\Sigma$ .
Note that the eigenvalues  $\lambda_1(\Sigma),\ldots \lambda_k(\Sigma)$  are non-negative since  $\boldsymbol{\Sigma}$  is a covariance matrix. Extrapolating beyond the Gaussian case, this poses max-sliced entropy as an information-theoretic generalization of PCA for unsupervised dimensionality reduction. An analogous extension using the Renyi entropy of order 2 was previously considered in [50] for the purpose of binary classification. In that regard,  $\overline{\mathsf{sh}}_k(X)$  can be viewed as the  $\alpha$  -Renyi variant when  $\alpha \rightarrow 1$
3.3 Generalizations Beyond Linear Slicing
The notion of mSMI readily generalizes beyond linear slicing. Fix  $d_x, d_y \geq 1$ ,  $k \leq d_x \wedge d_y$ , and consider two (nonempty) function classes  $\mathcal{G} \subseteq \{g : \mathbb{R}^{d_x} \to \mathbb{R}^k\}$  and  $\mathcal{H} \subseteq \{h : \mathbb{R}^{d_y} \to \mathbb{R}^k\}$ .
Definition 3 (Generalized mSMI). The generalized mSMI between  $(X,Y)\sim \mu_{XY}\in \mathcal{P}(\mathbb{R}^{d_x}\times \mathbb{R}^{d_y})$  w.r.t. the classes  $\mathcal{G}$  and  $\mathcal{H}$  is  $\overline{\mathrm{SI}}_{\mathcal{G},\mathcal{H}}(X;Y)\coloneqq \sup_{(g,h)\in \mathcal{G}\times \mathcal{H}}\mathsf{l}\big(g(X);h(Y)\big)$
The generalized variant reduces back to  $\overline{\mathrm{S}}\mathrm{l}_k(X;Y)$  by taking  $\mathcal{G} = \mathcal{G}_{\mathrm{proj}}\coloneqq \{\mathfrak{p}^{\mathrm{A}}:\mathrm{A}\in \mathrm{St}(k,d_x)\}$  and  $\mathcal{H} = \mathcal{H}_{\mathrm{proj}}\coloneqq \{\mathfrak{p}^{\mathrm{B}}:\mathrm{B}\in \mathrm{St}(k,d_y)\}$ , but otherwise allows more flexibility in the way  $(X,Y)$  are mapped into  $\mathbb{R}^k$ . We also have that if  $\mathcal{G}\subseteq \mathcal{G}'$  and  $\mathcal{H}\subseteq \mathcal{H}'$ , then  $\overline{\mathrm{S}}\mathrm{l}_{\mathcal{G},\mathcal{H}}(X;Y)\leq \overline{\mathrm{S}}\mathrm{l}_{\mathcal{G}',\mathcal{H}'}(X;Y)\leq \mathsf{l}(X;Y)$ , which corresponds to Property 1 from Proposition 1. Further observations are as follows.
Proposition 4 (Properties). For any classes  $\mathcal{G},\mathcal{H}$ , we have that  $\overline{\mathrm{SI}}_{\mathcal{G},\mathcal{H}}$  always satisfies Properties 3-5 from Proposition 1. If further  $\mathcal{G}_{\mathrm{proj}}\subseteq \mathcal{G}$  and  $\mathcal{H}_{\mathrm{proj}}\subseteq \mathcal{H}$ , then  $\overline{\mathrm{SI}}_{\mathcal{G},\mathcal{H}}$  also satisfies Property 2.
We omit the proof as it follows by the same argument as Proposition 1, up to replacing the linear projections with the functions  $(g,h)\in \mathcal{G}\times \mathcal{H}$ . In practice, the classes  $\mathcal{G}$  and  $\mathcal{H}$  are chosen to be parametric, typically realized by artificial neural networks. As discussed in Remark 5 ahead, this is well-suited to the neural estimation framework for mSMI (both standard and generalized). Lastly, note that  $\overline{\mathrm{SI}}_{\mathcal{G},\mathcal{H}}(X;Y)$  corresponds to the objective of multi-view representation learning [51], which considers the maximization of the mutual information between NN-based representation of the considered variables. We further investigate this relation in Section 5.3.
4 Neural Estimation of Max-SMI
We study estimation of mSMI from data, seeking an efficiently computable and scalable approach subject to formal performance guarantees. Towards that end, we observe that the mSMI is compatible with neural estimation [29, 38] due to its convenient variational form. In what follows we derive the neural estimator, describe the algorithm to compute it, and provide non-asymptotic error bounds.
4.1 Estimator and Algorithm
Fix  $d_x, d_y \geq 1$ ,  $k \leq d_x \wedge d_y$ , and  $\mu_{XY} \in \mathcal{P}(\mathbb{R}^{d_x} \times \mathbb{R}^{d_y})$ ; we suppress  $k, d_x, d_y$  from our notation of the considered function classes. Neural estimation is based on the DV variational form:
$$
\mathsf {l} (X; Y) = \sup  _ {f \in \mathcal {F}} \mathcal {L} _ {\mathsf {D V}} (f; \mu_ {X Y}), \quad \mathcal {L} _ {\mathsf {D V}} (f; \mu_ {X Y}) := \mathbb {E} [ f (X, Y) ] - \log \left(e ^ {\mathbb {E} [ f (\tilde {X}, \tilde {Y}) ]}\right),
$$
where  $(X,Y)\sim \mu_{XY}$ $(\tilde{X},\tilde{Y})\sim \mu_X\otimes \mu_Y$  , and  $\mathcal{F}$  is the class of all measurable functions  $f:$ $\mathbb{R}^{d_x}\times \mathbb{R}^{d_y}\to \mathbb{R}$  (often referred to as DV potentials) for which the expectations above are finite. As mSMI is the maximal mutual information between projections of  $X,Y$  , we have
$$
\overline {{\mathsf {S l}}} _ {k} (X; Y) = \sup  _ {(A, B) \in \operatorname {S t} (k, d _ {x}) \times \operatorname {S t} (k, d _ {y})} \sup  _ {f \in \mathcal {F}} \mathcal {L} _ {\mathsf {D V}} \left(f; (\mathfrak {p} ^ {A}, \mathfrak {p} ^ {B}) _ {\sharp} \mu_ {X Y}\right) = \sup  _ {f \in \mathcal {F} ^ {\text {p r o j}}} \mathcal {L} _ {\mathsf {D V}} (f; \mu_ {X Y}),
$$
where  $\mathcal{F}^{\mathrm{proj}}\coloneqq \left\{f\circ (\mathfrak{p}^{\mathrm{A}},\mathfrak{p}^{\mathrm{B}}):f\in \mathcal{F},(\mathrm{A},\mathrm{B})\in \mathrm{St}(k,d_x)\times \mathrm{St}(k,d_y)\right\}$ . The RHS above is given by optimizing the DV objective  $\mathcal{L}_{\mathrm{DV}}$  over the composed class  $\mathcal{F}^{\mathrm{proj}}$ , which first projects  $(X,Y)\mapsto (\mathrm{A}^{\intercal}X,\mathrm{B}^{\intercal}Y)$  and then applies a DV potential  $f:\mathbb{R}^k\times \mathbb{R}^k\to \mathbb{R}$  to the projected variables.
Neural estimator. Neural estimators parametrize the DV potential by neural nets, approximate expectations by sample means, and optimize the resulting empirical objective over parameter space. Let  $\mathcal{F}_{\mathfrak{nn}}$  be a class of feedforward networks with input space  $\mathbb{R}^k\times \mathbb{R}^k$  and real-valued outputs. Given i.i.d. samples  $(X_{1},Y_{1}),\ldots ,(X_{n},Y_{n})$  from  $\mu_{XY}$ , we first generate negative samples (i.e., from  $\mu_X\otimes \mu_Y$ ) by taking  $(X_{1},Y_{\sigma (1)}),\ldots ,(X_{n},Y_{\sigma (n)})$ , where  $\sigma \in S_n$  is a permutation such that
2One may instead use the form that stems from convex duality:  $\mathsf{I}(U;V) = \sup_f\mathbb{E}[f(U,V)] - \mathbb{E}\big[e^{f(\tilde{U},\tilde{V})} - 1\big].$
For now, we leave the architecture (number of layers/neurons, parameter bounds, nonlinearity) implicit to allow flexibility of implementation; we will specialize to a concrete class when providing theoretical guarantees.
$\sigma(i) \neq i$ , for all  $i = 1, \ldots, n$ . The neural estimator of  $\overline{\mathrm{S}}\overline{\mathrm{l}}_k(X;Y)$  is now given by
$$
\hat {\mathsf {S l}} _ {k} ^ {\mathcal {F} _ {\mathrm {n n}}} \left(X ^ {n}, Y ^ {n}\right) := \sup  _ {f \in \mathcal {F} _ {\mathrm {n n}} ^ {\mathrm {p r o j}}} \frac {1}{n} \sum_ {i = 1} ^ {n} f \left(X _ {i}, Y _ {i}\right) - \log \left(\frac {1}{n} \sum_ {i = 1} ^ {n} e ^ {f \left(X _ {i}, Y _ {\sigma (i)}\right)}\right), \tag {5}
$$
where  $\mathcal{F}_{\mathrm{nn}}^{\mathrm{proj}} \coloneqq \left\{f \circ (\mathfrak{p}^{\mathrm{A}}, \mathfrak{p}^{\mathrm{B}}) : f \in \mathcal{F}_{\mathrm{nn}}, (\mathrm{A}, \mathrm{B}) \in \mathrm{St}(k, d_x) \times \mathrm{St}(k, d_y)\right\}$  is the composition of the neural network class with the projection maps. The projection maps can be absorbed into the network architecture as a first linear layer that maps the  $(d_x + d_y)$ -dimensional input to a  $2k$ -dimensional feature vector, which is then further processed by the original  $f \in \mathcal{F}_{\mathrm{nn}}$  network. Note that projection onto the Stiefel manifold can be efficiently and differentiably computed via QR decomposition. Hence, the Stiefel manifold constraint can be easily enforced by setting  $A, B$  to be the projections of unconstrained  $d \times k$  matrices. Further details on the implementation are given in Supplement C.
Remark 5 (Nonlinear slicing). For learning tasks that may need more expressive representations of  $(X,Y)$ , one may employ the nonlinear mSMI variant from Section 3.3. In practice, the classes  $\mathcal{G} = \{g_{\theta}\}$  and  $\mathcal{H} = \{h_{\phi}\}$  are taken to be parametric, realized by neural networks. These networks naturally compose with the DV critic  $f_{\psi}$ , resulting in a single compound model  $f_{\psi} \circ (g_{\theta}, h_{\phi})$ .
4.2 Performance Guarantees
Neural estimation involves three sources of error: (i) function approximation of the DV potential; (ii) empirical estimation of the means; and (iii) optimization, which comes from employing suboptimal (e.g., gradient-based) routines. Our analysis provides sharp non-asymptotic bounds for errors of type (i) and (ii), leaving the account of the optimization error for future work. We focus on a class of  $\ell$ -neuron shallow ReLU networks, although the ideas extend to other nonlinearities and deep architectures. Define  $\mathcal{F}_{\mathrm{nn}}^{\ell}$  as the class of all  $f:\mathbb{R}^k\times \mathbb{R}^k\to \mathbb{R}$ ,  $f(z) = \sum_{i = 1}^{\ell}\beta_i\phi (\langle w_i,z\rangle +b_i) + \langle w_0,z\rangle +b_0$ , whose parameters satisfy  $\max_{1\leq i\leq \ell}\| w_i\| _1\vee |b_i|\leq 1$ ,  $\max_{1\leq i\leq \ell}|\beta_i|\leq \frac{a_\ell}{2\ell}$ , and  $|b_0|,\| w_0\| _1\leq a_\ell$ , where  $\phi (z) = z\lor 0$  is the ReLU activation and  $a_{\ell} = \log \log \ell \lor 1$ .
Consider the neural mSMI estimator  $\widehat{\mathsf{S}}\mathsf{l}_k^{n,\ell} \coloneqq \widehat{\mathsf{S}}\mathsf{l}_k^{\mathcal{F}_{\mathfrak{mn}}^\ell}(X^n,Y^n)$  (see (5)). We provide convergence rates for it over an appropriate distribution class, drawing upon the results of [37] for neural estimation of  $f$ -divergences. For compact  $\mathcal{X}\subset \mathbb{R}^{d_x}$  and  $\mathcal{V}\subset \mathbb{R}^{d_y}$ , let  $\mathcal{P}_{\mathrm{ac}}(\mathcal{X}\times \mathcal{Y})$  be the set of all Lebesgue absolutely continuous joint distribution  $\mu_{XY}$  with  $\operatorname{spt}(\mu_{XY})\subseteq \mathcal{X}\times \mathcal{Y}$ . Denote the Lebesgue density of  $\mu_{XY}$  by  $f_{XY}$ . The distribution class of interest is<sup>4</sup>
$$
\mathcal {P} _ {k} (M, b) := \left\{\mu_ {X Y} \in \mathcal {P} _ {\mathrm {a c}} (\mathcal {X} \times \mathcal {Y}): \begin{array}{l} \exists r \in \mathcal {C} _ {b} ^ {k + 3} (\mathcal {U}) \text {f o r s o m e o p e n s e t} \mathcal {U} \supset \mathcal {X} \times \mathcal {Y} \\ \text {s . t .} \log f _ {X Y} = r | _ {\mathcal {X} \times \mathcal {Y}}, 1 (X; Y) \leq M \end{array} \right\}, \tag {6}
$$
which, in particular, contains distributions whose densities are bounded from above and below on  $\mathcal{X} \times \mathcal{Y}$  with a smooth extension to an open set covering  $\mathcal{X} \times \mathcal{Y}$ . This includes uniform distributions, truncated Gaussians, truncated Cauchy distributions, etc. The following theorem provides the convergence rate for the mSMI neural estimator, uniformly over  $\mathcal{P}_k(M, b)$ .
Theorem 1 (Neural estimation error). For any  $M, b \geq 0$ , we have
$$
\sup  _ {\mu_ {X, Y} \in \mathcal {P} _ {k} (M, b)} \mathbb {E} \left[ \left| \overline {{\mathsf {S l}}} _ {k} (X; Y) - \widehat {\mathsf {S l}} _ {k} ^ {n, \ell} \right| \right] \leq C k ^ {\frac {1}{2}} \left(\ell^ {- \frac {1}{2}} + k n ^ {- \frac {1}{2}}\right).
$$
where  $C$  depends on  $M, b, k,$  and the radius of the ambient space  $\| \mathcal{X}\times \mathcal{Y}\| \coloneqq \sup_{(x,y)\in \mathcal{X}\times \mathcal{Y}}\| (x,y)\|$ .
The theorem is proven in Supplement A.4 by adapting the error bound from [38, Proposition 2] to hold for  $\mathsf{I}(\mathsf{A}^{\mathsf{T}}X;\mathsf{B}^{\mathsf{T}}Y)$ , uniformly over  $(\mathrm{A},\mathrm{B})\in \mathrm{St}(k,d_x)\times \mathrm{St}(k,d_y)$ . To that end, we show that for any  $\mu_{XY}\in \mathcal{P}_k(b,M)$ , the log-density of  $(\mathrm{A}^{\mathsf{T}}X,\mathrm{B}^{\mathsf{T}}Y)\sim (\mathfrak{p}^{\mathrm{A}},\mathfrak{p}^{\mathrm{B}})_{\sharp}\mu_{XY}$  admits an extension (to an open set containing the support) with  $k + 3$  continuous and uniformly bounded derivatives.
Remark 6 (Parametric rate and optimality). Taking  $\ell \asymp n$ , the resulting rate in Theorem 1 is parametric, and hence minimax optimal. This result implicitly assumes that  $M$  is known when picking the neural net parameters. This assumption can be relaxed to mere existence of (an unknown)  $M$ , resulting in an extra polylog  $(\ell)$  factor multiplying the  $n^{-1/2}$  term.
Here,  $\mathcal{C}_b^s (\mathcal{U})\coloneqq \{f\in \mathcal{C}^s (\mathcal{U}):\max_{\alpha :\| \alpha \| _1\leq s}\| D^\alpha f\|_{\infty ,\mathcal{U}}\leq b\}$ , where  $D^{\alpha},\alpha = (\alpha_{1},\ldots ,\alpha_{d})\in \mathbb{Z}_{\geq 0}^{d}$ , is the partial derivative operator of order  $\sum_{i = 1}^{d}\alpha_{i}$ . The restriction of  $f:\mathbb{R}^d\to \mathbb{R}$  to  $\mathcal{X}\subseteq \mathbb{R}^d$  is  $f|_{\mathcal{X}}$ .
Remark 7 (Comparison to average-SMI). Neural estimation of classic mutual information under the framework of [38] requires the density to have Hölder smoothness  $s \geq \lfloor (d_x + d_y) / 2 \rfloor + 3$ . For  $\overline{\mathrm{SI}}_k(X;Y)$ , smoothness of  $k + 3$  is sufficient (even though the ambient dimension is the same), which means it can be estimated over a larger distribution class. Similar gains in terms of smoothness levels were observed for aSMI in [26]. Nevertheless, we note that mSMI is more compatible with neural estimation than average-slicing [25, 26]. The mSMI neural estimator integrates the max-slicing into the neural network architecture and optimizes a single objective. The aSMI neural estimator from [26] requires an additional Monte Carlo integration step to approximate the integral over the Steifel manifolds. This results in an extra  $k^{1/2}m^{-1/2}$  term in the error bound, where  $m$  is the number of Monte Carlo samples, introducing a burdensome computational overhead (see Section 5.1).
Remark 8 (Non-ReLU networks). Theorem 1 employs the neural estimation bound from [38], which relies on [52] to control the approximation error. As noted in [38], their bound extends to any other sigmoidal bounded activation with  $\lim_{z\to -\infty}\sigma (z) = 0$  and  $\lim_{z\to \infty}\sigma (z) = 1$  by appealing to the approximation bound from [53] instead. Doing so would allow relaxing the smoothness requirement on the extension to  $r\in \mathcal{C}_b^{k + 2}$  in (6), but at the expense of scaling the hidden layer parameters as  $\ell^{1 / 2}\log \ell$  (as opposed to the ReLU-based bound, where the parameter scale is independent of  $\ell$ ).
5.1 Neural Estimation
We compare the performance of neural estimation methods for mSMI and aSMI on a synthetic dataset of correlated Gaussians. Let  $X, Z \sim \mathcal{N}(0,1)$  be i.i.d. and set  $Y = \rho X + \sqrt{1 - \rho^2} Z$ , for  $\rho \in (0,1)$ . The goal is to estimate the  $k$ -dimensional mSMI and aSMI between  $(X,Y)$ . We train our mSMI neural estimator and the aSMI neural estimator from [26, Section 4.2] based on  $n$  i.i.d. samples, and compare their performance as a function of  $n$ . Both average and max-sliced algorithms converge at similar rates; however, aSMI has significantly higher time complexity due to the need to train multiple neural estimators (one for each projection direction). This is shown in Figure 1, where we compare the average epoch time for each algorithm against the dataset size. Implementation details are given in Supplement C.
Figure 1: Neural estimation performance with  $\rho = 0.5$ . Convergence vs.  $n$  in upper figures and average epoch time vs.  $n$  in lower figure.
5.2 Independence Testing
In this experiment, we compare mSMI and aSMI for independence testing. We follow the setting from [26, Section 5], generating  $d$ -dimensional samples correlated in a latent  $d'$ -dimensional subspace and estimating the information measure to determine dependence. We estimate the aSMI with the method from [26], using  $m = 1000$  Monte Carlo samples and the Kozachenko-Leonenko estimator for the mutual information between the projected variables [54]. We then compute AUC-ROC over 100 trials,
Figure 2: ROC-AUC comparison. Dashed and solid lines show results for aSMI [26] and mSMI (ours), respectively. Blue plots correspond to  $(d,d^{\prime}) = (10,4)$ , while red correspond to  $(d,d^{\prime}) = (20,6)$ .
considering various ambient and projected dimensions. For mSMI, as we cannot differentiate through the Kozachenko-Leonenko estimator, we resort to gradient-free methods. We employ the LIPO algorithm from [55] with a stopping criterion of 1000 samples. This choice is motivated by the Lipschitzness of  $(\mathrm{A},\mathrm{B})\mapsto \mathsf{l}(\mathrm{A}^{\intercal}\mathrm{X};\mathrm{B}^{\intercal}\mathrm{Y})$  w.r.t. the Frobenius norm on  $\operatorname {St}(k,d_x)\times \operatorname {St}(k,d_y)$  (cf. [26, Lemma 2]). Figure 2 shows that when  $k > 2$ , mSMI captures independence better than aSMI, particularly in the lower sample regime. We hypothesize that this is due to the fact that the shared signal lies in a low-dimensional subspace, which mSMI can isolate and perhaps better exploit than aSMI, which averages over all subspaces. When  $k$  is much smaller than the shared signal dimension  $d^{\prime}$ , mSMI fails to capture all the information and aSMI, which takes all slices into account, may be preferable. Results are averaged over 10 seeds. Further implementation details are in Supplement C.
5.3 Multi-View Representation Learning
We next explore mSMI as an information-theoretic generalization of CCA by examining its utility in multi-view representation learning—a popular CCA application. Without using class labels, we obtain mSMI-based  $k$ -dimensional representations of the top and bottom halves of MNIST images (considered as two separate views of the digit image). This is done by com
putting the  $k$ -dimensional mSMI between the views and using the maximizing projected variables as the representations. We compare to similarly obtained CCA-based representations, following the method of [20]. Both linear and nonlinear (parameterized by an MLP neural network) slicing models are optimized with similar initialization and data but different loss functions. Performance is evaluated via downstream 10-class classification accuracy, utilizing the learned top-half representations. Results are averaged over 10 seeds. As shown in Table 1, mSMI outperforms CCA for learning meaningful representations. Interestingly, linear representations learned by mSMI outperform nonlinear representations from the CCA methodology, demonstrating the potency of mSMI. Full implementation details and additional results are given in Supplements C and D, respectively.
The aSMI is not considered for this experiment since it does not provide a concrete latent space representation (as it is an averaged quantity). Moreover, if one were to maximize aSMI as an objective to derive such representations, this would simply lead back to computing mSMI; cf. Remark 3.
k	Linear CCA	Linear mSMI	MLP DCCA	MLP mSMI
1	0.261±0.03	0.274±0.02	0.284±0.03	0.291±0.02
2	0.32±0.02	0.346±0.02	0.314±0.03	0.417±0.02
4	0.42±0.01	0.478±0.02	0.441±0.04	0.546±0.01
8	0.553±0.03	0.666±0.01	0.645±0.02	0.665±0.01
12	0.614±0.02	0.751±0.01	0.697±0.01	0.753±0.01
16	0.673±0.02	0.775±0.01	0.730±0.02	0.779±0.01
20	0.704±0.007	0.79±0.006	0.774±0.01	0.798±0.01
Table 1: Downstream classification accuracy from MNIST representations by CCA and mSMI.
5.4 Learning Fair Representations
Another common application of dependence measures is learning fair representations of data. We seek a data transformation  $Z = f(X)$  that is useful for predicting some outcome or label  $Y$ , while being statis
tically independent of some sensitive attribute  $A$  (e.g., gender, race, or religion of the subject). In other words, a fair representation is one that is not affected by the subjects' protected attributes so that downstream predictions are not biased against protected groups, even if the training data may have been biased. Following the setup of [39], we measure utility and fairness using the HGR maximal correlation  $\rho_{\mathrm{HGR}}(\cdot ,\cdot) = \sup_{h,g}\rho \bigl {(}h(\cdot),g(\cdot)\bigr)$ , seeking large  $\rho_{\mathrm{HGR}}(Z,Y)$  and small  $\rho_{\mathrm{HGR}}(Z,A)$  where  $h$  and  $g$  are parameterized by neural networks. As solving this minimax problem directly is difficult in practice, following [39] we learn  $Z$  by optimizing the bottleneck equation  $\rho_{\mathrm{HGR}}(Z,Y) - \beta \overline{\mathsf{S}}\mathsf{l}_k(Z,A)$  where we use a neural estimator for the mSMI and  $\beta ,k$  are hyperparameters.
Table 2 shows results on the US Census Demographic dataset extracted from the 2015 American Community Survey, which has 37 features collected over 74,000 census tracts. Here  $Y$  is the fraction of children below the poverty line in a tract, and  $A$  is the fraction of women in the tract. Following the same experimental setup as [39], the learned  $Z$  is 80-dimensional. As [39] showed that their "Slice" approach significantly outperformed all other baselines on this experiments under a computational
Table 2: Learning a fair representation of the US Census Demographic dataset, following the setup of [39]. Results are shown as the median over 10 runs with random data splits. The fairest result is  $k = 6$ .
N/A	Slice [39]	mSMI (ours)
k=1	k=2	k=3	k=4	k=5	k=6	k=7
ρHGR(Z,Y)↑	0.949	0.967	0.955	0.958	0.952	0.942	0.940	0.957	0.933
ρHGR(Z,A)↓	0.795	0.116	0.220	0.099	0.067	0.048	0.029	0.026	0.047
(b) MNIST,  $k = 5$
(c) MNIST,  $k = 10$
Figure 3: MNIST images generated via the max-sliced InfoGAN.
constraint $^5$ , we apply the same computational constraint to our approach and compare only to Slice and to the "N/A" fairness-agnostic model trained on the bottleneck objective with  $\beta = 0$ . Note that for  $k > 1$ , mSMI learns a more fair representation  $Z$  (lower  $\rho_{\mathrm{HGR}}(Z, A)$ ) than Slice, while retaining a utility  $\rho_{\mathrm{HGR}}(Z, Y)$  on par with the fairness agnostic N/A model. We emphasize that due to the reasons outlined in Section 5.3, aSMI is not suitable for the considered task and is thus not included in the comparison. Results on the Adult dataset are shown in Supplement E.
5.5 Max-Sliced InfoGAN