Scalable Infomin Learning
Authors: Yanzhi Chen
Weihao Sun
Yingzhen Li
Adrian Weller
Venue: NeurIPS 2022 Accept (2022)
Keywords: mutual information
representation learning
fairness
disentangled representation learning
domain adaptation

Abstract: The task of infomin learning aims to learn a representation with high utility while being uninformative about a specified target, with the latter achieved by minimising the mutual information between the representation and the target. It has broad applications, ranging from training fair prediction models against protected attributes, to unsupervised learning with disentangled representations. Recent works on infomin learning mainly use adversarial training, which involves training a neural network to estimate mutual information or its proxy and thus is slow and difficult to optimise. Drawing on recent advances in slicing techniques, we propose a new infomin learning approach, which uses a novel proxy metric to mutual information. We further derive an accurate and analytically computable approximation to this proxy metric, thereby removing the need of constructing neural network-based mutual information estimators. Compared to baselines, experiments on algorithmic fairness, disentangled representation learning and domain adaptation verify that our method can more effectively remove unwanted information with limited time budget.

Scalable Infomin Learning
Yanzhi Chen', Weihao Sun?, Yingzhen Li", Adrian Weller 'University of Cambridge, 'Rutgers University, 'Imperial College London, *Alan Turing Institute
Abstract
The task of infomin learning aims to learn representation with high utility while being uninformative about a specified target, with the latter achieved by minimis- ing the mutual information between the representation and the target. It has broad applications, ranging from training fair prediction models against protected at- tributes, unsupervised learning with disentangled representations. Recent works on infomin learning mainly use adversarial training, which involves training a neural network to estimate mutual information or its proxy and thus is slow and difficult to optimise. Drawing on recent advances in slicing techniques, we pro- pose new infomin learning approach, which uses a novel proxy metric to mutual information. We further derive an accurate and analytically computable approx- imation this proxy metric, thereby removing the need of constructing neural network-basee mutual information estimators. Experiments on algorithmic fair- ness, disentangled representation learning and domain adaptation verify that our method can effectively remove unwanted information with limited time budget.
1 Introduction
Learning representations that uninformative about some target but still useful for downstream applications is an important task in machine learning with many applications in areas including algorithmic fairness [1, 2, 3, 4], disentangled representation learning [5, 6, 7, 8], information bottle- neck [9, 10), and invariant representation learning [11, 12, 13, 14).
A popular method for the above task is adversarial training [1, 3, 4, 7, 15, 2, 111, where two neural networks, namely the encoder and the adversary, are trained jointly to compete with each other. The encoder's s goal is to learn a representation with high utility but contains information about information about target from the
target. The adversary, on the contrary, tries to recover learned representation much as possible. This leads to minmax game similar to that in generative adversarial networks [16]. Adversarial training is effective with a strong adversary, however, is often challenging to train the adversary thoroughly in practice, due to time constraints and/or optimisation difficulties [17, 18]. In fact, recent studies have revealed that adversarial approaches may not faithfully produce infomin representation in some cases [19, 20, 18, 21, 22]. This motivates us to seek a good, adversarial training- free alternative for scalable infomin learning.
In this work, we propose new method for infomin learning which is almost as powerful as using a strong adversary but is highly scalable. Our method inspired by recent advances in information theory which proposes to estimate mutual information in the sliced space [23]. We highlight the following contributions:
We show that for infomin learning, accurate estimate of mutual information (or its bound) is unnecessary: testing and optimising statistical independence in some sliced spaces is sufficient;
We develop analytical approximation to such sliced independence test, along with scalable algorithm for infomin learning based on this approximation. No adversarial training is required.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Importantly, the proposed method can be applied to wide range of infomin learning tasks without any constraint on the form of variables or any assumption about the distributions. This contrasts our method to other adversarial training- -ree methods which are either tailored for discrete or univariate variables [24, 25, 25, 26, 22] or rely on variational approximation to distributions [10, 27, 19, 12].
2
Background
Infomin representation learning. Let X E RD' be the data, Y RD be the target we want to predict from X. The task we consider here is to learn some representation Z = f(X) that is useful for predicting Y but is uninformative about some target T E R Formally, this can be written as min ((f(():)) 3 I(f()):T
(1)
where f is an encoder, C is some loss function quantifying the utility of Z for predicting Y and [ff()); quantifies the amount of information left in 2 about T. 3 controls the trade-off between utility and uninformativeness. Many tasks in machine learning can be seen as special cases of this objective. For example, by setting 7 to be (a set of) sensitive attributes e.g. race, gender age, arrive at fair representation learning [28, 1, 2, 21]. When using a stochastic encoder, by setting Y to be X and T to be some generative factors e.g.. class label, arrive at disentangled representation learning [5, 6, 7. 8). Similarly, the information bottleneck method [9, 10] corresponds to setting T = X, which learns representations expressive for predicting Y while being compressive about X.
Adversarial training for infomin learning. A key ingredient in objective (1) is to quantify ((f()); as the informativeness between f(X) and T. One solution is to train a predictor for T' from f(X) and use the prediction error as a measure of ((f()): [1. 3, 4, 29]. Another ap- proach is to first train classifier to distinguish between samples from p(Z, T)) vs. p(Z)p(T) [7, 15] or to distinguish samples Z ~ p([[T) with different T [2. 11], then use the classification error to quantify (xx T). All these methods involve the training of neural network t to provide power-bound estimate to [[f(X); T). yielding a minmax optimisation problem
min max ((f(X);)) + B (((()):TT
(2)
where I)) is estimator constructed using t that lower- bounds ((f()): The time complexity of optimising (2) is ال,ي where L1 and Le are the number of gradient steps for the min and the max step respectively. The strength of t is crucial for the quality of the Iearned repre- sentation [19, 20, 18, 21, 22). For a strong adversary, a large L2 is possibly needed, but this means long training time. Conversely, weak adversary may not produce truly infomin representation.
3 Methodology
We propose an alternative to adversarial training for optimising (1). Our idea is to learn representa- tion by the following objective, which replaces ((x()); 7) in objective (1) with its 'sliced' version:
B ():
(3)
where 57 denotes the sliced mutual information, which was also considered in [23]. Informally, ST is 'facet' of mutual information that is much easier estimate (ideally has closed form) but can still to some extent reflect the dependence between Z and T. Optimising (3) is then equivalent to testing and minimising the dependence between Z and T from one facet. Importantly, while testing dependence through only a single facet may be insufficient, by testing and minimising dependence through various facets across a large number of mini-batches we eventually see I I(Z:T) - O.
We show one instance for realising SI whose empirical approximation $1 has an analytic expres- sion. The core of our method is Theorem 1, which is inspired by [4, 23].
Theorem 1. Let Z E R. and T E R be two random variables that have moments. Z and T are statistically independent and only if SI, T) where ST(Z, T) is defined as follows
SI(Z:T)= sup p(h(0) Z).g( T)),
(4)
b.
where o the Pearson correlation, h : - R are measl-meas non-constant functions, and i E SD-1, @ E 8d-1 are vectors on the surfaces on D-dimensional and d-dimensional hyperspheres.

---

Proof. See the Appendix.
We sketch here how this result relates to [4, 23]. [23] considers TI) defined as the expected mutual information ([T Z, oJ TTT of slices of Z and Т, where the expectation is taken over respective Haar measures 0 E SD-1, Φ gd-1, Instead of considering the mutual information T(O" 2, ') in the average case, we handle Pearson correlation over the supreme functions h, 9 as defined above, which links to Rényi's maximal correlation [4, 30, 31, 32] and has some interesting properties suitable for infomin representation learning (explained later).
We call 0 and & the slices for Z and ''' respectively, and 0 Z, Φ T the sliced Z and ' respectively.
Intuitively, Theorem says that in order to achieve (Z: - O, we need not estimate [ T) in the original space; rather we can test (and optimise) independence in the sliced space as realised by (4). Note other realisations of the sliced mutual information S/ can also be used. The major merit of the realisation (4) is that (a) it is bounded in [0, 1]. which eases hyperparameter tuning and makes learning stable; (b) allows an analytic expression for its empirical approximation, as shown below.
Analytic approximation to SI. An empirical approximation to (4) is
TI zs sup sup p(h,(0) Z), g),
i.j hug)
(5) i.e. we approximate (4) by randomly sampling number of slices 0, á uniformly from the surface of two hyperspheres gD and g and pick those slices where the sliced Z and the sliced T maximally associated. With large number of slices, it is expected that (5) will approximate (4) well. refer to [23] for a theoretical analysis on the number of required slices in astimator-agnostic settings. In Appendix B we also investigate empirically how this number will affect performance.
where 0, ~ (~~(( i,.,..,
Φf ~ ~~((s)-)) 1.,,...
For each slicing direction, we further assume that the supreme functions hú, 9j : IR - for that direction can be well approximated by K-order polynomials given sufficiently large K, i.e.
K
K
(((a) Σ wa)(a))
()(a) R (a(a) = (a)*
0
where (.) is a monotonic function which maps the input to the range of [-1, 1]. Its role is to ensure that or (a) always has finite moments, so that the polynomial approximation is well-behaved. Note no information is lost by applying a.)) Here we take (.)) as the tanh function. Other approximations can also be used (e.g. those based random features [33]), which can be explored in the future.
One may wonder if ignoring the higher-order terms (a))* K' > K will cause inaccuracy in depen- dence modelling. In Appendix A we show that (ha(a) h₂(a)l S and (a)(a) g, [a)) S e for Vi,ja a, then as e 0 we have |pty - ptyl = o(e) where pi = (0,;(0, 2), T)) is the true correlation and Pig is its K-order approximation. In such case we have Pig ss 0 whenever Ply R O.
With this polynomial approximation. the solving of h4, 9j reduces to finding their weights ws Dy:
Wi, Uy arg max p(w T Z v. TT), w.v
7 = [o 2). (0 }
-o (
This is known as canonical correlation analysis [34] and can be solved analytically by eigendecom- position. Hence
find the weights for all pairs of h, 9, by doing S eigendecomposition.
In fact, the functions h, 9j for all i can be solved simultancously by performing larger eigende- composition only once. We do this by finding w, that maximise the following quantity:
$Ie.
sup p(w Z,oTT'. we
(6)
where - Z), (0,T
T
0, 0~ i- 1. oy ~ u( -= 1...
The benefits for solving Si, for all slices jointly are two-fold. The first better computational efficiency in practice, as it avoids invoking for loop and has better affinity to modern deep learning
3

---

infrastructure and libraries (e.g. Tensorflow [35] and Py [36]) which are optimised for matrix- based operations. We highlight that this joint-solving strategy will not violate our original objective (i.e. minimising (5)), as backed up by Theorem 2 below. The intuition behind Theorem 2 is that if all Z and 7 together cannot achieve a high correlation, they alone can not either:
Theorem 2. Provided that each h4s 9j in (5) are K-order polynomials given the sampled θ [0: 1:1iii: 1. Φ- [ we have STe (Z:T) S c c supi, suph,,, p(h,(0) () S c.
Proof. See Appendix A.
Theorem 2 says that the solution of (6) yields an upper bound of (5). This essentially means that we are safe replace (5) with (6). In addition computational efficiency, another benefit for solving 9j. h jointly is stronger power in independence testing. More specifically, it allows us to unify the power of different slices: while some sliced directions may be weak for detecting dependence, they together as a whole (by treating them as a 'big slice' [37]) can compensate for each other, yielding more powerful test. For these reasons, we use ST a T):T) as the approximation to ST(Z:T) T).
Algorithm 1 Adversarial Infomin Learning
Algorithm 2 Slice Infomin Learning
Input: data D = (X("), (())TT())) T(()) Jn= Output: Z f(X) that optimises (1) Hyperparams: 3, N', L1, Le
Input: data D = {x("), y("), [N Output: Z f(X) that optimises (1) Hyperparams: 3, N', S.
Parameters:
encoder f, MI estimator &
Parameters: encoder f. weights w, v in ST
for 2 in 1 to L1 do
for / in 1 to L do sample mini-batch B from D sample D' from D whose size N' < N
sample mini-batch B from D sample D' from D whose size N' N
> Max-step
Max-step
for l2 in 1 to L2 do
sample S slices θ = (0, 1 4 = 1
l I. +1 vV V. with data in D
solve the parameters w, $1 analytically
end for < Min-step
with θ 4b D' by eigendecomposition
D Min-step
{+ f (
with data in B
with data in B
end for return Z = f(X)
end for return Z = f(X)
Mini-batch learning algorithm. Given the above approximation (6) to SI, we can now elaborate the details of mini-batch learning. For each mini-batch B, we execute the following steps:
Max-step. Sample S slices θ = {0:)]-1, = { and a subset of the data D' C (here [D' can be larger than [B). Learn the weights w, of (6) with θ, 4, D' by eigendecomposition:
Min-step. Set $I 7) p(w Z', Z', TT') with w, solved in the max-step and Z', T" defined in (6). Update f by SGD: f + 1/ (L)()(( (X)( T)) with the mini-batch data.
The whole learning procedure is shown in Algorithm 2. Compared to that of adversarial methods [2, 11, 1, 3, 21, 4] as shown in Algorithm 1, we replace the optimisation of neural net in the max step with an analytical eigendecomposition step. The time complexity of eigendecomposition is O($)).
As an optional strategy, during mini-batch learning, one may actively seek more informative slices for independence testing by optimising the sampled slices with a few gradient steps (e.g. 2-5):
- θ - (e
② Φ EVaŠ1e,,
which still very cheap to execute. Such a strategy is may be useful when most of the sampled slices are weak in detecting dependence, which typically happens in later iterations of learning where ((;TT) O. We can activate such strategy whenever the estimated SI is very low (e.g. TT( < 0.05). Since the optimisation of slices only happens late learning iterations, and we only apply small number of gradient steps, the overall execution time will only increase negligibly.

---

4 Related works
Neural mutual information estimators. A set of neural network-based methods [38, 39, 40, 41] have been proposed to estimate the mutual information (MI) between two random variables, most of which work by maximising a lower bound of MI [42]. These neural MI estimators are in general more powerful than non- parammaeetrc methods [43, 44, 45, 46] when trained thoroughly, yet the time spent on training may become the computational bottleneck when applied infomin learning.
Upper bound for mutual information. Another line of method for realising the goal of infomin learning without adversarial training is to find an upper bound for mutual information [10, 19, 12, 47, 48]. However, unlike lower bound estimate, upper bound often requires knowledge of either the conditional densities or the marginal densities [42] which are generally not available practice. As such, most of these methods introduce a variational approximation to these densities whose choice/estimate may be difficult. Our method on the contrary needs not approximate any densities.
Slicing techniques. A series of successes have been witnessed for the use of slicing methods in machine learning and statistics [49], with applications in generative modelling [50, 51, 52, 53], statistical test [54] and mutual information estimate [23]. Among them, the work [23] who proposes the concept of sliced mutual information is very related to this work and directly inspires our method. Our contribution is a novel realisation of sliced mutual information suitable for infomin learning.
Fair machine learning. One application of our method is to encourage the fairness of predictor. Much efforts have been devoted for the same purpose, however most of the existing methods can either only work at the classifier level [25, 24, 4, 55], or only focus on the case where the sensitive attribute is discrete or univariate [22, 56, 26, 55, 28), or require adversarial training [2, 11, 3, 21, 4]. Our method the contrary has restriction the form of the sensitive attribute, can be used in both representation level and classifier level, and require no adversarial training of neural networks.
Disentangled representation learning. Most of the methods in this field work by penalising the discrepancy between the joint distribution P ((Z) and the product of marginals Q I ((a) has to resort to Monte Carlo estimate (3-TCVAE [27]). to train a neural network estimator (Factor VAE [7]) or to assess the discrepancy between P and Q by only their moments (DIP- [57]). Our method avoids assessing distribution discrepancy directly and instead perform independence test in the sliced space.
[6, 57, 27, 58, 7].' However, such discrepancy is often non-trivial to estimate,
5 Experiments
We evaluate our approach on four tasks: independence testing, algorithmic fairness, disentangled representation learning, domain adaptation. Code is available at github. com htt///yyzaaiiia boyzzaaiiinfoni inf omin.
Evaluation metric. To assess how much information is left in the learned representation 2 E R about the target T E K we calculate the Rényi's maximal correlation p² (Z,T) between Z and T:
p" (2, T) = sup p (h()), h.g
= 1 = g()) which has the properties PZ = 0 if and only if Z T T and =
for some deterministic functions h. [30]. One can also understand this metric as the easiness of predicting (the transformed) T from Z, or vice versa. ? As there is no analytic solution for the supre- mum in (8), we approximate them by two neural networks h. trained with SGD. Early stopping and dropout are applied to avoid overfitting. The reliability of this neural approximation has been verified by the literature [4] and is also confirmed by our experiments; see Appendix B.
This metric is closely related to existing metrics/losses used in fairness and disentangled represen- tation learning such demographic parity (DP) [24] and total correlation (TC) [7]. For example, if p² (Z, + 0 then it is guaranteed that Ý - T for any predictor Y F(Z), p² (Z T) is an upper bound for DP Similarly, p² (Z, T) coincides with TC which also assesses whether Z 1 T. In additional to this metric, we will also use some task-specific metric; see each experiment below.
Baselines. We compare the proposed method (denoted as "Slice"" with the following approaches:
I Note there exist methods hased on group theory [59, 60, 61] which do not assess distribution discrepancy. II can be shown p² (2,T) is equivalent the normalised mean square error between h(Z) and g(T).
5

---

tanhlel
1.2
100
1.0
Persson
0.7
0.77
0.7
dCar
0.5
0.5
0.5
0.4 0.8
1.0
(a) a
(b) a²
(c) sin(a)
(d) tanh(a)
Figure 1: Comparison of the test power of different independence test methods. The x-axis corre-
sponds to different values for the dependence level and the y-axis corresponds to the test power.
Pearson, which quantifies I (Z;T) by the Pearson correlation coefficient DK Ex It was used in [5, 57] an easy-to-compute proxy to MI to learn disentangled representations;
Σx p(Zx; Tt).
dCorr, i.c. distance correlation, a non- non-parametric method for the quantifying the independence between two vectors [46]. was applied [41] as a surrogate MI for representation learning;
Neural Rényi, adversarial method for fair machine learning [25] which quantifies I(Z; T) by the Rényi correlation p² 7) suph, p(h( Z), (T)) with à. approximated by neural net- works. It can be seen as training predictor to predict (the transformed) T from Z and is closely related to many existing methods in algorithmic fairness and domain adaptation [1, 2, 3, 4, 29];
Neural TC, an adversarial method for learning disentangled representation [7, 62] which quanti- fies (Z; by the total correlation TC(Z,T) T) = K LL [p( 2, ())pp(()p((T) To computes TC, classifier is trained to classify samples from pl Z, T') and samples from p p())p(T). This method also be seen variant of the popular MINE method [38] for mutual information estimate.
y- y-CLUB, i.e. variational contrastive log upper bound, which introduces a (learnable) variational distribution ZT) form an upper bound of MI [47]: I(Z:T) S Ep(Z.T [log (T') 2)] Ep( [log []] Like adversarial method, ( 2) can be learned by few gradient steps.
For fair comparison, for adversarial raining-based approaches (i.e. Neural Rényi, Neural TC), we ensure that the training time of the neural networks in these methods least the same as the execution time of our method or longer. We do this by controlling the number of adversarial steps L2 in Algorithm The same setup used for v-CLUB. See each experiment for the detailed time.
Hyperparameter settings. Throughout our experiments, we use 200 slices. We find that this setting robust across different tasks. An ablation study the number of slices given in Appendix B. The order of the polynomial used in (6) namely K is set as K = 3 and is fixed across different tasks.
Computational resource. All experiments are done with single NVIDIA GeForce Tesla T4 GPU.
5.1 Independence testing
We first verify the efficacy of our method as leight but powerful independence test. For this purpose, we investigate the test power of the proposed method on various synthetic dataset with different association patterns between two random variables X E R10 Y E IR. and compared to that of the baselines. The test power is defined as the ability to discern samples from the joint distribution p(X, Y) and samples from the product of marginal p( (Y))(Y) and is expressed as a ae, Xa ~ 11-3,3), 3), A = A = 0.2, E ~ (Ne:; 0, I), a E (0, and (-) is a scaling operation that scales the operand to the range of [0, 1] according to the minimum and maximum values in the population. The function t(-) determines the association pattern between X and Y and is chosen from one of the following: t(a) = a² sin(a), tanh(a). The factor controls the strength of dependence between X and Y.
probability p E [0. 1]. The data is generated as Y (1 ax) (I)(A)))))
All tests are done on 100 samples and are repeated 1,000 times. We choose this sample number as it is a typical batch size in mini-batch learning. For methods involving the learning of parameters (i.e. Slice, Neural Rényi, Neural TC), we learn their parameters from 10,000 samples. The time for learning the parameters of Slice, Neural Rényi and Neural TC are 0.14s, 14.37s, 30. 30.18s 18s respectively. For completeness, we also compare with the 'optimal test' which calculates the Rényi correlation (x, Y) = ((XX))(())) with the functions h, exactly the same as the data generating process.

---

Table 1: Learning fair representations on the US Census Demographic dataset. Here the utility of the representation is measured by 0² (Z Y), while (Z. is used to quantify the faimess of the representation. Training time is also provided as the seconds required per max step.
N/A
Pearson
dCorr
Slice
Neural Rényi
Neural TC
vCLUB
1
0.95
0.00
0.95 +: 0.00
0.95
0.00
0.95 + 0.01
0.95 d: 0.01
0.95 +: 0.02
0.94 0.02
('(2,7)
0.92
0.02
0.84 + 0.08
U.47 0.08
0.07 0.02
0.23 + 0.10
0.27 + 0.03
0.16 0.10
time (sec./max step)
0.000
0.012
0.087
0.102
0.092
0.097
0.134
Table 2: Learning fair representations on the UCI Adult dataset. Here the utility of the representation measured by p" (Z, while p² 7) is used to quantify the faimess of the representation.
N/A
Pearson
dCorr
Slice
Neural Rényi
Neural TC
vCLUB
0.99 + 0.00
0.99 + 0.00
0.97 = 0.01
0.98 + 0.01
0.97 = 0.01
0.98 + 0.02
0.97 + 0.02
(²(Z,T) 4
0.94 + 0.02
0.91 + 0.06
0.71 # 0.06
0.08 + 0.02
0.17 + 0.08
0.36 + 0.13
0.26 + 0.12
time (soc./max step)
0.000
0.015
0.071
0.112
0.107
0,131
0,132
Figure shows the power of different methods under various association patterns and dependence levels a. Overall, we see that the proposed method can effectively detect dependence in all cases, and has test power comparable to neural network-based methods. Non-parametric tests, by contrast, fail to detect dependence in quadratic and periodic cases, possibly due to insufficient power. Neural TC is the most powerful test among all the methods considered, yet it requires the longest time to train. We also see that the proposed method is relatively less powerful when a 2 0.8, but in such cases the statistical dependence between X and Y is indeed very weak (also see Appendix B). The results suggest that our slice method can provide effective training signals for infomin learning tasks.
5.2 Algorithmic fairness
For this task, aim to learn fair representations Z E I that are minimally informative about some sensitive attribute T. We quantify how sensitive Z is w.r. T by Rényi correlation p² (²( calculated using two neural nets. Smaller p² (Z, T) is better. The utility of the learned representation i.e., CZ Y) is quantified by (2, Y). This formulation for utility, as aforementioned, is equivalent to measuring how well we can predict Y from Z. In summary, the learning objective is:
max p" (Z;Y) Bi(
where iZ; T) is estimated by the methods mentioned above. For each dataset considered, we use 20,000 data for training and 5,000 data for testing respectively. We carefully tune the hyperparameter 8 for each method that the utility p² (Z; Y) of that method is close to that of the plain model (i.c. the model trained with 3 = 0, denoted as N/A below; other experiments below have the same setup). For all methods, we use 5, 000 samples in the max step (so N' = 5, 000 in Algorithm 1, 2).
US Census Demographic data. This dataset is an extraction of the 2015 American Community Survey, with 37 features about 74,000 census tracts. The target Y to predict is the percentage of children below poverty line tract, and the sensitive attribute T is the ratio of women in that tract. The result is shown in Table 1. From the table we see that the proposed slice method produces highly fair representation with good utility. The low (Z, T) value indicates that it unlikely to predict T from Z in our method. While adversarial methods can also to some extent achieve fairness, it is still not comparable to our method, possibly because the allocated training time is insufficient (in Appendix B we study how the effect of the training time). Non- pon-maraettcricc methods can not produce truly fair representation, despite they are fast to execute. v-CLUB, which estimates an upper bound of MI, achieves better fairness than adversarial methods on average, but has higher variance [63].
UCI Adult data. This dataset contains census data for 48,842 instances, with 14 attributes describ- ing their education background, age, race, marital status, etc. Here, the target Y to predict is whether the income of an instance is higher than 50,000 USD, and the sensitive attribute
the race group. The result is summarised in Table 2. Again, we see that the proposed slice method outperforms other methods terms of both fairness and utility. For this dataset, Neural Rényi also achieves good fairness, although the gap to our method is still large. Neural TC, by contrast, can not achieve
7

---

X3
G t.)
GiZ .g)
G14), F2)
Glay,
(a) Adversarial training G)Z..
Gizu (2)
22
G122. 577
(b) Slice
Figure 2: Label swapping experiments Dsprite dataset. Left: the original image X. Middle:
reconstructing X R G(Z, T) using Z ( and true label T. Right: reconstructing X' G(Z,T") using Z ( and swapped label T" T. Changing T should only affects the style.
Gizz, 6))
GiZ
O127,
G12,.57 57
(a) Adversarial training G(2yG))
Giz
O12
(b) Slice
Figure 3: Label swapping experiments on CMU-PIE dataset. Left: the original image X. Middle: reconstructing X x G(Z, T)) using Z E(X) and the true label T. Right: reconstructing X' G(Z,T") using Z ( and swapped label T" + T. Changing T only affect the expression.
comparable level fairness under the time budget given - a phenomenon also observed Census dataset. This is possibly because the networks in Neural TC require longer time to train. The v-CLUB method does not work very satisfactorily on this task, possibly because the time allocated to learn the variational distribution q( (T))) not enough, leading loose upper bound of (Z)
US
5.3 Disentangled representation learning
We next apply our method to the task of disentangled representation learning, where we wish to discover some latent generative factors irrelevant the class label T. Here, train conditional autoencoder X zJ G(Z, T) to learn representation Z E(X) which encodes label-irrelevant in- formation of X. The target to recover is X. The utility of Z is therefore quantified as the reconstruction error: L(Z; = E[| 7) - x11) resulting in the following learning objective:
max [[[
x1 (Z:
The conditional autoencoder uses a architecture similar to that convolutional GAN [64], with the difference being that we insert adaption layer 7' MIP(Z, T) before feeding the features to the decoder. See Appendix B for the details of its architccture. All images are resized to 32 x 32. For all methods, we use 10, 000 samples in the max step (so N' 10, 000 in Algorithms and 2).
Dsprite. 2D shape dataset [65] where each image generated by four latent factors: shape, rotation, scale, locations. Here the class label T is the shape, which ranges from (square, ellipse, heart). For this dataset, we train the autoencoder 100 iterations with batch size of 512. The dimen- sionality of the representation for this task is 20 i.e. Z E 20 As in the previous experiments, we provide quantitative comparisons of utility and disentanglement of different methods in Table 3. In addition, we provide qualitative comparison in Figure 2 which visualises the original image, the reconstructed image and the reconstructed image with swapped label. From Table 3, we see that the proposed method achieve very low p² (2, T) while maintaining good MSE, suggesting that we

---

Table 3: Learning label-irrelevant representations on the Dsprite dataset. Here the utility of the representation is measured by MSE, while p (Z. T) is used to quantify the level of disentanglement of the representation. Training time is also provided the seconds needed per max step. ATc(T)) is the accuracy trying to predict T from Z. As there are classes, the ideal value for Acc is 0.33*.
N/A
Pearson
dCorr
Slice
Neural Rényi
Neural TC
vCLUB
MSE
0.37 0.01
0.44 # 0.02
0.55 = 0.03
0.50 + 0.01
0.61 + 0.04
0.49 0.03
0.65 + 0.04
p' (2(
0.91 +: 0.03
0.81 # 0.07
0.62 = 0.07
0.08 + 0.02
0.48 = 0.05
0.31 = 0.06
0.22 :: 0.08
Acc(T)
0.98 0.01
0.89 +: 0.03
0.76
0.05
0.32 0.02
0.55 +: 0.04
0.54 0.04
0.48 0.03
time (scc./max step)
0.000
0.201
0.412
0.602
0.791
0.812
0.689
Table 4: Learning label-irrelevant representations on the CMU-PIE dataset. Here the utility of the representation measured by MSE, while p² p² T) is used to quantify the level of disentanglement of the representation. Training time is also provided as the seconds needed per max step. AT) is the accuracy trying to predict T from Z. As there are 2 classes, the ideal value for Acc(T) is 0.50*.
N/A
Pearson
dCorr
Slice
Neural Réayi
Neural TC
vCLUB
MSE L
1.81 + 0.04
1.85 + 0.05
2.08 + 0.08
2.15 + 0.07
2.46 + 0.06
1.99 + 0.12
2.02 + 0.10
0.76
0.04
0.55 + 0.03
U.27 + 0.07
0.07 + 0.01
0.36 + 0.04
0.39 + 0.06
0.16 0.06
Aco(T)
0.91 + 0.00
0.76 + 0.03
0.71 + 0.06
0.51 + 0.03
0.73 + 0.03
0.76 + 0.04
0.68 + 0.05
time (sec./max step)
0.000
0.184
0.332
0.581
0.750
0.841
0.621
*For the plain model, Ace(T)) is not necessarily around 1.0, Z needs not encode all content of the image.
may have discovered the true label-irrelevant generative factor for this dataset. This is confirmed vi- sually by Figure 2(b), where by changing 7 in reconstruction we only change the style. By contrast, the separation between T and Z less evident in adversarial approach, as can be seen from Table 3 as well from Figure 2(a) (see c.g. the reconstructed ellipses in the third column of the figure. They are more like a interpolation between ellipses and squares).
CMU-PIE. A colored face image dataset [66] where each face image has different pose, illumina- tion and expression. We use its cropped version [67]. Here the class label T is the expression, which ranges from (neutral, smile). We train an autoencoder with 200 iteration and a batch size of 128. The dimensionality of the representation for this task is 128 i.e. Z E R 128 Figure 3 and Table 4 shows the qualitative and quantitative results respectively. From Figure 3, we see that our method can well disentangle expression and non-expression representations: one can easily modify the expression of a reconstructed image by only changing T. Other visual factors of the image including pose, illu- mination, and identity remain the same after changing T'. Adversarial approach can to some extent achieve disentanglement between Z and T, however such disentanglement is imperfect: not all of the instances can change the expression by only modifying T. This is also confirmed quantitatively by Table 4, where one can see the relatively high o² (Z, I') values in adversarial methods. For this task, v- v-CLUB also achieves a low p" (Z, T) value, though it is still outperformed by our method.
5.4 Domain adaptation
We finally consider the task of domain adaptation, where we want to learn some representation Z that can generalise across different datasets. For this task, a common assumption that we have assess to two dataset D.- (x(()) y( and D, = (XO)] 1m 1 whose classes are the same but are collected differently. Only the data in D, has known labels. Following [47], we learn Z as follows:
Z. = fc(X),
Za= fxx)
L. = Ex,YED T log C(Z,),, min L. + L.a + 3i (Ze, Za).
ExED (log D(Za)) + ExED, [log(1 - D(2a))},
where Ze, Za are disjoint parts of Z that encode the content information and the domain information of separately. C is the content classifier that maps X to (K 1)-simplex (K is the number of classes) and D is the domain classifier that distinguishes the domain from which X comes. Since the

---

Doritin
Domain
(a) M 7 MM, adversarial
(b) M + MM, slice
(c) C S, adversarial
(d) C + S, slice
Figure 4: T-SNE plots of the learned content representations Ze domain adaptation tasks. (a)(c)
show the cases when the adversary is not trained thoroughly (i.e. L2 in Algorithm 1 is set too small).
Table 5: Learning domain-invariane representations. Here Acc(Y,) and Acc(Y,) are the classifica- tion accuracy the source and the target domains respectively. Time used per max step given.
MNIST MNIST-M
CIFARIO -
Neural
Stice
Acc(Y))
99.3 + 0.04
99.2 0.02
99.2 à 0.04
99.0 + 0.00
Acs(i))
90.0 = 0:03
92.5 + 0.03
92.4 0.00
92.1 0.04
A
46.3 0.03
0:45
80.1 0.17
93.8 4 0.10
Acc(K)) 1
75.9 0:09
82.3 4 0.88
0.08
78.5 4: 0:11
o' (Z,,. 2z)
0.86 0.05
0.06 0.01
0.64 à: 0.04
0.49 4 0.12
(Zl,, Ze)
0.43 +: 0.05
0.08 + 0.01
0.39 0.07
0.42 4: 0.09
time (sec.
0.000
2,578
3,282
3.129
time (sec./xep)
0.000
3.146
3.222
3,080
classifier C' only sees labels in Dz, call D. the source domain and D, the target domain. For the two encoders fo and fa, we use Resnets [68] with 7 blocks trained with 100 iterations and a batch size of 128. Here Ze Za E R256, We use N' 5, 000 samples in the max step for all methods.
MNIST + MNIST-M. Two digit datasets with the same classes but different background colors. Both datasets have 50,000 training samples. Table 5 shows the result, indicating that our method can more effectively remove the information about the domain. This is further confirmed by the T-SNE [69] plot in Figure 4, where one can hardly distinguish the samples of Ze from the two domains. This naturally leads to higher target domain accuracy Acc(Y,) than other methods.
CIFARIO + STL10. Two datasets of natural images sharing 9 classes. There are 50,000 and 5,000 training samples in the two datasets respectively. Following existing works [70, 71, 47), we remove the oorrllppeppppgggg classes from both datasets. Table 5 and Figure show the result. Again, we see that our method can more effectively remove domain information from the learned representation.
Conclusion
This work proposes method for infomin learning without adversarial training. A major chal- lenge is how to estimate mutual information accurately and efficiently, as MI is generally intractable. We sidestep this challenge by only testing and minimising dependence in sliced space, which can be achieved analytically, and we showed this is sufficient for our goal. Experiments on algorithmic fairness, disentangled representation learning and domain adaptation verify method's efficacy.
Through our controlled experiments, we also verify that adversarial approaches indeed may not pro- duce infomin representation reliably observation consistent with recent studies. This suggests that existing adversarial approaches may not converge to good solutions, or may need more time for convergence, with more gradient steps needed to train the adversary fully. The result also hints at the potential of diverse randomisation methods as an alternative to adversarial training in some cases.
While we believe our method can be used in many applications for societal benefit (e.g. for promot- ing fairness), since is general technique, one must always be careful to prevent societal harms.
Acknowledgement
AW acknowledges support from Turing AI Fellowship under grant EP/V025279/1, The Alan Tur- ing Institute, and the Leverhulme Trust via CFI. YC acknowledges funding from Cambridge Trust.
10

---

References
[1] David Madras, Elliot Creager, Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. In International Conference Machine Learning, pages 3384-3393, PMLR, 2018.
[2] Harrison Edwards and Amos Storkey, Censoring representations with an adversary. In ICLR, 2016.
[3] Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 11-21, 2018.
[4] Vincent Grari, Boris Ruf, Sylvain Lamprier, and Marcin Detyniecki. Fairness-aware neural Réyni minimization for continuous features. arXiv preprint 19Xiv:9911.04990 2019.
[5] Brian Cheung. Jesse A Livezey, Arjun K Bansal, and Bruno A Olshausen. Discovering hidden factors of variation in deep networks. arXiv preprint arXiv:1412.658; 2014.
[6] Irina Higgins, Loic Matthey, Arka Pal. Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta- VAE: Learning basic visual con- cepts with constrained variational framework. 2016.
[7] Hyunjik Kim and Andriy Mnih. Disentangling by factorising. International Conference on Machine Learning, pages 2649-2658. PMLR, 2018.
[8] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In International Conference on Machine Learning, pages 4114 4124. PMLR, 2019.
[9] Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE information theory workshop (ITW), pages 1-5. IEEE, 2015.
[10] Alexander A Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational infor- mation bottleneck. arXiv preprint arXiv: 111110011000 2016.
[11] Qizhe Xie, Zihang Dai, Yulun Du Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. Advances neural information processing systems, 30, 2017.
[12] Proteek Chandan Roy and Vishnu Naresh Boddeti. Mitigating information leakage in image representations: A maximum entropy approach. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2586 2594,2594, 2019.
[13] Ayush Jaiswal, Daniel Moyer, Greg Ver Steeg, Wael AbdAlmageed, and Premkumar Natarajan. Invariant representations through adversarial forgetting. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 34, pages 2272-4279, 2020.
[14] Renjie Xie, Wei Xu, Jiabao Yu, Aiqun Hu, Derrick Wing Kwan Ng, and A Lee Swindle- hurst. Disentangled representation learning for rf fingerprint extraction under unknown chan- nel statistics. arXiv preprint arXiv:2208.08 027224 2022.
[15] Pengyu Cheng, Weituo Hao, and Lawrence Carin. Estimating total correlation with mutual information bounds. arXiv preprint arXiv:201 1.04994 2020.
[16] Ian Goodfellow, Jean Pouget-Abadee Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[17] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
11

---

[18] Congzheng Song and Vitaly Shmatikov. Overleaming reveals sensitive attributes. arXiv preprint arXiv: arXiv: 1990.1114422. 2019.
[19] Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invari- ant representations without adversarial training. Advances in Neural Information Processing Systems, 31, 2018.
[20] Rui Feng, Yang Yang, Yuehan Lyu, Chenhao Tan, Yizhou Sun, and Chumping Wang. Learning fair representations via an adversarial framework. arXiv preprint vrXiv:/904. 13341, 2019.
[21] Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In The 22nd International Conference on Artificial Intelli- gence and Statistics, pages 2164-2173, PMLR, 2019.
[22] Mislav Balunovic, Anian Ruoss, and Martin Vechev. Fair normalizing flows. In International Conference on Learning Representations, 2021.
[23] Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable measure of statistical dependence. Advances in Neural Information Processing Systems, 34, 2021.
[24] Jérémie Mary, Clément Calauzenes, and Noureddine El Karoui. Fairness-aware learning for continuous attributes and treatments. In International Conference Machine Learning, pages 4382-4391. PMLR, 2019.
[25] Sina Baharlouei, Maher Nouiehed, Ahmad Beirami, and Meisam Razaviyayn. R) 'l'enii fair inference. arXiv preprint arXiv: 12005, 2019.
[26] Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. Wasserstein fair classification. In Uncertainty in Artificial Intelligence, pages 862-872. PMLR, 2020.
[27] Ricky TQ Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[28] Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair represen- tations. In International conference machine learning, pages 325-333. PMLR, 2013.
[29] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain addersaiaaa training of neural networks. The journal of machine learning research, 17(1):2096-200 2016.
[30] Alfréd Rényi On measures of dependence. Acta mathematica hungarica, 10(3-4)444-451,, 1959.
[31] Hans Gebelein. Das statistische problem der korrelation als variations-und cigenwertproblem und sein zusammenhang mit der ausgleichsrechnung. ZAMM- of Applied Mathematics and Mechanics/Leitschrff für Angewandte Mathematik und Mechanik, 21 (16))3644379) 1941.
[32] Hermann O Hirschfeld. A connection between correlation and contingency. In Mathematical Proceedings of the Cambridge Philosophical Society, volume 31, pages 520-524. Cambridge University Press, 1935.
[33] David Lopez-Paz, Philipp Hennig, and Bernhard Schölkopf. The randomized dependence coefficient. Advances in neural information processing systems, 26, 2013.
[34] Harold Hotelling. Relations between two sets of variates. Biometrika, 28(3/4):321-377, 1936.
[35] Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. (TensorFlow): system for (Large-Scale) machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), pages 265-283 2016.
12

---

[36] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[37] Ziv Goldfeld, Kristjan Greenewald, Theshani Nuradha, and Galen Reeves. k-sliced mu- tual information: quantitative study of scalability with dimension. arXiv preprint arXiv:2206.0883 2022.
[38] Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR, 2018.
[39] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildnn, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.0661 2018.
[40] Aaron Van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv e-prints, pages -xii-1887, 2018.
[41] Yanzhi Chen, Dinghuai Zhang. Michael Gutmann, Aaron Courville, and Zhanxing Zhu. Neural approximate sufficient statistics for implicit models. arXiv preprint vrXiv:2010.10077 2020.
[42] Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pages 5171- 5180. PMLR, 2019.
[43] Arthur Gretton, Kenji Fukumizu, Choon Teo, Le Song, Bernhard Schölkopf, and Alex Smola. kernel statistical test of independence. Advances in neural information processing systems, 20, 2007.
[44] Francis R Bach and Michael Jordan. Kernel independent component analysis. Journal of machine learning research, 3(Jl)):1-48, 2002.
[45] Barnabás Póczos, Zoubin Ghahramani, and Jeff' Schneider. Copula-based kernel dependency measures. arXiv preprini arXiv: 1206.4620 2012.
[46] Gábor J Székely and Maria L Rizzo. Partial distance correlation with methods for dissimilari- ties. The Annals of Statistics, 2286:2233 2014.
[47] Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International conference on machine learning, pages 1779-1788. PMLR, 2020.
[48] Boxin Wang, Shuohang Wang, Yu Cheng, Zhe Gan, Ruoxi Jía, Bo Li, and Jingjing Liu. In- fobert:
Improving robustness of language models from an information theoretic perspective. arXiv preprint arXiv:2010.022329 2020.
[49] Kimia Nadjahi, Alain Durmus, Lénaic Chizat, Soheil Kolouri, Shahin Shahrampour, and Umut Simsekli. Statistical and topological properties of sliced probability divergences. Advances in Neural Information Processing Systems, 33:20802-20814 2020.
[50] Ishan Deshpande, Ziyu Zhang, and Alexander G Schwing. Generative modeling using the sliced wasserstein distance. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3483-3491, 2018
[51] Soheil Kolouri, Phillip E Pope, Charles E Martin, and Gustavo K Rohde. Sliced wasserstein auto-encoders. In International Conference on Learning Representations, 2018
[52] Mark Rowland, Jiri Hron, Yunhao Tang, Krzysztof Choromanski, Tamas Sarlos, and Adrian Weller. Orthogonal estimation of wasserstein distances. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 186-195. PMLR, 2019.
13

---

[53] Yang Song, Sahaj Garg, Jiaxin Shi, and Stefano Ermon. Sliced score matching: A scalable approach to density and score estimation. In Uncertainty in Artificial Intelligence, pages 574- 584. PMLR, 2020.
[54] Wenbo Gong, Yingzhen Li, and José Miguel Hemández-Lobato. Sliced kernelized stein dis- crepancy, arXiv preprint arXiv:2006. 16531, 2020.
[55] Yilun Xu, Hao He, Tianxiao Shen, and Tommi S Jaakkola. Controlling directions orthogonal to classifier. In International Conference on Learning Representations, 2021.
[56] Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, Luca Oneto, and Massimiliano Pon-
til. Fair regression with wasserstein barycenters. Advances in Neural Information Processing Systems, 33:7321-7331, 2020.
[57] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv: 1.0088,, 2017.
[58] Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in f-vae. arXiv preprint arXiv: 1045:9900.0 2018.
[59] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Mathey, Danilo Rezende, and Alexander Lerchner. Towards a definition of disentangled representations. arXiv preprint arXiv: 2018.
[60] Robin Quessard, Thomas Barrett, and William Clements. Learning disentangled representa- tions and group structure of dynamical environments. Advances in Neural Information Pro- cessing Systems, 33:19777-19737, 2020.
[61] Xinqi Zhu, Chang Xu, and Dacheng Tao. Commutative lie group vae for disentanglement learning. In International Conference on Machine Learning, pages 12924-12934, PMLR, 2021.
[62] Elliot Creager, David Madras, Jöm-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In Inter- national conference on machine learning, pages 1436-1445. PMLR, 2019.
[63] Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual infor- mation estimators. In International Conference on Learning Representations, 2019.
[64] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
[65] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset.
-attssss 2017.
[66] Ralph Gross, lain Matthews, Jeffrey Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Image and vision computing, 287788877 2010.
[67] Yu Tian, Xi Peng, Long Zhao, Shaoting Zhang, and Dimitris N Metaxas. Cr-gan: learning complete representations for multi-view generation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 942-948, 2018
[68] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 770-778, 2016.
[69] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, (11), 2008
[70] Geoff French, Michal Mackiewiez, and Mark Fisher. Self-ensembling for visual domain adap- tation. In International Conference on Learning Representations, 2018.
[71] Rui Shu, Hung Bui, Hirokazu Narui, and Stefano Ermon. dirt-t approach to unsupervised domain adaptation. In International Conference on Learning Representatinss, 2018.
14


Reviews:
[{"review_id": "MzPMjm83zUp", "reviewer": "Reviewer_hqSG", "summary": "The authors propose a method for infomin learning based on information theory tools from previous work, that estimates the Mutual Information (MI) in the “sliced space”, which intuitively is a facet of MI.", "questions": "- On the disentanglement experiment more recent work could be considered as baseline, for example LieGroupVAE (https://arxiv.org/pdf/2106.03375.pdf) ?\n- In the disentanglement experiment qualitative Figure2-3. Should latent traversals also be explored, to understand if the representation is disentangled?\n- In the fairness experiment ρ∗(Z, T ) seems to get non-comparable results for the baselines. Should other better performing methods be considered as baselines?\n", "limitations": "One of the claims is the scalability, however a dedicated experiment demonstrating how the model scales compared to previous baselines is missing.\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The method seems explained well in terms of theoretical justifications\n- The experiments seem to be provided with sufficient details\n\nWeaknesses\n- In the disentanglement experiment other metrics could be used to make sure that models are learning disentangled representations, e.g. MIG, factorVAE score, etc\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3dWgIzjhqtE", "reviewer": "Reviewer_dNaL", "summary": "The authors propose an adversary-free method for informin-based representation-learning based on the recently-proposed sliced mutual information estimator, noting the optimisation difficulties and costs associated with the adversarial approaches that have long been the go-to methods for tasks such as disentanglement and fair-representation learning. This is achieved by substituting the empirical approximation of sliced mutual  information (SI) with a closed-form solution that can be well-approximated with K-degree polynomials, leading to a problem that can be efficiently solved using canonical correlation analysis. The authors establish error bounds on said analytic approximation and show that the entailed functions can be solved in parallel for each slice without violating the objective.\nExperiments are performed on a number of datasets spanning the interrelated tasks of independence  testing, disentanglement, and fair-representation learning, with Renyi correlation between the learned representation and Y (aiming to maximise) and T (aiming to minimise) as the evaluation metric. The proposed method, Slice, is shown to  outperform other infomin-learning approaches by a significant\nmargin while remaining computationally efficient.\n", "questions": "-", "limitations": "I am satisfied with the extent to which the the limitations and ethics of the work have been addressed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper builds on recent research on mutual-information estimation to develop a method that is both adversarial free\nand computationally-efficient, and how it might be practically applied to problems of disentanglement and fair-representation\nlearning -- to my knowledge, the analytic approximation, derived from canonical correlation analysis, is novel within the\ngiven context, and seems to be both theoretically and practically sound.\n- The motivation for the seeking out non-adversarial alternatives to infomin learning is clearly established and the description of the method is generally well-structured and easy-to-follow, though I feel there is some room for improvement in the choice of notation (such as in the decision to use D and d for the dimensionality of Z and T, respectively).\n- The analysis of the results is satisfactory; the conclusion is serviceable but feels rushed and I'm not convinced there's sufficient evidence given in the paper to 'verify' the failure modes of adversarial methods.\n- The method is benchmarked against a good range of relevant baselines and using an appropriate suite of datasets\n(though it would be nice to see variation in the domain of the fairness datasets) and results are aggregated over multiple\nreplicates in all cases. The procedures associated with each set of experiments are outlined in sufficient detail.\n- While the existing baselines are solid, for the fair-representation learning experiments it would be nice to see a simpler \nadversarial cross-entropy-based baseline a la DANN/LAFTR, given its prominence in the literature, despite its inability\nto capture higher order dependencies like its Renyi counterpart.\n- Given the statements made about the convergence of the adversarial methods, I feel it would also be of interest to include ablations in which the number of adversarial steps is increased enough to determine whether this is a relevant factor, or whether the problems with these methods truly are of a more fundamental nature.\n- The method, Slice, seems to perform impressively in terms of both expectation and variance. While Slice does incur decent computational overhead compared the simpler baselines, such as 'Pearson', it is nonetheless shown to be significantly more efficient than its adversarial competitors which suffer additional problems due to their parametric/minimax nature. The results included in the main paper, together with those included in the Appendices, support these conclusions convincingly enough, with the experimental setups used to produce them largely consistent with that of past work.\n- While the authors do discuss the relationship between Renyi correlation and DP, and good rationale for adopting solely the former, it would nonetheless be good to additionally include the the latter for the evaluation of the method in the context of representation learning, given the conventions of the pre-existing literature on the problem.\n- The images in Figures 2 and 3 comparing how Slice and an adversarial method (which one?) cope with label-swapping are rather small and there's no indication as to what the label corresponds to within the figures themselves or their captions. Those things aside, I think it would be helpful if the differences between the methods were highlighted as it's not immediately obvious where exactly one should be looking.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0yCXVwNjE9Y", "reviewer": "Reviewer_d477", "summary": "In this paper, the authors leverage sliced mutual information and propose a tractable infomin learning algorithm. Compared to the existing methods, the proposed algorithm requires neither adversarial training nor a neural-based mutual information estimator. Instead, the authors replace the mutual information term with a sliced version in the infomin objective, and derive an analytic approximation to sliced mutual information. Moreover, solving hi and gj in the sliced mutual information together yields an upper bound of the analytic solution, and thus more efficient in practice. The authors compare the proposed method with 4 different baselines, including the non-parametric method and adversarial training method. They evaluate them on three tasks: in the independence test, they show that sliced mutual information is less effective than the adversarial training methods, but they require longer training time; in the fairness experiment, under a similar level of utility, the proposed method has significantly better fairness than the baselines; in the disentangled representation learning experiment, they show a similar observation to the fairness experiment that the proposed method achieves much better disentanglement without sacrificing the utility a lot.", "questions": "Please answer the questions in the weaknesses section. \n\nBoth your method and MI upper bound [1] utilize neural networks for MI estimation, could the authors elaborate more on the advantages of your method?\n\nI am willing to increase my scores if my questions and concerns are resolved.\n", "limitations": "See the weakness section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The paper is overall well written. The theoretical analysis is sound and well motivated.\n- The idea of using proxy metrics instead of directly working on the intractable MI is interesting\n- The experimental setup is sound with different baselines\n\n**Weaknesses**:\n- I think the paper misses one important and related paper [1], and thus lacks further discussion of one important implementation of infomin optimization that leverages neural-based methods to give a trackable estimation of the upper bound of mutual information.   This should serve as a strong baseline and compare with the proposed method, non-parametric methods, and adversarial training methods as well. I am expecting to see more quantitative analysis on the comparisons. \n- I am also looking forward to seeing more larger-scale experiments, as the paper is claimed to be “scalable”. For example, the CLUB MI estimator [1] is known to be applied to large language models like BERT to improve the adversarial robustness [2]. \n- The paper misses some important implementation details: (1) a brief introduction and citation to the datasets Dsprite and CMU-pie are missing. (2) how the networks h,g are trained needs more details, e,g, the training objective. (3) I am also looking forward to seeing a detailed pseudo algorithm in terms of the details mini-batch learning to improve the reproducibility.\n- I am also expecting to see how is the sliced MI related to the real MI. It might be better to give some toy examples where MI can be calculated and see how close the methods are.\n\n[1] Cheng, Pengyu, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan and Lawrence Carin. “CLUB: A Contrastive Log-ratio Upper Bound of Mutual Information.” ICML (2020).\n\n[2] Wang, Boxin, Shuohang Wang, Yu Cheng, Zhe Gan, R. Jia, Bo Li and Jingjing Liu. “InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective.” ICLR (2021)\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept