Controlling Directions Orthogonal to a Classifier

1 INTRODUCTION . Many machine learning applications require explicit control of directions that are orthogonal to a predefined one . For example , to ensure fairness , we can learn a classifier that is orthogonal to sensitive attributes such as gender or race ( Zemel et al. , 2013 ; Madras et al. , 2018 ) . Similar , if we transfer images from one style to another , content other than style should remain untouched . Therefore images before and after transfer should align in directions orthogonal to style . Common to these problems is the task of finding an orthogonal classifier . Given any principal classifier operating on the basis of principal variables , our goal is to find a classifier , termed orthogonal classifier , that predicts the label on the basis of orthogonal variables , defined formally later . The notion of orthogonality is clear in the linear case . Consider a joint distribution PXY over X ∈ Rd and binary label Y . Suppose the label distribution is Bernoulli , i.e. , PY = B ( Y ; 0.5 ) and class-conditional distributions are Gaussian , PX ∣Y =y = N ( X ; µy , σ2yI ) , where the means and variances depend on the label . If the principal classifier is linear , w1 = Pr ( Y = 1∣θ⊺1x ) , any classifier w2 , in the setW2 = { Pr ( Y = 1∣θ⊺2x ) ∣ θ⊺1θ2 = 0 } , is considered orthogonal to w1 . Thus the two classifiers w1 , w2 , with orthogonal decision boundaries ( Fig . 1 ) focus on distinct but complementary attributes for predicting the same label . Finding the orthogonal classifier is no longer straightforward in the non-linear case . To rigorously define what we mean by the orthogonal classifier , we first introduce the notion of mutually orthogonal random variables that correspond to ( conditinally ) independent latent variables mapped to observations through a diffeomorphism ( or bijection if discrete ) . Each r.v . is predictive of the label but represents complementary information . Indeed , we show that the orthogonal random variable maximizes the conditional mutual information with the label given the principal counterpart , subject to an independence constraint that ensures complementarity . Our search for the orthogonal classifier can be framed as follows : given a principal classifier w1 using some unknown principal r.v . for prediction , how do we find its orthogonal classifier w2 relying solely on orthogonal random variables ? The solution to this problem , which we call classifier orthogonalization , turns out to be surprisingly simple . In addition to the principal classifier , we assume access to a full classifier wx that predicts the same label based on all the available information , implicitly relying on both principal and orthogonal latent variables . The full classifier can be trained normally , absent of constraints1 . We can then effectively “ subtract ” the contribution of w1 from the full classifier to obtain the orthogonal classifier w2 which we denote as w2 = wx ∖w1 . The advantage of this construction is that 1The full classifier may fail to depend on all the features , e.g. , due to simplicity bias ( Shah et al. , 2020 ) . we do not need to explicitly identify the underlying orthogonal variables . It suffices to operate only on the level of classifier predictions . We provide several use cases for the orthogonal classifier , either as a predictor or as a discriminator . As a predictor , the orthogonal classifier predictions are invariant to the principal sensitive r.v. , thus ensuring fairness . As a discriminator , the orthogonal classifier enforces a partial alignment of distributions , allowing changes in the principal direction . We demonstrate the value of such discriminators in 1 ) controlled style transfer where the source and target domains differ in multiple aspects , but we only wish to align domain A ’ s style to domain B , leaving other aspects intact ; 2 ) domain adaptation with label shift where we align feature distributions between the source and target domains , allowing shifts in label proportions . Our results show that the simple method is on par with the state-of-the-art methods in each task . 2 NOTATIONS AND DEFINITION . Symbols . We use the uppercase to denote random variable ( e.g. , data X , label Y ) , the lowercase to denote the corresponding samples and the calligraphic letter to denote the sample spaces of r.v. , e.g. , data sample space X . We focus on the setting where label space Y is discrete , i.e. , Y = { 1 , ⋯ , C } , and denote the C − 1 dimensional probability simplex as ∆C . A classifier w ∶ X →∆C is a mapping from sample space to the simplex . Its y-th dimension w ( x ) y denotes the predicted probability of label y for sample x . Distributions . For random variables A , B , we use the notation pA , pA∣B , pAB to denote the marginal/conditional/joint distribution , i.e. , pA ( a ) = p ( A = a ) , pA∣B ( a∣b ) = p ( A = a∣B = b ) , pAB ( a , b ) = p ( A = a , B = b ) . Sometimes , for simplicity , we may ignore the subscript if there is no ambiguity , e.g. , p ( a∣b ) is an abbreviation for pA∣B ( a∣b ) . We begin by defining the notion of an orthogonal random variable . We consider continuous X , Z1 , Z2 and assume their supports are manifolds diffeomorphic to the Euclidean space . The probability density functions ( PDF ) are in C1 . Given a joint distribution pXY , we define the orthogonal random variable as follows : Definition 1 ( Orthogonal random variables ) . We say Z1 and Z2 are orthogonal random variables w.r.t pXY if they satisfy the following properties : ( i ) There exists a diffeomorphism f ∶ Z1 ×Z2 → X such that f ( Z1 , Z2 ) =X . ( ii ) Z1 and Z2 are statistically independent given Y , i.e. , Z1 ⊥ Z2∣Y . The orthogonality relation is symmetric by definition . Note that the orthogonal pair perfectly reconstructs the observations via the diffeomorphism f ; as random variables they are also sampled independently from class conditional distributions p ( Z1∣Y ) and p ( Z2∣Y ) . For example , we can regard foreground objects and background scenes in natural images as being mutually orthogonal random variables . Remark . The definition of orthogonality can be similarly developed for discrete variables and discretecontinuous mixtures . For discrete variables , for example , we can replace the requirement of diffeomorphism with bijection . Since the diffeomorphism f is invertible , we can use z1 ∶ X → Z1 and z2 ∶ X → Z2 to denote the two parts of the inverse mapping so that Z1 = z1 ( X ) and Z2 = z2 ( X ) . Note that , for a given joint distribution pXY , the decomposition into orthogonal random variables is not unique . There are multiple pairs of random variables that represent valid mutually orthogonal latents of the data . We can further justify our definition of orthogonality from an information theoretic perspective by showing that the choice of z2 attains the maximum of the following constrained optimization problem . Proposition 1 . Suppose the orthogonal r.v . of z1 ( X ) w.r.t pXY exists and is denoted as z2 ( X ) . Then z ( X ) = z2 ( X ) is a maximizer of I ( z ( X ) ; Y ∣z1 ( X ) ) subject to I ( z ( X ) ; z1 ( X ) ∣Y ) = 0 . We defer the proof to Appendix B.1 . Proposition 1 shows that the orthogonal random variable maximizes the additional information about the label we can obtain from X while remaining conditionally independent of the principal random variable . This ensures complementary in predicting the label . 3 CONSTRUCTING THE ORTHOGONAL CLASSIFIER . Let Z1 = z1 ( X ) and Z2 = z2 ( X ) be mutually orthogonal random variables w.r.t pXY . We call Z1 the principal variable and Z2 the orthogonal variable . In this section , we describe how we can construct the Bayes optimal classifier operating on features Z2 from the Bayes optimal classifier relying on Z1 . We formally refer to the classifiers of interests as : ( 1 ) principal classifier w1 ( x ) y = p ( Y = y∣Z1 = z1 ( x ) ) ; ( 2 ) orthogonal classifier w2 ( x ) y = p ( Y = y∣Z2 = z2 ( x ) ) ; ( 3 ) full classifier wx ( x ) y = p ( Y = y∣X = x ) . 3.1 CLASSIFIER ORTHOGONALIZATION . Our key idea relies on the bijection between the density ratio and the Bayes optimal classifier ( Sugiyama et al. , 2012 ) . Specifically , the ratio of densities pX ∣Y ( x∣i ) and pX ∣Y ( x∣j ) , assigned to an arbitrary point x , can be represented by the Bayes optimal classifier w ( x ) i = Pr ( Y = i∣x ) as pX∣Y ( x∣i ) pX∣Y ( x∣j ) = pY ( j ) w ( x ) i pY ( i ) w ( x ) j . Similar , the principal classifier w1 gives us associated density ratios of class-conditional distributions over Z1 . For any i , j ∈ Y , we have pZ1 ∣Y ( z1 ( x ) ∣i ) pZ1 ∣Y ( z1 ( x ) ∣j ) = pY ( j ) w1 ( x ) i pY ( i ) w1 ( x ) j . These can be combined to obtain density ratios of class-conditional distribution pZ2∣Y and subsequently calculate the orthogonal classifier w2 . We additionally rely on the fact that the diffeomorphism f permits us to change variables between x and z1 , z2 : pX ∣Y ( x∣i ) = pZ1∣Y ( z1∣i ) ∗ pZ2∣Y ( z2∣i ) ∗ volJf ( z1 , z2 ) , where volJf is volume of the Jacobian ( Berger et al. , 1987 ) of the diffeomorphism mapping . Taken together , pZ2∣Y ( z2∣i ) pZ2∣Y ( z2∣j ) = pZ1∣Y ( z1∣i ) ∗ pZ2∣Y ( z2∣i ) ∗ volJf ( z1 , z2 ) pZ1∣Y ( z1∣j ) ∗ pZ2∣Y ( z2∣j ) ∗ volJf ( z1 , z2 ) / pZ1∣Y ( z1∣i ) pZ1∣Y ( z1∣j ) = wx ( x ) i wx ( x ) j / w1 ( x ) i w1 ( x ) j ( 1 ) Note that since the diffeomorphism f is shared with all classes , the Jacobian is the same for all labelconditioned distributions on Z1 , Z2 . Hence the Jacobian volume terms cancel each other in the above equation . We can then finally work backwards from the density ratios of pZ2∣Y to the orthogonal classifier : Pr ( Y = i∣Z2 = z2 ( x ) ) = Pr ( Y = i ) wx ( x ) i w1 ( x ) i /∑ j ( Pr ( Y = j ) wx ( x ) j w1 ( x ) j ) ( 2 ) We call this procedure classifier orthogonalization since it adjusts the full classifier wx to be orthogonal to the principal classifier w1 . The validity of this procedure requires overlapping supports of the classconditional distributions , which ensures the classifier outputs wx ( x ) i , w1 ( x ) i to remain non-zero for all x ∈ X , i ∈ Y . Empirically , we usually have access to a dataset D = { ( xt , yt ) } nt=1 with n iid samples from the joint distribution pXY . To obtain the orthogonal classifier , we need to first train the full classifier ŵx based on the dataset D. We can then follow the classifier orthogonalization to get an empirical orthogonal classifier , denoted as w2 = ŵx ∖w1 . We use symbol ∖ to emphasize that the orthogonal classifier uses complementary information relative to z1 . Algorithm 1 summarizes the construction of the orthogonal classifier . Generalization bound . Since wx is trained on a finite dataset , we consider the generalization bound of the constructed orthogonal classifier . We denote the population risk as R ( w ) = −EpXY ( x , y ) [ logw ( x ) y ] and the empirical risk as R̂ ( w ) = − 1∣D∣ ∑ ( xi , yi ) ∈D logw ( xi ) yi . For a function familyW whose elements map X to the simplex ∆C , we define ŵx = infwx∈W R̂ ( wx ) , w∗x = infwx∈W R ( wx ) . We further denote the Rademacher complexity of function familyW with sample size ∣D∣ as R∣D∣ ( W ) . Algorithm 1 Classifier Orthogonalization Input : principal classifier w1 , dataset D. Train an empirical full classifier ŵx on D by empirical risk minimization . Construct an orthogonal classifier ŵx ∖w1 via classifier orthogonalization ( Eq . ( 2 ) ) . return the empirical orthogonal classifier ŵ2 = ŵx ∖w1 Theorem 1 . Assume py is uniform distribution , ∀wx ∈W takes values in ( m,1 −m ) C with m ∈ ( 0 , 12 ) , and 1/pX ∣Y ( x∣y ) ∈ ( 0 , γ ) ⊂ ( 0 , +∞ ) holds for ∀x ∈ X , y ∈ Y . Then for any δ ∈ ( 0,1 ) with probability at least 1 − δ , we have : ∣R ( ŵx ∖w1 ) −R ( w∗x ∖w1 ) ∣ ≤ ( 1 + γ ) ⎛ ⎜ ⎝ 2R∣D∣ ( W ) + 2 log 1 m ¿ ÁÁÀ2 log 1 δ ∣D∣ ⎞ ⎟ ⎠ Theorem 1 shows that the population risk of the empirical orthogonal classifier in Algorithm 1 would be close to the optimal risk if the maximum value of the reciprocal of class-conditioned distributions 1/pX ∣Y ( x∣y ) and the Rademacher term are small . Typically , the Rademacher complexity term satisfies R∣D∣ ( W ) = O ( ∣D∣− 1 2 ) ( Bartlett & Mendelson , 2001 ; Gao & Zhou , 2016 ) . We note that the empirical full classifier may fail in specific ways that are harmful for our purposes . For example , the classifier may not rely on all the key features due to simplicity bias as demonstrated by ( Shah et al. , 2020 ) . There are several ways that this effect can be mitigated , including Ross et al . ( 2020 ) ; Teney et al . ( 2021 ) .

Summary: The paper presents classifier orthogonalization technique that works for non-linear classifiers. The idea is to find a method that orthogonalizes the full classifier w.r.t. the principal classifier so that the resulting classifier is statistically orthogonal to principal classifier. The algorithm turns out to be very simple, only requiring access to the full classifier $P(Y|x)$ and the principal classifier $P(Y|z(x))$, where $z$ is a control variable you want to orthogonalize against. The paper also discusses an alternative method of classifier orthogonalization using importance sampling.   The effectiveness of the proposed classifier orthogonalization technique has been demonstrated through 3 applications: controlled style transfer, domain adaptation, and classifier fairness. For controlled style transfer, authors modified the CycleGAN's generator update step by orthogonalizing discriminator w.r.t. the controlled style variable. For domain adaptation, they modified VADA, an adversarial domain adaptation approach, by orthogonalizing discriminator based on the label-based principal classifier, so that discriminator wouldn't discriminate the domain based on the frequency count of labels from each domain. For fairness, full classifier is orthogonalized by the sensitive attribute classifier.