Controlling Directions Orthogonal to a Classifier

1 INTRODUCTION . Many machine learning applications require explicit control of directions that are orthogonal to a predefined one . For example , to ensure fairness , we can learn a classifier that is orthogonal to sensitive attributes such as gender or race ( Zemel et al. , 2013 ; Madras et al. , 2018 ) . Similar , if we transfer images from one style to another , content other than style should remain untouched . Therefore images before and after transfer should align in directions orthogonal to style . Common to these problems is the task of finding an orthogonal classifier . Given any principal classifier operating on the basis of principal variables , our goal is to find a classifier , termed orthogonal classifier , that predicts the label on the basis of orthogonal variables , defined formally later . The notion of orthogonality is clear in the linear case . Consider a joint distribution PXY over X ∈ Rd and binary label Y . Suppose the label distribution is Bernoulli , i.e. , PY = B ( Y ; 0.5 ) and class-conditional distributions are Gaussian , PX ∣Y =y = N ( X ; µy , σ2yI ) , where the means and variances depend on the label . If the principal classifier is linear , w1 = Pr ( Y = 1∣θ⊺1x ) , any classifier w2 , in the setW2 = { Pr ( Y = 1∣θ⊺2x ) ∣ θ⊺1θ2 = 0 } , is considered orthogonal to w1 . Thus the two classifiers w1 , w2 , with orthogonal decision boundaries ( Fig . 1 ) focus on distinct but complementary attributes for predicting the same label . Finding the orthogonal classifier is no longer straightforward in the non-linear case . To rigorously define what we mean by the orthogonal classifier , we first introduce the notion of mutually orthogonal random variables that correspond to ( conditinally ) independent latent variables mapped to observations through a diffeomorphism ( or bijection if discrete ) . Each r.v . is predictive of the label but represents complementary information . Indeed , we show that the orthogonal random variable maximizes the conditional mutual information with the label given the principal counterpart , subject to an independence constraint that ensures complementarity . Our search for the orthogonal classifier can be framed as follows : given a principal classifier w1 using some unknown principal r.v . for prediction , how do we find its orthogonal classifier w2 relying solely on orthogonal random variables ? The solution to this problem , which we call classifier orthogonalization , turns out to be surprisingly simple . In addition to the principal classifier , we assume access to a full classifier wx that predicts the same label based on all the available information , implicitly relying on both principal and orthogonal latent variables . The full classifier can be trained normally , absent of constraints1 . We can then effectively “ subtract ” the contribution of w1 from the full classifier to obtain the orthogonal classifier w2 which we denote as w2 = wx ∖w1 . The advantage of this construction is that 1The full classifier may fail to depend on all the features , e.g. , due to simplicity bias ( Shah et al. , 2020 ) . we do not need to explicitly identify the underlying orthogonal variables . It suffices to operate only on the level of classifier predictions . We provide several use cases for the orthogonal classifier , either as a predictor or as a discriminator . As a predictor , the orthogonal classifier predictions are invariant to the principal sensitive r.v. , thus ensuring fairness . As a discriminator , the orthogonal classifier enforces a partial alignment of distributions , allowing changes in the principal direction . We demonstrate the value of such discriminators in 1 ) controlled style transfer where the source and target domains differ in multiple aspects , but we only wish to align domain A ’ s style to domain B , leaving other aspects intact ; 2 ) domain adaptation with label shift where we align feature distributions between the source and target domains , allowing shifts in label proportions . Our results show that the simple method is on par with the state-of-the-art methods in each task . 2 NOTATIONS AND DEFINITION . Symbols . We use the uppercase to denote random variable ( e.g. , data X , label Y ) , the lowercase to denote the corresponding samples and the calligraphic letter to denote the sample spaces of r.v. , e.g. , data sample space X . We focus on the setting where label space Y is discrete , i.e. , Y = { 1 , ⋯ , C } , and denote the C − 1 dimensional probability simplex as ∆C . A classifier w ∶ X →∆C is a mapping from sample space to the simplex . Its y-th dimension w ( x ) y denotes the predicted probability of label y for sample x . Distributions . For random variables A , B , we use the notation pA , pA∣B , pAB to denote the marginal/conditional/joint distribution , i.e. , pA ( a ) = p ( A = a ) , pA∣B ( a∣b ) = p ( A = a∣B = b ) , pAB ( a , b ) = p ( A = a , B = b ) . Sometimes , for simplicity , we may ignore the subscript if there is no ambiguity , e.g. , p ( a∣b ) is an abbreviation for pA∣B ( a∣b ) . We begin by defining the notion of an orthogonal random variable . We consider continuous X , Z1 , Z2 and assume their supports are manifolds diffeomorphic to the Euclidean space . The probability density functions ( PDF ) are in C1 . Given a joint distribution pXY , we define the orthogonal random variable as follows : Definition 1 ( Orthogonal random variables ) . We say Z1 and Z2 are orthogonal random variables w.r.t pXY if they satisfy the following properties : ( i ) There exists a diffeomorphism f ∶ Z1 ×Z2 → X such that f ( Z1 , Z2 ) =X . ( ii ) Z1 and Z2 are statistically independent given Y , i.e. , Z1 ⊥ Z2∣Y . The orthogonality relation is symmetric by definition . Note that the orthogonal pair perfectly reconstructs the observations via the diffeomorphism f ; as random variables they are also sampled independently from class conditional distributions p ( Z1∣Y ) and p ( Z2∣Y ) . For example , we can regard foreground objects and background scenes in natural images as being mutually orthogonal random variables . Remark . The definition of orthogonality can be similarly developed for discrete variables and discretecontinuous mixtures . For discrete variables , for example , we can replace the requirement of diffeomorphism with bijection . Since the diffeomorphism f is invertible , we can use z1 ∶ X → Z1 and z2 ∶ X → Z2 to denote the two parts of the inverse mapping so that Z1 = z1 ( X ) and Z2 = z2 ( X ) . Note that , for a given joint distribution pXY , the decomposition into orthogonal random variables is not unique . There are multiple pairs of random variables that represent valid mutually orthogonal latents of the data . We can further justify our definition of orthogonality from an information theoretic perspective by showing that the choice of z2 attains the maximum of the following constrained optimization problem . Proposition 1 . Suppose the orthogonal r.v . of z1 ( X ) w.r.t pXY exists and is denoted as z2 ( X ) . Then z ( X ) = z2 ( X ) is a maximizer of I ( z ( X ) ; Y ∣z1 ( X ) ) subject to I ( z ( X ) ; z1 ( X ) ∣Y ) = 0 . We defer the proof to Appendix B.1 . Proposition 1 shows that the orthogonal random variable maximizes the additional information about the label we can obtain from X while remaining conditionally independent of the principal random variable . This ensures complementary in predicting the label . 3 CONSTRUCTING THE ORTHOGONAL CLASSIFIER . Let Z1 = z1 ( X ) and Z2 = z2 ( X ) be mutually orthogonal random variables w.r.t pXY . We call Z1 the principal variable and Z2 the orthogonal variable . In this section , we describe how we can construct the Bayes optimal classifier operating on features Z2 from the Bayes optimal classifier relying on Z1 . We formally refer to the classifiers of interests as : ( 1 ) principal classifier w1 ( x ) y = p ( Y = y∣Z1 = z1 ( x ) ) ; ( 2 ) orthogonal classifier w2 ( x ) y = p ( Y = y∣Z2 = z2 ( x ) ) ; ( 3 ) full classifier wx ( x ) y = p ( Y = y∣X = x ) . 3.1 CLASSIFIER ORTHOGONALIZATION . Our key idea relies on the bijection between the density ratio and the Bayes optimal classifier ( Sugiyama et al. , 2012 ) . Specifically , the ratio of densities pX ∣Y ( x∣i ) and pX ∣Y ( x∣j ) , assigned to an arbitrary point x , can be represented by the Bayes optimal classifier w ( x ) i = Pr ( Y = i∣x ) as pX∣Y ( x∣i ) pX∣Y ( x∣j ) = pY ( j ) w ( x ) i pY ( i ) w ( x ) j . Similar , the principal classifier w1 gives us associated density ratios of class-conditional distributions over Z1 . For any i , j ∈ Y , we have pZ1 ∣Y ( z1 ( x ) ∣i ) pZ1 ∣Y ( z1 ( x ) ∣j ) = pY ( j ) w1 ( x ) i pY ( i ) w1 ( x ) j . These can be combined to obtain density ratios of class-conditional distribution pZ2∣Y and subsequently calculate the orthogonal classifier w2 . We additionally rely on the fact that the diffeomorphism f permits us to change variables between x and z1 , z2 : pX ∣Y ( x∣i ) = pZ1∣Y ( z1∣i ) ∗ pZ2∣Y ( z2∣i ) ∗ volJf ( z1 , z2 ) , where volJf is volume of the Jacobian ( Berger et al. , 1987 ) of the diffeomorphism mapping . Taken together , pZ2∣Y ( z2∣i ) pZ2∣Y ( z2∣j ) = pZ1∣Y ( z1∣i ) ∗ pZ2∣Y ( z2∣i ) ∗ volJf ( z1 , z2 ) pZ1∣Y ( z1∣j ) ∗ pZ2∣Y ( z2∣j ) ∗ volJf ( z1 , z2 ) / pZ1∣Y ( z1∣i ) pZ1∣Y ( z1∣j ) = wx ( x ) i wx ( x ) j / w1 ( x ) i w1 ( x ) j ( 1 ) Note that since the diffeomorphism f is shared with all classes , the Jacobian is the same for all labelconditioned distributions on Z1 , Z2 . Hence the Jacobian volume terms cancel each other in the above equation . We can then finally work backwards from the density ratios of pZ2∣Y to the orthogonal classifier : Pr ( Y = i∣Z2 = z2 ( x ) ) = Pr ( Y = i ) wx ( x ) i w1 ( x ) i /∑ j ( Pr ( Y = j ) wx ( x ) j w1 ( x ) j ) ( 2 ) We call this procedure classifier orthogonalization since it adjusts the full classifier wx to be orthogonal to the principal classifier w1 . The validity of this procedure requires overlapping supports of the classconditional distributions , which ensures the classifier outputs wx ( x ) i , w1 ( x ) i to remain non-zero for all x ∈ X , i ∈ Y . Empirically , we usually have access to a dataset D = { ( xt , yt ) } nt=1 with n iid samples from the joint distribution pXY . To obtain the orthogonal classifier , we need to first train the full classifier ŵx based on the dataset D. We can then follow the classifier orthogonalization to get an empirical orthogonal classifier , denoted as w2 = ŵx ∖w1 . We use symbol ∖ to emphasize that the orthogonal classifier uses complementary information relative to z1 . Algorithm 1 summarizes the construction of the orthogonal classifier . Generalization bound . Since wx is trained on a finite dataset , we consider the generalization bound of the constructed orthogonal classifier . We denote the population risk as R ( w ) = −EpXY ( x , y ) [ logw ( x ) y ] and the empirical risk as R̂ ( w ) = − 1∣D∣ ∑ ( xi , yi ) ∈D logw ( xi ) yi . For a function familyW whose elements map X to the simplex ∆C , we define ŵx = infwx∈W R̂ ( wx ) , w∗x = infwx∈W R ( wx ) . We further denote the Rademacher complexity of function familyW with sample size ∣D∣ as R∣D∣ ( W ) . Algorithm 1 Classifier Orthogonalization Input : principal classifier w1 , dataset D. Train an empirical full classifier ŵx on D by empirical risk minimization . Construct an orthogonal classifier ŵx ∖w1 via classifier orthogonalization ( Eq . ( 2 ) ) . return the empirical orthogonal classifier ŵ2 = ŵx ∖w1 Theorem 1 . Assume py is uniform distribution , ∀wx ∈W takes values in ( m,1 −m ) C with m ∈ ( 0 , 12 ) , and 1/pX ∣Y ( x∣y ) ∈ ( 0 , γ ) ⊂ ( 0 , +∞ ) holds for ∀x ∈ X , y ∈ Y . Then for any δ ∈ ( 0,1 ) with probability at least 1 − δ , we have : ∣R ( ŵx ∖w1 ) −R ( w∗x ∖w1 ) ∣ ≤ ( 1 + γ ) ⎛ ⎜ ⎝ 2R∣D∣ ( W ) + 2 log 1 m ¿ ÁÁÀ2 log 1 δ ∣D∣ ⎞ ⎟ ⎠ Theorem 1 shows that the population risk of the empirical orthogonal classifier in Algorithm 1 would be close to the optimal risk if the maximum value of the reciprocal of class-conditioned distributions 1/pX ∣Y ( x∣y ) and the Rademacher term are small . Typically , the Rademacher complexity term satisfies R∣D∣ ( W ) = O ( ∣D∣− 1 2 ) ( Bartlett & Mendelson , 2001 ; Gao & Zhou , 2016 ) . We note that the empirical full classifier may fail in specific ways that are harmful for our purposes . For example , the classifier may not rely on all the key features due to simplicity bias as demonstrated by ( Shah et al. , 2020 ) . There are several ways that this effect can be mitigated , including Ross et al . ( 2020 ) ; Teney et al . ( 2021 ) .

Summary: - The paper introduces the notion of "orthogonal classifiers": classifiers that rely on orthogonal variables. It starts with the simple linear case, and adds a definition that also applies to the non-linear case. - The paper proposes two methods to identify a classifier orthogonal to a given one. - It then describes 3 use cases: style transfer, domain adaptation, and fairness.