Fairness-Aware Classiﬁer
with Prejudice Remover Regularizer
Toshihiro Kamishima1 , Shotaro Akaho1 , Hideki Asoh1 , and Jun Sakuma2
1
National Institute of Advanced Industrial Science and Technology (AIST),AIST Tsukuba Central 2, Umezono 1-1-1, Tsukuba, Ibaraki, 305-8568 Japan
mail@kamishima.net, {s.akaho,h.asoh}@aist.go.jp
http://www.kamishima.net
2
University of Tsukuba, 1-1-1 Tennodai, Tsukuba, 305-8577 Japan;
and, Japan Science and Technology Agency, 4-1-8, Honcho, Kawaguchi, Saitama,332-0012 Japan
jun@cs.tsukuba.ac.jp

Abstract. With the spread of data mining technologies and the accumulation of social data, such technologies and data are being used for
determinations that seriously aﬀect individuals’ lives. For example, credit
scoring is frequently determined based on the records of past credit data
together with statistical prediction techniques. Needless to say, such determinations must be nondiscriminatory and fair in sensitive features,such as race, gender, religion, and so on. Several researchers have recently begun to attempt the development of analysis techniques that areaware of social fairness or discrimination. They have shown that simply
avoiding the use of sensitive features is insuﬃcient for eliminating biases
in determinations, due to the indirect inﬂuence of sensitive information.
In this paper, we ﬁrst discuss three causes of unfairness in machine learning. We then propose a regularization approach that is applicable to any
prediction algorithm with probabilistic discriminative models. We further apply this approach to logistic regression and empirically show its
eﬀectiveness and eﬃciency.
Keywords: fairness, discrimination, logistic regression, classiﬁcation,social responsibility, information theory.

1

Introduction

Data mining techniques are being increasingly used for serious determinations
such as credit, insurance rates, employment applications, and so on. For example, credit scoring is frequently determined based on the records of past credit
data together with statistical prediction techniques. Needless to say, such serious
determinations must guarantee fairness in both social and legal viewpoints; thatis, they must be unbiased and nondiscriminatory in relation to sensitive features
such as gender, religion, race, ethnicity, handicaps, political convictions, andso on. Thus, sensitive features must be carefully treated in the processes andalgorithms for data mining.
P. Flach et al. (Eds.): ECML PKDD 2012, Part II, LNCS 7524, pp. 35–50, 2012.
c Springer-Verlag Berlin Heidelberg 2012
36

T. Kamishima et al.

There are reasons other than the need to avoid discrimination for prohibiting
the use of certain kinds of features. Pariser pointed out a problem that friend
candidates recommended to him in Facebook were biased in terms of their political convictions without his permission [15]. For this problem, it would be
helpful to make recommendations that are neutral in terms of the user’s speciﬁed feature, i.e., the candidate friends’ political convictions. Further, there arefeatures that cannot legally be exploited due to various regulations or contracts.
For example, exploiting insider information and customer data are respectively
restricted by stock trading regulation and privacy policies.
Several researchers have recently begun to attempt the development of analytic techniques that are aware of social fairness or discrimination [17,3]. They
have shown that the simple elimination of sensitive features from calculations
is insuﬃcient for avoiding inappropriate determination processes, due to the indirect inﬂuence of sensitive information. For example, when determining credit
scoring, the feature of race is not used. However, if people of a speciﬁc race live
in a speciﬁc area and address is used as a feature for training a prediction model,
the trained model might make unfair determinations even though the race feature is not explicitly used. Such a phenomenon is called a red-lining eﬀect [3] orindirect discrimination [17].
In this paper, we formulate causes of unfairness in data mining, develop widely
applicable and eﬃcient techniques to enhance fairness, and evaluate the eﬀectiveness and eﬃciency of our techniques. First, we consider unfairness in termsof its causes more deeply. We describe three types of cause: prejudice, underestimation, and negative legacy. Prejudice involves a statistical dependence between
sensitive features and other information; underestimation is the state in whicha classiﬁer has not yet converged; and negative legacy refers to the problems of
unfair sampling or labeling in the training data. We also propose measures toquantify the degrees of these causes.
Second, we then focus on indirect prejudice and develop a technique to reduce
it. This technique is implemented as regularizers that restrict the learner’s behaviors. This approach can be applied to any prediction algorithm with discriminative probabilistic models, such as logistic regression. In solving classiﬁcation
problems that pay attention to sensitive information, we have to consider the
trade-oﬀ between the classiﬁcation accuracy and the degree of resultant fairness.
Our method provides a way to control this trade-oﬀ by adjusting the regularization parameter. We propose a prejudice remover regularizer, which enforces adetermination’s independence from sensitive information.
Finally, we perform experiments to test the eﬀectiveness and eﬃciency of our
methods. We evaluate the eﬀectiveness of our approach and examine the balance
between prediction accuracy and fairness. We demonstrate that our method canlearn a classiﬁcation model by taking into account the diﬀerence in inﬂuence of
diﬀerent features on sensitive information.
Note that in the previous work, a learning algorithm that is aware of social
discrimination is called discrimination-aware mining. However, we hereafter use
the terms, “unfairness” / “unfair” instead of “discrimination” / “discriminatory”

Fairness-Aware Classiﬁer with Prejudice Remover Regularizer

37

for two reasons. First, as described above, these technologies can be used forvarious purposes other than avoiding discrimination. Second, because the term
discrimination is frequently used for the meaning of classiﬁcation in the data
mining literature, using this term becomes highly confusing.
We discuss causes of unfairness in section 2 and propose our methods forenhancing fairness in section 3. Our methods are empirically compared with a2-naı̈ve-Bayes method proposed by Calders and Verwer in section 4. Section 5shows related work, and section 6 summarizes our conclusions.

2

Fairness in Data Analysis

After introducing an example of the diﬃculty in fairness-aware learning, we show
three causes of unfairness and quantitative measures.
2.1

Illustration of the Diﬃculties in Fairness-Aware Learning

We here introduce an example from the literature to show the diﬃculties infairness-aware learning [3], which is a simple analytical result for the data set
described in section 4.2. The researchers performed a classiﬁcation problem. The
sensitive feature, S, was gender, which took a value, Male or Female, and thetarget class, Y , indicated whether his/her income is High or Low. There weresome other non-sensitive features, X. The ratio of Female records comprised
about 1/3 of the data set; that is, the number of Female records was muchsmaller than that of Male records. Additionally, while about 30% of Male records
were classiﬁed into the High class, only 11% of Female records were. Therefore,
Female–High records were the minority in this data set.
In this data set, we describe how Female records tend to be classiﬁed into the
Low class unfairly. Calders and Verwer deﬁned a discrimination score (hereafterreferred to as the Calders-Verwer score (CV score) by subtracting the conditional
probability of the positive class given a sensitive value from that given a nonsensitive value. In this example, a CV score is deﬁned as
Pr[Y =High|S=Male] − Pr[Y =High|S=Female].
The CV score calculated directly from the original data is 0.19. After training
a naı̈ve Bayes classiﬁer from data involving a sensitive feature, the CV score on
the predicted classes increases to about 0.34. This shows that Female records
are more frequently misclassiﬁed to the Low class than Male records; and thus,
Female–High individuals are considered to be unfairly treated. This phenomenon
is mainly caused by an Occam’s razor principle, which is commonly adopted
in classiﬁers. Because infrequent and speciﬁc patterns tend to be discarded togeneralize observations in data, minority records can be unfairly neglected. Even
if the sensitive feature is removed from the training data for a naı̈ve Bayes classiﬁer, the resultant CV score is 0.28, which still shows an unfair treatment forminorities. This is caused by the indirect inﬂuence of sensitive features. This

38

T. Kamishima et al.

event is called by a red-lining eﬀect [3], a term that originates from the historical practice of drawing red lines on a map around neighborhoods in whichlarge numbers of minorities are known to dwell. Consequently, simply removing
sensitive features is insuﬃcient, and another techniques have to be adopted tocorrect the unfairness in data mining.
2.2

Three Causes of Unfairness

In this section, we discuss the social fairness in data analysis. Previous works[17,3] have focused on unfairness in the resultant determinations. To look more
carefully at the problem of fairness in data mining, we shall examine the underlying causes or sources of unfairness. We suppose that there are at least threepossible causes: prejudice, underestimation, and negative legacy.
Before presenting these three causes of unfairness, we must introduce several notations. Here, we discuss supervised learning, such as classiﬁcation andregression, which is aware of unfairness. Y is a target random variable to bepredicted based on the instance values of features. The sensitive variable, S, andnon-sensitive variable, X, correspond to sensitive and non-sensitive features, respectively. We further introduce a prediction model M[Y |X, S], which models
a conditional distribution of Y given X and S. With this model and a true
distribution over X and S, Pr∗ [X, S], we deﬁnePr[Y, X, S] = M[Y |X, S]Pr∗ [X, S].

(1)

Applying marginalization and/or Bayes’ rule to this equation, we can calculate
other distributions, such as Pr[Y, S] or Pr[Y |X]. We use P̃r[·] to denote sample
distributions. P̂r[Y, X, S] is deﬁned by replacing a true distribution in (1) with
its corresponding sample distribution:P̂r[Y, X, S] = M[Y |X, S]P̃r[X, S],(2)and induced distributions from P̂r[Y, X, S] are denoted by using P̂r[·].
Prejudice. Prejudice means a statistical dependence between a sensitive variable, S, and the target variable, Y , or a non-sensitive variable, X. There are three
types of prejudices: direct prejudice, indirect prejudice, and latent prejudice.
The ﬁrst type is direct prejudice, which is the use of a sensitive variable ina prediction model. If a model with a direct prejudice is used in classiﬁcation,
the classiﬁcation results clearly depend on sensitive features, thereby generating
a database containing direct discrimination [17]. To remove this type of prejudice, all that we have to do is simply eliminate the sensitive variable from theprediction model. We then show a relation between this direct prejudice andstatistical dependence. After eliminating the sensitive variable, equation (1) canbe rewritten as
Pr[Y, X, S] = M[Y |X]Pr∗ [S|X]Pr∗ [X].
This equation states that S and Y are conditionally independent given X, i.e.,Y⊥⊥ S — X. Hence, we can say that when the condition Y ⊥⊥ S — X is not satisﬁed, the prediction model has a direct prejudice.

Fairness-Aware Classiﬁer with Prejudice Remover Regularizer

39

The second type is an indirect prejudice, which is statistical dependence between a sensitive variable and a target variable. Even if a prediction model lacks
a direct prejudice, the model can have an indirect prejudice and can make an
unfair determination. We give a simple example. Consider the case that all Y ,X, and S are real scalar variables, and these variables satisfy the equations:
Y = X + εY

and

S = X + εS ,where εY and εS are mutually independent random variables. Because
Pr[Y, X, S] is equal to Pr[Y |X] Pr[S|X] Pr[X], these variables satisfy the condition Y ⊥⊥ S — X, but do not satisfy the condition Y ⊥⊥S. Hence, the adopted
prediction model does not have a direct prejudice, but may have an indirect prejudice. If the variances of εY and εS are small, Y and S become highly correlated.
In this case, even if a model does not have a direct prejudice, the determination clearly depends on sensitive information. Such resultant determinations arecalled indirect discrimination [17] or a red-lining eﬀect [3] as described in section 2.1. To remove this indirect prejudice, we must use a prediction model thatsatisﬁes the condition Y ⊥⊥S.
We next show an index to quantify the degree of indirect prejudice, whichis straightforwardly deﬁned as the mutual information between Y and S. However, because a true distribution in equation (1) is unknown, we adopt sample
distributions in equation (2) over a given sample set, D:PI =P̂r[y, s] ln(y,s)∈DP̂r[y, s]P̂r[y]P̂r[s].

(3)

We refer to this index as a (indirect) prejudice index (PI for short). For convenience, the application of the normalization technique for mutual information[21] leads to a normalized prejudice index (NPI for short):

NPI = PI/( H(Y )H(S)),(4)
where H(·) is an entropy function. PI/H(Y ) is the ratio of information of S used
for predicting Y , and PI/H(S) is the ratio of information that is exposed if avalue of Y is known. This NPI can be interpreted as the geometrical mean ofthese two ratios. The range of this NPI is [0, 1].
The third type of prejudice is latent prejudice, which is a statistical dependence between a sensitive variable, S, and a non-sensitive variable, X. Consider
an example that satisﬁes the equations:
Y = X1 + εY ,X = X1 + X2 ,andS = X2 + εS ,where εY ⊥⊥εS and X1 ⊥⊥X2 . Clearly, the conditions Y ⊥⊥ S — X and Y ⊥⊥S aresatisﬁed, but X and S are not mutually independent. This dependence doesn’t
cause a sensitive information to inﬂuence the ﬁnal determination, but it would
be exploited for training learners; thus, this might violate some regulations orlaws. Removal of latent prejudice is achieved by making X and Y independent
from S simultaneously. Similar to a PI, the degree of a latent prejudice can bequantiﬁed by the mutual information between X and S.

40

T. Kamishima et al.

Underestimation. Underestimation is the state in which a learned model is
not fully converged due to the ﬁniteness of the size of a training data set. Givena learning algorithm that can acquire a prediction model without indirect prejudice, it will make a fair determination if inﬁnite training examples are available.
However, if the size of the training data set is ﬁnite, the learned classiﬁer may
lead to more unfair determinations than that observed in the training sample
distribution. Though such determinations are not intentional, they might awake
suspicions of unfair treatment. In other words, though the notion of convergenceat inﬁnity is appropriate in a mathematical sense, it might not be in a social
sense. We can quantify the degree of underestimation by assessing the resultant
diﬀerence between the training sample distribution over D, P̃r[·], and the distribution induced by a model, P̂r[·]. Along this line, we deﬁne the underestimationindex (UEI) using the Hellinger distance:  
2 
 
1

P̂r[y, s] − P̃r[y, s] =1 −P̂r[Y, S]P̃r[Y, S]. (5)UEI = 2
y,s∈D

y,s∈D

Note that we did not adopt the KL-divergence because it can be inﬁnite andthis property is inconvenient for an index.
Negative Legacy. Negative legacy is unfair sampling or labeling in the training
data. For example, if a bank has been refusing credit to minority people without
assessing them, the records of minority people are less sampled in a training data
set. A sample selection bias is caused by such biased sampling depending on thefeatures of samples. It is known that the problem of a sample selection bias can
be avoided by adopting speciﬁc types of classiﬁcation algorithms [24]. However,
it is not easy to detect the existence of a sample selection bias only by observing
training data. On the other hand, if a bank has been unfairly rejecting the loansof the people who should have been approved, the labels in the training data
would become unfair. This problem is serious because it is hard to detect andcorrect. However, if other information, e.g., a small-sized fairly labeled data set,can be exploited, this problem can be corrected by techniques such as transfer
learning [10].
Regulations or laws that demand the removal of latent prejudices are rare.
We investigate UEIs in the experimental sections of this paper, but we don’t
especially focus on underestimation. As described above, avoiding a negative
legacy can be diﬃcult if no additional information is available. We therefore
focus on the development of a method to remove indirect prejudice.

3

Prejudice Removal Techniques

We here propose a technique to reduce indirect prejudice. Because this technique
is implemented as a regularizer, which we call a prejudice remover, it can beapplied to wide variety of prediction algorithms with probabilistic discriminativemodels.

Fairness-Aware Classiﬁer with Prejudice Remover Regularizer

3.1

41

General Framework

We focused on classiﬁcation and built our regularizers into logistic regression
models. Y , X, and S are random variables corresponding to a class, non-sensitive
features, and a sensitive feature, respectively. A training data set consists of
the instances of these random variables, i.e., D = {(y, x, s)}. The conditional
probability of a class given non-sensitive and sensitive features is modeled by
M[Y |X, S; Θ], where Θ is the set of model parameters. These parameters are
estimated based on the maximum likelihood principle; that is, the parameters
are tuned so as to maximize the log-likelihood:ln M[yi |xi , si ; Θ].
(6)L(D; Θ) =(yi ,xi ,si )∈D

We adopted two types of regularizers. The ﬁrst regularizer is a standard one
to avoid over-ﬁtting. We used an L2 regularizer Θ22 . The second regularizer,
R(D, Θ), is introduced to enforce fair classiﬁcation. We designed this regularizer
to be easy to implement and to require only modest computational resources. By
adding these two regularizers to equation (6), the objective function to minimize
is obtained:
λ(7)−L(D; Θ) + ηR(D, Θ) + Θ22 ,2
where λ and η are positive regularization parameters.
We dealt with a classiﬁcation problem in which the target value Y is binary{0, 1}, X takes a real vectors, x, and S takes a discrete value, s, in a domain S.
We used a logistic regression model as a prediction model:
M[y|x, s; Θ] = yσ(x ws ) + (1 − y)(1 − σ(x ws )),(8)

where σ(·) is a sigmoid function, and the parameters are weight vectors forx, Θ = {ws }s∈S . Note that a constant term is included in x without loss ofgenerality. We next introduce a regularizer to reduce the indirect prejudice.
3.2

Prejudice Remover

A prejudice remover regularizer directly tries to reduce the prejudice index andis denoted by RPR . Recall that the prejudice index is deﬁned as
PI =


Y,S



P̂r[Y, S] ln

P̂r[Y, S]
P̂r[S]P̂r[Y ]=P̃r[X, S]

X,SX,S P̃r[X, S] can be replaced with (1/|D|)

M[Y |X, S; Θ] lnYP̂r[Y, S]
P̂r[S]P̂r[Y ].

(xi ,si )∈D , and then the scaling
factor, 1/|D|, can be omitted. The argument of the logarithm can be rewritten
as P̂r[Y |si ]/P̂r[Y ], by reducing P̂r[S]. We obtain(xi ,si )∈D y∈{0,1}M[y|xi , si ; Θ] ln

P̂r[y|si ]P̂r[y].

42

T. Kamishima et al.

The straight way to compute P̂r[y|s] is to marginalize M[y|X, s; Θ]Pr∗ [X|s] over
X. However, if the domain of X is large, this marginalization is computationallyheavy. We hence take an approach by which this marginalization is replaced witha sample mean. More speciﬁcally, this marginalization is formulated byPr∗ [X|s]M[y|X, s; Θ]dX,P̂r[y|s] =dom(X)

where dom(X) is the domain of X. We approximated this formula by the following sample mean:(xi ,si )∈D s.t. si =s M[y|xi , s; Θ]P̂r[y|s] ≈.
(9)|{(xi , si ) ∈ D s.t. si = s}|Similarly, we approximated P̂r[y] byP̂r[y] ≈(xi ,si )∈D M[y|xi , si ; Θ]

|D|

.

(10)

Note that in our preliminary work [12], we took the approach of replacing X
with x̄s , which is a sample mean vector of x over a set of training samples whose
corresponding sensitive feature is equal to s. However, we unfortunately failed
to obtain good approximations by this approach.
Finally, the prejudice remover regularizer RPR (D, Θ) is(xi ,si )∈D y∈{0,1}M[y|xi , si ; Θ] ln

P̂r[y|si ]P̂r[y],(11)

where P̂r[y|s] and P̂r[y] are equations (9) and (10), respectively. This regularizer
becomes large when a class is determined mainly based on sensitive features;thus, sensitive features become less inﬂuential in the ﬁnal determination. In thecase of logistic regression, the objective function (7) to minimize is rewritten asln M[yi |xi , si ; Θ] + η RPR (D, Θ) +(yi ,xi ,si )λws 22 ,
2

(12)

s∈S

where M[y|x, s; Θ] is equation (8) and RPR (D, Θ) is equation (11). In our experiment, parameter sets are initialized by applying standard logistic regression
to training sets according to the values of a sensitive feature, and this objectivefunction is minimized by a conjugate gradient method. After this optimization,
we obtain an optimal parameter set, {ws∗ }.
The probability of Y = 1 given a sample without a class label, (xnew , snew )can be predicted by
Pr[Y =1|xnew , snew ; {ws∗ }] = σ(xnew ws∗new ).

Fairness-Aware Classiﬁer with Prejudice Remover Regularizer

4

43

Experiments

We compared our method with Calders and Verwer’s method on the real data
set used in their previous study [3].
4.1

Calders-Verwer’s 2-naı̈ve-Bayes

We brieﬂy introduce Calders and Verwer’s 2-naı̈ve-Bayes method (CV2NB forshort), which was found to be the best of three methods in the previous study
using the same dataset [3]. The generative model of this method isM[Xi |Y, S].
(13)
Pr[Y, X, S] = M[Y, S]i

M[Xi |Y, S] models a conditional distribution of Xi given Y and S, and theparameters of these models are estimated in a similar way as in the estimationof parameters of a naı̈ve Bayes model. M[Y, S] models a joint distribution Y
and S. Because Y and S are not mutually independent, the ﬁnal determination
might be unfair. While each feature depends only on a class in the case of theoriginal naı̈ve Bayes, every non-sensitive feature, Xi , depends on both Y and S
in the case of CV2NB. M[Y, S] is then modiﬁed so that the resultant CV scoreapproaches zero. Note that we slightly changed this algorithm as described in[12], because the original algorithm may fail to stop.
4.2

Experimental Conditions

We summarize our experimental conditions. We tested a previously used real
data set [3], as shown in section 2.1. This set includes 16281 data in an adult.test
ﬁle of the Adult / Census Income distributed at the UCI Repository [7]. The target variable indicates whether or not income is larger than 50M dollars, and thesensitive feature is gender. Thirteen non-sensitive features were discretized by
the procedure in the original paper. In the case of the naı̈ve Bayes, parameters ofmodels, M[Xi |Y, S], are estimated by a MAP estimator with multinomial distribution and Dirichlet priors. In our case of logistic regression, discrete variables
are represented by 0/1 dummy variables coded by a so-called 1-of-K scheme.
The regularization parameter for the L2 regularizer, λ, is ﬁxed to 1, because theperformance of pure logistic regression was less aﬀected by this parameter in our
preliminary experiments. We tested six methods: logistic regression with a sensitive feature (LR), logistic regression without a sensitive feature (LRns), logistic
regression with a prejudice remover regularizer (PR), naı̈ve Bayes with a sensitive
feature (NB), naı̈ve Bayes without a sensitive feature (NBns), and Calders andVerwer’s 2-naı̈ve-Bayes (CV2NB). We show the means of the statistics obtainedby the ﬁve-fold cross-validation.
4.3

Experimental Results

Table 1 shows accuracies (Acc), NPI and UEI in section 2, and CV scores (CVS).
MI denotes mutual information between sample labels and predicted labels; NMI

44

T. Kamishima et al.
Table 1. A summary of experimental results
method

Acc

NMI

NPI

UEI

CVS

PI/MI

LR
LRns
PR η=5PR η=15PR η=30

0.851
0.850
0.842
0.801
0.769

0.267
0.266
0.240
0.158
0.046

5.21E-02
4.91E-02
4.24E-02
2.38E-02
1.68E-02

0.040
0.039
0.088
0.212
0.191

0.189
0.184
0.143
0.050
0.010

2.10E-01
1.99E-01
1.91E-01
1.62E-01
3.94E-01

NB
NBns
CV2NB

0.822
0.826
0.813

0.246
0.249
0.191

1.12E-01
7.17E-02
3.64E-06

0.068
0.043
0.082

0.332
0.267
-0.002

4.90E-01
3.11E-01
2.05E-05

NOTE: n1 En2  denotes n1 × 10n2 . L2 regularizer: λ = 1.

was obtained by normalizing this MI in a process similar to NPI. PI/MI quantiﬁes
a prejudice index that was sacriﬁced by obtaining a unit of information about
the correct label. This can be used to measure the eﬃciency in the trade-oﬀ
between prediction accuracy and prejudice removal. The smaller PI/MI value
indicates higher eﬃciency in this trade-oﬀ.
We ﬁrst compare the performance of our method with that of baselines inTable 1. Compared with NBns, our method was superior both in accuracy and
NPI at η = 5; and hence, ours was superior in the eﬃciency index, PI/MI.
When comparing LRns, the prejudice in decisions was successfully removed byour prejudice remover in exchange for the prediction accuracy. We next moved
on to the inﬂuence of the parameter, η, which controls the degree of prejudice removal. We expected that the larger the η, the more prejudice would be removed,
whereas accuracy might be sacriﬁced. According to Table 1, as η increased, ourPR generally become degraded in accuracy.
To further investigate the change of performance depending on this parameterη, we demonstrated the variations in accuracy (Acc), normalized prejudice index (NPI), and the trade-oﬀ eﬃciency between accuracy and prejudice removal(PI/MI) in Figure 1. We focus on our PR method. The increase of η generally
damaged accuracy because the prejudice remover regularizer is designed to remove prejudice by sacriﬁcing accuracy in prediction. This eﬀect was observedby the increase in NPI. The peak in trade-oﬀ eﬃciency was observed at η = 15.
More prejudice could be removed by increasing η, but the accuracy in prediction
was fairly damaged.
We next compared our PR with other methods. By observing Figure 1(c),our PR demonstrated better performance in trade-oﬀs between accuracy andprejudice removal than the NBns. When compared to the baseline LRns, moreprejudice was successfully removed by increasing η. The Figure 1(a) showed that
this was achieved by sacriﬁcing the prediction accuracy. The eﬃciencies in the
trade-oﬀs of our PR was better than those of LRns if η ranged between 0 and
20. The performance of CV2NB was fairly good, and our PR was inferior to it
except for accuracy at the lower range of η.

Fairness-Aware Classiﬁer with Prejudice Remover Regularizer
0.08

0.86

45

0.5
0.4

0.84
0.06

0.3

0.82

NBns
LRns
CV2NB
PR

0.78

0.76

0.2

0.04

0.80

0

10

0.02

20

30

0.01

0

(a) Acc

10

20

30

0.1

0

(b) NPI

10

20

30

(c) PI/MI

Fig. 1. The change in performance for the Adult / Census Income data according to η
NOTE: Horizontal axes represent the parameter η, and vertical axes represent statistics in each subtitle. Solid, chain, dotted, and broken lines indicate the statistics of
PR, CV2NB, LRns, and NBns, respectively. Larger Acc values indicate better performance, and smaller NPI and PI/MI values indicate better performance. NPI andPI/MI of CV2NB were out of the bounds of these charts and are properly noted inTable 1.

1.00

10

−1

10

−1

10

−2

10

−2

10

−3

10

−3

10

−4

10

−4

10

−5

10

−5

CV2NB
PR

0.95

0.90

0.85

0.80
0

100

(a) Acc

200

300

0

100

(b) NPI

200

300

0

100

200

300

(c) PI/MI

Fig. 2. The change in performance for our synthetic data according to η
NOTE: The meanings of axes and line styles are the same as in Figure 1.

To show how the diﬀerence in the prejudice removal between CV2NB and PR
is brought about by the ability of our method to take into account the diﬀerence
in inﬂuence of diﬀerent features on sensitive information, we applied our PR
to a synthetic data set. To synthesize data, i was sampled from the normal
distribution N (0, 1), and si ∈ {0, 1} was sampled uniformly at random. The
ﬁrst feature xai = i , and the second feature xbi = 1 + i if si = 1; otherwise
xbi = −1+i . The class yi was set to 0 if xai +xbi < 0; otherwise 1. We generated
20 000 samples and applied CV2NB and our PR by changing η from 0 to 300.
Because xai and xbi are equivalent up to bias, these two features are comparable
in usefulness for class prediction. The ﬁrst feature, xai , is independent from si ,while the second feature, xbi , depends on si .
We showed the change in three indexes, accuracy, NPI, and PI/MI, on independently generated test data according to the parameter η in Figure 1. Unfortunately, results became unstable if η is larger than 200 because the objectivefunction (12) has many local minima for large η. However, when comparing the

46

T. Kamishima et al.
Table 2. The learned weight vectors w0 and w1 in equation (8)
w0

w1

η=0
[11.3, 11.3 ,−0.0257] [11.3, 11.4 ,0.0595]
η = 150[55.3,−53.0, −53.6 ] [56.1,−54.1, 53.6 ]NOTE: The ﬁrst, second, and third elements of ws were weights for the ﬁrst feature,
xai , the second feature, xbi , and a bias constant, respectively.

results in Table 1 with those in this ﬁgure, the diﬀerences in NPI derived by
CV2NB and PR became much smaller.
To exemplify the reason for these diﬀerences, we then showed the learned
weight vectors w0 and w1 in equation (8) in Table 2. By observing the weights
more carefully, the weights for xai and xbi were roughly equal when η = 0.
However, when η = 150, the absolute values of weights for xbi were smaller thanthose for xai . This indicates that to remove prejudice, our PR tries to ignorefeatures that depend on a sensitive feature. Therefore, if there are features thatare useful for classiﬁcation and additionally independent from a sensitive feature,
our PR can remove prejudice eﬀectively. In other words, our method is designed
to learn a classiﬁcation model by taking into account diﬀerence in inﬂuence of
diﬀerent features on sensitive information. On the other hand, according to thegenerative model (13), CV2NB treats all features equally and simply modiﬁes theM[Y, S] for removing prejudice. Therefore, CV2NB cannot learn a model thatreﬂects such diﬀerences.
This diﬀerence would cause the following eﬀect in practical use. When considering a case of credit scoring, because CV2NB treats all features equally, scores of
all individuals who are in a sensitive state would be raised equally. However, therepayment capacities of these individuals are certainly unequal, and our method
can change credit scoring by taking into account individuals’ repayment capacity.
On the other hand, if the repayment capacities of all individuals in a sensitive
state are nearly equal, our method cannot reduce prejudice without degrading
prediction accuracy. However, CV2NB can remove prejudice independently of thestates of individuals’ repayment capacity. Note that fair decision-making thattakes into account the diﬀerences in eﬀects of features has also been discussed
in [13,23].
In summary, our PR could successfully reduce indirect prejudice when compared with baseline methods. Our method is inferior to CV2NB in its eﬃciency
of prejudice removal, but it can learn a classiﬁcation rule by taking into account the diﬀerence in inﬂuence of diﬀerent features on sensitive information.
Additionally, our framework has the advantage that it can be applied to any
probabilistic discriminative classiﬁer.

5

Related Work

Several analytic techniques that are aware of fairness or discrimination haverecently received attention. Pedreschi et al. emphasized the unfairness inFairness-Aware Classiﬁer with Prejudice Remover Regularizer

47

association rules whose consequents include serious determinations [17].They
advocated the notion of α-protection, which is the condition that association
rules were fair. Given a rule whose consequent exhibited determination is disadvantageous to individuals, it would be unfair if the conﬁdence of the rule
substantially increased by adding a condition associated with a sensitive feature
to the antecedent part of the rule. The α-protection constrains the rule so thatthe ratio of this increase is at most α. They also suggested the notions of directdiscrimination and indirect discrimination. A direct discriminatory rule directly
contains a sensitive condition in its antecedent, and while an indirect discriminatory rule doesn’t directly contain a sensitive condition, the rule is considered
to be unfair in the context of background knowledge that includes sensitive information. Their work has since been extended [18]. Various kinds of indexes
for evaluating discriminatory determinations were proposed and their statisticalsigniﬁcance has been discussed. A system for ﬁnding such unfair rules has been
proposed [20].
Calders and Verwer proposed several methods to modify naı̈ve Bayes for enhancing fairness as described in section 4.1 [3]. Kamiran et al. developed algorithms for learning decision trees while taking fairness consideration [11]. Whenchoosing features to divide training examples at non-leaf nodes of decision trees,
their algorithms take care of the information gain regarding sensitive information as well as about target decisions. Additionally, the labels at leaf nodes arechanged so as to avoid unfair decisions.
Luong et al. proposed a notion of situation testing, wherein a determinationis considered unfair if diﬀerent determinations are made for two individuals allof whose features are equal except for sensitive ones [13]. Such unfairness was
detected by comparing the determinations for records whose sensitive features
are diﬀerent, but are neighbors in non-sensitive feature space. If a target determination diﬀers, but non-sensitive features are completely equal, then a target
variable depends on a sensitive variable. Therefore, this situation testing has
connection to our indirect prejudice.
Dwork et al. argued a data transformation for the purpose of exporting data
while keeping aware of fairness [5]. A data set held by a data owner is transformed
and passed to a vendor who classiﬁes the transformed data. The transformation
preserves the neighborhood relations of data and the equivalence between the
expectations of data mapped from sensitive individuals and from non-sensitive
ones. In a sense that considering the neighborhood relations, this approach isrelated to the above notion of situation testing. Because their proposition 2.2 implies that the classiﬁcation results are roughly independent from the membershipin a sensitive group, their approach has relation to our idea of prejudice.
Žliobaitė et al. discussed handling conditional discrimination [23]. They considered the case where even if the diﬀerence between probabilities of receiving
advantageous judgment given diﬀerent values of sensitive features, some extent
of the diﬀerence can be explained based on the values of non-sensitive features.
For example, even though females are less frequently admitted to a university
than males, this decision is considered as fair if this is due to the fact that

48

T. Kamishima et al.

females tend to try more competitive programs. They proposed a sampling technique to remove the unfair information from training samples while excluding
such explainable factors.
In a broad sense, fairness-aware learning is related to causal inference [16],because the ﬁnal decision becomes unfair if the decision depends on a sensitive
status. Fairness in data mining can be interpreted as a sub-notion of legitimacy,
which means that models can be deployed in the real world [19]. Gondek and
Hofmann devised a method for ﬁnding clusters that were not relevant to a given
grouping [8]. If a given grouping contains sensitive information, this method canbe used for clustering data into fair clusters. Independent component analysis
might be used to maintain the independence between features [9].
The removal of prejudice is closely related to privacy-preserving data mining [1], which is a technology for mining useful information without exposing
individual private records. The privacy protection level is quantiﬁed by mutual
information between the public and private realms [22]. In our case, the degreeof indirect prejudice is quantiﬁed by mutual information between classiﬁcation
results and sensitive features. Due to the similarity of these two uses of mutual
information, the design goal of fairness-aware learning can be considered theprotection of sensitive information when exposing classiﬁcation results. In ourcase, the leaked information is quantiﬁed by mutual information, but other criteria for privacy, such as diﬀerential privacy [14], might be used for the purposeof maintaining fairness.
Techniques of cost-sensitive learning [6] might be helpful for addressing underestimation problems.
As described in section 2.2, the problem of negative legacy is closely related
to transfer learning. Transfer learning is “the problem of retaining and applying
the knowledge learned in one or more tasks to eﬃciently develop an eﬀective
hypothesis for a new task” [10]. Among many types of transfer learning, theproblem of a sample selection bias [24] would be related to the negative legacy
problem. Sample selection bias means that the sampling is not at random, butbiased depending on some feature values of data. Another related approach totransfer learning is weighting samples according the degree of usefulness for thetarget task [4]. Using these approaches, if given a small amount of fairly labeled
data, other data sets that might be unfairly labeled would be correctly processed.

6

Conclusions and Future Work

The contributions of this paper are as follows. First, we proposed three causes of
unfairness: prejudice, underestimation, and negative legacy. Prejudice refers to
the dependence between sensitive information and the other information, either
directly or indirectly. We further classiﬁed prejudice into three types and developed a way to quantify them by mutual information. Underestimation is thestate in which a classiﬁer has not yet converged, thereby producing more unfair
determinations than those observed in a sample distribution. Negative legacy is
the problem of unfair sampling or labeling in the training data. Second, we developed techniques to reduce indirect prejudice. We proposed a prejudice removerFairness-Aware Classiﬁer with Prejudice Remover Regularizer

49

regularizer, which enforces a classiﬁer’s independence from sensitive information.
Our methods can be applied to any algorithms with probabilistic discriminativemodels and are simple to implement. Third, we showed experimental resultsof logistic regressions with our prejudice remover regularizer. The experimental
results showed the eﬀectiveness and characteristics of our methods.
Research on fairness-aware learning is just beginning, and there are many
problems yet to be solved: for example, the deﬁnition of fairness in data analysis,
measures for fairness, and maintaining other types of laws or regulations. The
types of analytic methods are severely limited at present. Our method can beeasily applied to regression, but fairness-aware clustering and ranking methods
are also needed. Because of the lack of convexity of the objective function, ourmethod is occasionally trapped by local minima. To avoid this, we plan to try
other types of independence indexes, such as kurtosis, which has been used forindependent component analysis. If a sensitive feature is a multivariate variable
whose domain is large or is a real variable, our current prejudice remover cannot
be applied directly; these limitations must be overcome.
The use of data mining technologies in our society will only become greaterand greater. Unfortunately, their results can occasionally damage people’s lives[2]. On the other hand, data analysis is crucial for enhancing public welfare. For
example, exploiting personal information has proved to be eﬀective for reducing
energy consumption, improving the eﬃciency of traﬃc control, preventing infectious diseases, and so on. Consequently, methods of data exploitation that do notdamage people’s lives, such as fairness/discrimination-aware learning, privacypreserving data mining, or adversarial learning, together comprise the notion ofsocially responsible mining, which it should become an important concept in thenear future.
Acknowledgments. We wish to thank Dr. Sicco Verwer for providing detail
information about his work and to thank the PADM2011 workshop organizers
and anonymous reviewers for their valuable comments. This work is supported
by MEXT/JSPS KAKENHI Grant Number 16700157, 21500154, 22500142,
23240043, and 24500194, and JST PRESTO 09152492.

References
1. Aggarwal, C.C., Yu, P.S. (eds.): Privacy-Preserving Data Mining: Models and Algorithms. Springer (2008)
2. Boyd, D.: Privacy and publicity in the context of big data. In: Keynote Talk of
The 19th Int’l Conf. on World Wide Web (2010)
3. Calders, T., Verwer, S.: Three naive bayes approaches for discrimination-free classiﬁcation. Data Mining and Knowledge Discovery 21, 277–292 (2010)
4. Dai, W., Yang, Q., Xue, G.R., Yu, Y.: Boosting for transfer learning. In: Proc. ofthe 24th Int’l Conf. on Machine Learning, pp. 193–200 (2007)
5. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., Zemel, R.: Fairness through awareness. arxiv.org:1104.3913 (2011)
6. Elkan, C.: The foundations of cost-sensitive learning. In: Proc. of the 17th Int’l
Joint Conf. on Artiﬁcial Intelligence, pp. 973–978 (2001)

50

T. Kamishima et al.

7. Frank, A., Asuncion, A.: UCI machine learning repository. School of Information and Computer Sciences, University of California, Irvine (2010),
http://archive.ics.uci.edu/ml
8. Gondek, D., Hofmann, T.: Non-redundant data clustering. In: Proc. of the 4thIEEE Int’l Conf. on Data Mining, pp. 75–82 (2004)
9. Hyvärinen, A., Karhunen, J., Oja, E.: Independent Component Analysis. WileyInterscience (2001)
10. NIPS workshop — inductive transfer: 10 years later (2005),
http://iitrl.acadiau.ca/itws05/
11. Kamiran, F., Calders, T., Pechenizkiy, M.: Discrimination aware decision tree learning. In: Proc. of the 10th IEEE Int’l Conf. on Data Mining, pp. 869–874 (2010)
12. Kamishima, T., Akaho, S., Sakuma, J.: Fairness-aware learning through regularization approach. In: Proc. of The 3rd IEEE Int’l Workshop on Privacy Aspects ofData Mining, pp. 643–650 (2011)
13. Luong, B.T., Ruggieri, S., Turini, F.: k-NN as an implementation of situation
testing for discrimination discovery and prevention. In: Proc. of the 17th Int’l
Conf. on Knowledge Discovery and Data Mining, pp. 502–510 (2011)
14. Nissim, K.: Private data analysis via output perturbation. In: Aggarwal, C.C.,
Yu, P.S. (eds.) Privacy-Preserving Data Mining: Models and Algorithms, ch. 4.
Springer (2008)
15. Pariser, E.: The Filter Bubble: What The Internet Is Hiding From You. Viking(2011)
16. Pearl, J.: Causality: Models, Reasoning and Inference, 2nd edn. Cambridge University Press (2009)17. Pedreschi, D., Ruggieri, S., Turini, F.: Discrimination-aware data mining. In: Proc.
of the 14th Int’l Conf. on Knowledge Discovery and Data Mining (2008)
18. Pedreschi, D., Ruggieri, S., Turini, F.: Measuring discrimination in sociallysensitive decision records. In: Proc. of the SIAM Int’l Conf. on Data Mining, pp.
581–592 (2009)19. Perlich, C., Kaufman, S., Rosset, S.: Leakage in data mining: Formulation, detection, and avoidance. In: Proc. of the 17th Int’l Conf. on Knowledge Discovery andData Mining, pp. 556–563 (2011)
20. Ruggieri, S., Pedreschi, D., Turini, F.: DCUBE: Discrimination discovery indatabases. In: Proc of The ACM SIGMOD Int’l Conf. on Management of Data,
pp. 1127–1130 (2010)
21. Strehl, A., Ghosh, J.: Cluster ensembles — a knowledge reuse framework for combining multiple partitions. Journal of Machine Learning Research 3, 583–617 (2002)
22. Venkatasubramanian, S.: Measures of anonimity. In: Aggarwal, C.C., Yu, P.S. (eds.)
Privacy-Preserving Data Mining: Models and Algorithms, ch. 4. Springer (2008)
23. Žliobaitė, I., Kamiran, F., Calders, T.: Handling conditional discrimination. In:
Proc. of the 11th IEEE Int’l Conf. on Data Mining (2011)
24. Zadrozny, B.: Learning and evaluating classiﬁers under sample selection bias. In:
Proc. of the 21st Int’l Conf. on Machine Learning, pp. 903–910 (2004)