TRANS-AM: Discovery Methodof Optimal Input Vectors Corresponding
to Objective Variables
Hiroaki Tanaka1(B) , Yu Suzuki1,2(B) , Koichiro Yoshino1,3(B) ,and Satoshi Nakamura1,2(B)1Graduate School of Information Science, Nara Institute of Science and Technology,
Ikoma, Japan{tanaka.hiroaki.sy2,ysuzuki,koichiro,s-nakamura}@is.naist.jp
2
Data Science Centre, Nara Institute of Science and Technology, Ikoma, Japan
3
PRESTO: Fundamental Information Technologies toward Innovative Social System
Design. Japan Science and Technology Agency, Kyoto, Japan

Abstract. In various ﬁelds, ensemble models by supervised learning areeﬀective, but the models cannot tell us how to modify the input vector sothat we will increase the objective variable more than a given thresholdor decrease it less than the threshold. In this paper, we propose a method,
TRANS-AM, that can discover an input vector satisfying the conditionof changing of the objective variable in regression problems by using aproperty of regression tree. The regression tree splits input space into
subspaces. There are subspaces with corresponding objective variables
satisfying such a condition. By transforming the input vector to new
input vectors belonging to one of the subspaces, we can discover a new
input vector whose distance from the original input vector is minimum
by satisfying the condition to change the objective variable. The reason
for “minimum” is the cost—if the new input vector is far from the original one, we need the signiﬁcant cost to modify the original input vector
to the new one. We evaluated the proposed method through numericalsimulations and investigated that the proposed method works well; theratio of the number of discovered input vectors satisfying the condition
per the number of discovered input vectors is 60% for the datasets generated through logistic function.

1

Introduction

As most of the researches in data mining and machine learning has focused on theaccuracy, eﬃciency, and robustness of diﬀerent techniques, little eﬀort has been
made for “actionable knowledge extraction” from advanced machine learningmodels. Here, “actionable knowledge extraction” means ﬁnding how to change
input vectors to improve the objective variables in supervised learning: improving
the objective variable in regression tasks. However, in the real world, we often
H. Tanaka—Presently with Research Laboratory, NTT DOCOMO Inc.
c Springer Nature Switzerland AG 2018

C. Ordonez and L. Bellatreche (Eds.): DaWaK 2018, LNCS 11031, pp. 216–228, 2018.
https://doi.org/10.1007/978-3-319-98539-8_17

TRANS-AM: Discovery Method of Optimal Input Vectors

217

face the diﬃculty of extracting such knowledge. In other words, we cannot obtainthe knowledge to answer the research question “How do we modify the input
vectors to increase the objective variables greater than a given threshold or
decrease them less than the threshold with minimum eﬀort?”
Let us consider a running example. We are strategists of a soccer team. It is
supposed that the winning percentage is predicted by random forest regression
whose objective variables are abilities of each position: speed, acceleration, dashspeed, and so on. If we can acquire the desirable input variables corresponding
to winning percentage which is upper than a given threshold—for example, wewant to keep the winning percentage upper than 80%, the threshold is 80; we
can plan the training so that the abilities of players will satisfy the desirable
input variables. In addition, if the desirable input variables corresponding to thewinning percentage which is upper than the threshold is far from the original
input variables, we cannot develop the good plan—it is diﬃcult to improve theplayers’ abilities exponentially. Therefore, the requirement is rewritten as “Wewant to know the desirable abilities of each player and the abilities are nearest
to original abilities of players.”
In this paper, we propose a method named, the Transforming-feAture Method(TRANS-AM) to answer the research question in a regression task. This method
enables us to modify the input vector adequately to increase or decrease theobjective variables by adding a small positive number ε to each input variablein the original input variables. With the property of a regression tree—it is
supposed that the objective variable is predicted by random forest regression[1]—, splitting the input space into subspaces, allows us to determine the input
vector whose objective variable improved. By transforming the input vector to
new input vectors belonging to one of the subspaces, we discover a new input
vector whose distance from the original input vector is minimum in those newvectors. The basic idea of the TRANS-AM is based on the method proposed byTolomei et al. [2]. Their method is built for classiﬁcation tasks, and we expanded
the method for regression tasks. We also relax a restriction of a regression tree
which is supposed in the previous study by Tolomei et al. [2]. Considering theprevious running example, the TRANS-AM enables us to plan the ideal training.
The contributions of this paper are as follows.
– We expanded the classiﬁcation task discussed in Tolomei et al. [2] to theregression task.
– We relaxed a restriction which is assumed in Tolomei et al. [2]: once we use
the input variable xi for a branch of tree we cannot use it for other branches.
The second contributions allows us to use more complex regression trees for the
random forest more than that used in Tolomei et al. [2].

2Related WorkWe ﬁrst discuss earlier studies on extracting actionable knowledge. Cao et al.
[3] and Robert et al. [4] focused on the development of eﬀective interestingness

218

H. Tanaka et al.

metrics. Liu et al. [5,6] proposed methods for pruning and summarizing thelearnt rules, as well as matching rules by similarity. Cao et al. [7,8] proposed adata mining method which is a paradigm shift from a research-centred discipline
to a practical tool for actionable knowledge. All the above methods depend on
the domains of datasets, but our proposed method does not.
We now discuss tree-based methods. Du et al. [9], Karim and Rahman [10],and Yang et al. [11,12] discussed post-proceeding methods speciﬁcally tailored
to decision trees. It is true that a decision tree is interpretable; therefore, we
can ﬁnd a modiﬁcation approach to improve the objective variable. However, adecision tree’s prediction precision is not good [13]. Our proposed method, on
the other hand, can use random forest whose prediction precision is better thanthat of a decision tree.
Finally, Cui et al. [14] proved that the optimal action extraction (OAE) problem similar to the problem we solve in Sect. 3 is generally NP-hard by reducing
it to DNF-MAXSAT [15] and formulated the problem in an integer linear programming formulation, which has been eﬃciently solved using current packages
with state-of-the-art solvers such as CPLEX [16]. Tolomei et al. [2] developed
a method of solving the OAE problem with an ε-satisfactory instance, which is
explained in Sect. 3. We call this method as actionable feature tweaking (AFT).
As mentioned in Sect. 1, the AFT requires the restriction: once we use the input
variable xi for a branch, we cannot use it for other branches. By this restriction, the modiﬁcation of input vectors is simpliﬁed. As we relax the restriction, we change the modiﬁcation approach, i.e. the approach to developing an
ε-satisfactory instance.

3

TRANS-AM: Proposed Method

In this section, we explain the TRANS-AM for feature transformation to increase
objective variables more than a given threshold or decrease them than thethreshold by expanding AFT. Actionable feature tweaking is used to change thelabel of an objective variable, i.e. the task addressed in AFT is classification.
We expanded AFT to change the objective variable to regression and liberalize one assumption of AFT regarding the root-to-leaf paths of each regression
tree. We published the source of TRANS-AM on https://github.com/settenQB/TRANS-AM.git
3.1

Notation

Let X ⊂ Rd be an input space and suppose that each x ∈ X is associated with anobjective variable y ∈ Y ⊂ R. We assume there is an unknown target function
f : X → Y. Most machine learning methods learn the function fˆ ≈ f fromN
dataset D = {(x n , yn )}n=1 . Speciﬁcally, fˆ is the estimate that best approximatesf on D, according to a speciﬁc loss function . The  measures the error between
predicted and observed values.

TRANS-AM: Discovery Method of Optimal Input Vectors

219

The interpretability of fˆ depends on the hypothesis space in which fˆ wasselected. In this study, we focused on random forest regression. Random forest
regressor T consists of K regression trees T1 , · · · , TK . We represent the estimateof each Tk , k = 1, · · · , K as ĥk . Then the estimate of T is calculated by the
sample mean of ĥk .
Regression tree Tk splits the input space X into subspaces as
X = Xk,1 ⊕ Xk,2 ⊕ · · · ⊕ Xk,M ,(1)and γk,m corresponds to the area Xk,m . Then the prediction with Tk is calculated byM
1 x ∈ Xk,mĥk (x ) =γk,m 1 [x ∈ Xk,m ] , 1 [x ∈ Xk,m ] =
.
(2)
0 x ∈
/ Xk,mm=1where 1[x ∈ Xk,m ] means the indicator function which is deﬁned as
1 x ∈ Xk,m.
1 [x ∈ Xk,m ] =0 x ∈/ Xk,m3.2(3)

Split Input Space with Regression Tree

Suppose that we are given the trained random forest regressor T and x satisfying
f (x ) = fˆ (x ) < t0 , where t0 is a hyperparameter meaning a lower threshold.
Our aim is to transform x to x  , which satisﬁes f (x  ) ≥ t1 , where t1 is also a
hyperparameter meaning an upper threshold.
We assume that Assumption 1 holds for any ﬁxed regression tree Tk .
Assumption 1. For any fixed regression tree Tk of the random forest T , 4 holds.
∃Xk,m ⊂ X

s.t.

∀x ∈ Xk,m , ĥk (x) = γk,m ≥ t1

(4)

t1
whose γk,m satisﬁes γk,m ≥By this assumption, we can select the subspace Xk,mt1
t1 , and Xk,m is written as
t1
Xk,m
=

d

 low upp θ i , θi,

θilow , θiupp ∈ R ∪ { −∞, ∞ } .

(5)

i=1

In (5), θilow and θiupp are decided by the random forest algorithm. In AFT, wehave to assume Assumption 2.
Assumption 2 (In AFT). The path pk,m of regression tree Tk from root tom-th leaf is represented as
 ≥
 ≥

.
(6)
pk,m = x1 ≥≤θ1 , x2 ≤θ2 , · · · , xd ≤θdAssumption (6) means that for all i either θilow = −∞ or θiupp = ∞ holds.
However, with the TRANS-AM, we do not adopt Assumption 2 because in many
actual cases the learned regression tree does not satisfy this assumption. Of
course we can build a regression tree by satisfying Assumption 2, but it is alittle messy and sacriﬁces prediction ﬂexibility.

220

3.3

H. Tanaka et al.

ε-Satisfactory Instancet1
in Tk ;Let Xk be the family of all Xk,m

Xk = { Xk,m | ∀x ∈ Xk,m , ĥk (x ) = γk,m ≥ t1 } ,(7)t1
where Xk,m satisfying ∀x ∈ Xk,m , ĥk (x ) = γk,m ≥ t1 is Xk,m
. Then, |Xk | ist1
often greater than 1. For all subspaces Xk,m ∈ X , we build the ε-satisfactory1
as following.
instances x tk(ε)⎧ upp⎪⎨θi − εt1
x k(ε) [i] = θilow + ε⎪⎩ θilow +θiupp2

θilow = −∞θiupp = ∞(8)otherwise1
In (8), ε means the distance between x tk(ε)
and the boundaries of subspace, i.e.,t1
the position of x k(ε) is determined by ε if θilow = −∞ or θiupp = ∞.
t1
1
∈ X . If x tk(ε)[i] =The ε-satisfactory instance deﬁned by (8) belongs to Xk,m
upp
t1
θi − ε, the i-th element of x k(ε) is the point transformed by −ε from θi1for the negative direction parallel to the xi axis (or vice versa). If x tk(ε)
[i] =
 low
 low upp 
uppt1
θi + θi/2, the i-th element of x k(ε) is the centre of interval θi , θi
.


t1
t1
t1
In any case x k(ε) ∈ Xk,m holds; therefore, ĥ x k(ε) ≥ t1 also holds.

3.4

Feature Transformation

t1
t1
Let X·,·
be the set of all ε-satisfactory instances. More speciﬁcally, X·,·is written by
K


t1
1
X·,·
=
x tk(ε),(9)k=1 m:X t1 ∈X
k,m


t1
1
1
where x tk(ε)
depends on subspace Xk,m
. Therefore, m:X t1 ∈X x tk(ε)
means thek,mset of all the ε-satisfactory instances made using Tk . Then the transformed input
vector we want is given by solving the following optimization problem.


t1
(10)
arg min
δ
x
,
x
x =
k(ε)


t

t

t

1 ∈X 1 |fˆ x 1≥t1
x k(ε)·,·k(ε)

1
In (10), cost function δ means the distance between x and x tk(ε). With this δ,
we choose optimal vector x , i.e. the selected x is nearest to the original input
vector. As we explained in Sect. 1, Cui et al. [14] proved that the OAE problem,
which is essentially identical to (10), is generally NP-hard, and the TRANSAM translates the OAE problem into a closed-form integer linear programming(ILP) problem, which can be eﬃciently solved by oﬀ-the-shelf ILP solvers. On

TRANS-AM: Discovery Method of Optimal Input Vectors

221

the other hand, Tolomei et al. [2] introduced an algorithm to obtain x  by using
an ε-satisfactory instance.
In this paper, we used an expanded algorithm of Tolomei et al. [2] to discovery
x  . We show the algorithm for ﬁnding x  in Algorithm 1. This algorithm seeksx  which is solution of the following optimization problem:(11)x  = arg min δ x , x k(ε)t1tt

1 ∈X 1x k(ε)·,·


1≥ t1 of (10), because the condition
In (11), we dropped the condition fˆ x tk(ε)
is implicitly satisﬁed by deﬁnition of ε-satisfactory instance. Algorithm 1 often
1
built
returns x  = x because sometimes all the ε-satisfactory instances x tj(ε)
t1with Algorithm 1 do not satisfy the 8-th line if-statement fˆ x j(ε) ≥ t1 . Hence
we should evaluate the TRANS-AM carefully in Sect. 4.

Algorithm 1. Algorithm of TRANS-AM
Require: T = { T1 , T2 , · · · , TK }, thresholds t0 , t1 , input vector x such that f (x ) < t0 ,cost function δ, and ε > 0
Ensure: x  satisfying fˆ (x  ) ≥ t1
1: x  ← x
2: δmin ← ∞3: for k = 1, 2, · · · , K do
4:
if fˆ (x ) < t1 ∧ ĥk (x ) < t1 then5:
make Xk
t1
6:
for Xk,m∈ Xk do1
7:
build ε-satisfactory instance x tj(ε)
1≥ t1 then8:
if fˆ x tj(ε)
19:if δ x , x tj(ε)< δmin then1x  ← x tj(ε)


1
11:
δmin ← δ x , x tj(ε)
12:end if13:
end if14:
end for15:
end if
16: end for
17: return x 

10:4

Numerical Simulation and Evaluation

In this section, we explain the results of numerical simulations. The aim with
TRANS-AM is to transform the input vector x with fˆ (x ) < t0 to x  satisfying

222

H. Tanaka et al.

f (x  ) ≥ t1 . If the random forest estimates the unknown target function f ,Algorithm 1 can return x  such that f (x  ) ≥ t1 . However, in practice the random
forest cannot estimate f completely, so the vector x  yielded from Algorithm
1 does not always satisfy f (x  ) ≥ t1 , i.e. Algorithm 1 sometimes yields x satisfying not f (x  ) ≥ t1 but fˆ (x ) ≥ t1 . Therefore, we should evaluate how many
vectors yielded from Algorithm 1 satisfy f (x  ) ≥ t1 . For such an evaluation, we
should know the target function f ; hence, we evaluated the TRANS-AM not
through application for real datasets but numerical simulations.
As we mentioned in Sect. 3, Algorithm 1 often returns x  = x Therefore, weuse (18) as an indicator for evaluating this problem.
4.1

Experimental Setting

We evaluated the TRANS-AM with artiﬁcial data. The artiﬁcial datasets weregenerated in the following steps, where 1 d is the d-dimensional vector whose
components are 1, Id is the d×d diagonal matrix whose diagonal components are
1, and Nd (1 d , Id ) means the d-dimensional Gaussian distribution whose mean
vector is 1 d and covariance matrix is Id .
i.i.d.

Step 1. generate x 1 , x 2 , · · · , x N ∼ Nd (1 d , Id ).
i.i.d.
Step 2. make independent variable y1 , y2 , · · · , yN by yn = f (x n ) + ηn , ηn ∼N (0, 1)
Note that f in the above steps is the same as the unknown target function in
Sect. 3. In our simulation, we used the following functions as f (x ).
f1 (x ) = a T x
f2 (x ) = sin a T xnf3 (x ) =ai sin xi

(12)
(13)
(14)

i=1


f4 (x ) = exp a T x
1f5 (x ) =1 + exp (a T x )

(15)
(16)

In functions (12)–(16), a ∼ N (0 , I) and ai is the i-th element of a. The TRANSAM has the parameters t0 , t1 and ε. Parameters t0 and t1 are determined byanalysts according to their aim, and ε should be turned using some kind ofdata-driven method. The following steps comprise the simulation process.
N0and test set
step 1. split the dataset D into training set Dtrain = {(x n , yn )}n=1
N1
Dtest = {(x n , yn )}n=1 , where N = N0 + N1
step 2. let t1 be the pupp percentile point of y ∈ Dtrain and let t0 be the plow
percentile point of y ∈ Dtrain .
step 3. ﬁx ε to one of candidates of ε and train the random forest regressor
using Dtrain .

TRANS-AM: Discovery Method of Optimal Input Vectors

223

step 4. choose the input vectors in Dtest whose objective variables are less thant0 .
step 5. transform the input vectors to the new input vectors x  with the
TRANS-AM.
step 6. evaluate the TRANS-AM by using three criterion that are shown in (17),(18), and (19).
Score P represents how many input vectors are transformed using the TRANSAM regardless of whether x  satisﬁes f (x  ) ≥ t1 , score Q indicates how many
input vectors are modiﬁed per number of input vectors, and score R represents
how many modiﬁed input vectors x  (= x ) satisfy f (x  ) ≥ t1 per number ofchanged input vectors. Algorithm 1 often yields the same vector as an input
vector. Therefore, we evaluated how many yielded vectors satisfy our purposef (x  ) ≥ t1 per number of transformed vectors with the score R.
|{ x  | f (x  ) ≥ t1 }|P = 

{ x | fˆ (x ) < t0 }

(17)|{ x  | x  = x }|
Q = 

{ x | fˆ (x ) < t0 }

(18)|{ x  | f (x  ) ≥ t1 }||{ x  | x  = x }|

(19)

R=

Among these three scores, a relationship P = QR holds. Score P is mostnoticeable score and scores Q and R construct P . We simulated all combinationsof ε ∈ { 0.01, 0.05, 0.1, 0.5, 1, 1.5 }, N0 = 1000, N1 = 250 and d = 50.
4.2

Result and Consideration

We show the simulation results in Figs. 1, 2 and 3. Figure 1 shows the relationshipbetween ε and score P , Fig. 2 shows the relationship between ε and score Q, andFig. 3 shows the relationship between ε and score Q. In the each ﬁgure, linear,sin, sinsum, exp, and logistic mean the datasets generated by (12), (13), (14),(15), and (16), respectively.
As shown in Fig. 1, for (plow , pupp ) = (40, 60), the dataset generated by (16)shows the best score P . In addition, (13) shows the second-best score P . This isbecause (16) and (13) are upper bounded, i.e. ∃K : const. s.t. ∀x ∈ R, f (x ) <K. As we explained in Sect. 3, the regression tree splits the input space intoM
subspaces { Xm }m=1 then corresponds γm with Xm . The precisions of approximating the functions that are not upper bounded with random forest become
worse due to this approximation. For example, suppose that we approximate thefunction g(x) = exp(x) and can use the regression tree. Then the input spaceis divided into M subspaces
X1 , · · · , XM . The prediction value γm of x ∈ Xm isgenerally (1/| { x ∈ Xm } |) x∈Xm x, where x is the training sample. Therefore,
the diﬀerence f (x) − γm  increases as x becomes larger.

224

H. Tanaka et al.

Fig. 1. Relationships between ε and P

Fig. 2. Relationships between ε and Q

Fig. 3. Relationships between ε and R

On the other hand, in Fig. 1(b) and (c), the score P for the dataset generatedby (16) becomes worse than that in the case of Fig. 1(a). The distribution off (x  ) is illustrated in Fig. 4. As shown in Fig. 4, the threshold is pulled in thepositive direction as percentile point becomes larger. Essentially, the objective
variable does not appear in the region upper than 1 due to the range of (16);

TRANS-AM: Discovery Method of Optimal Input Vectors

225

Fig. 4. Distributions of the objective variables generated by (16) and the threshold

we call the range of f (x) “essential range of the objective variable”—the noise
η aﬀects the range of the objective variable. The reason why the threshold in
the centre and right of Fig. 4 is greater than 1 is aﬀection of noise η. In the caseof Fig. 4(b) and (c), the setting of simulation—we aim to transform x into x satisfying f (x  ) ≥ pupp —is essentially impossible, because the objective variable
does not become greater than pupp . Therefore, the results of Fig. 4(b) and (c)are not strange. Videlicet, the settings of the centre and right of Fig. 5 are notsuitable, i.e. it is essentially impossible to obtain the transformed input vectorx  satisfying f (x  ) ≥ tupp because the threshold is out of the essential range ofthe objective variable.
As shown in Fig. 2(a) and (b), each score R is higher than 0.8, except for exp.
These results mean that for the datasets generated by (15), Algorithm 1 cannot
transform x into x  . This is also because random forest cannot approximate
Exponential Function (15) well. Of course, other functions (12) and (14) are
not upper bounded. However, Exponential Function (15) diverges to inﬁnity
earlier than (12) and (14). For f (x) = exp(ax), the diﬀerence between f (x)and f (x + c), where c is a very small positive integer, is larger than that forf (x) = ax. Therefore, the approximation precision for the dataset generated by(15) becomes worse than for the other. If the prediction precision is very thatt1
ˆbad, the condition f x j(ε) ≥ t1 in Algorithm 1 cannot be satisﬁed, i.e. x isyielded using Algorithm 1. Therefore, we can conclude that for the dataset whose
objective variable is generated by the exponential function of the input variableof the TRANS-AM cannot ﬁnd the modiﬁed x  . In the right ﬁgure of Fig. 2,
turning the ε well, the score Q become higher than 0.8.
As shown in the Fig. 3(a), the score Rs for the datasets generated by (13) and(16) can be greater than 0.4 by turning ε for plow = 40, pupp = 60. In Fig. 3(b)and 1(c), however, the score R for the datasets generated by (16) one of thelowest. The reason is referred to in the discussion of Fig. 1. As plow becomes
smaller and pupp becomes larger, the score R decrease.
Figure 5 illustrates the reason that the TRANS-AM performs well for thedataset generated by (13). The thresholds pupp for sin are not pulled in the226

H. Tanaka et al.

Fig. 5. Distributions of the objective variables generated by (13) and the thresholdpositive direction so strongly, as for logistic, i.e. the threshold appears in theessential range of the objective variable. Therefore, the TRANS-AM performed
better for the dataset generated by (13) than for the other datasets.
In all of that, the TRANS-AM does not work if the following Case 1 or 2
holds:
Case 1: the random forest cannot approximate the target function, e.g. theobjective variables are generated from an upper unbound function;
Case 2: the threshold is out of the essential range of the objective variable.
The case 1 corresponds to the case of exp (see Figs. 1, 2 and 3) and the case
2 corresponds to the case of sin and logistic (see the centre and right of Fig. 1
and 3). However, if neither of the two cases holds, TRANS-AM will work well—
we can obtain the input vectors satisfying that the objective variables are greaterthan the threshold from the candidates of input vectors by the score P ≥ 45.

5

Conclusion

We proposed the TRANS-AM for discovering a new input vector to increase theobjective variable greater than a given threshold or decrease it than the thresholdwith minimizing δ in a regression task. We relaxed the restrictions assumed byTolomei et al. [2] because the restrictions were not reasonable for random forest.
As we discussed in the introduction, we are often faced with the question “Howdo we modify the input vector to increase the objective variable greater than
the given threshold or decrease it than the threshold with minimum eﬀort?” The
TRANS-AM is an answer to this question and we have disclosed the situationthat the method works well.
In the TRANS-AM, we generated candidates of the new input vector by
ε-satisfactory instances. The ε-satisfactory instances satisfy the condition, i.e.
the objective variables corresponding to the ε-satisfactory instances are greaterthan the threshold. Then we select the new input vector which minimizes theTRANS-AM: Discovery Method of Optimal Input Vectors

227

distance between the original input vector and new one—we select the new input
vector by (11).
Considering the use case of the TRANS-AM, we have to pay attention to whatkind of input variables are contained in the input vector. The transformation by
TRANS-AM aﬀects all of the input variables generally, because of (8). Therefore,
in the use case, we have to use only input variables which can be changed. In
the case of example referred in Sect. 1—we are strategists of a soccer team, we
can use the energy, speed of players. However, we cannot use the body size ofthe players, because we cannot control their values.
We evaluated the TRANS-AM by moving ε and the thresholds—plow andpupp —in Sect. 4. Then we found that for the dataset whose objective variables
are generated by the sin formula, the TRANS-AM shows good precision; score
P and score R are maximum or pre-maximum in each case of Figs. 1 and 3.
On the other hand, for the dataset whose objective variables are generated by(16), the TRANS-AM does not work well except when (plow , pupp ) = (40, 60). In
such a situation, it is hard for TRANS-AM to ﬁnd the transformed input vector
satisfying the condition. These considerations are summarized as the Case 1 andCase 2. If we avoid the two cases, i.e. we can approximate the target function
by the random forest and the threshold is not out of the essential range of theobjective variable; we obtain the transformed input vector eﬃciently by using
TRANS-AM.
For future works, we plan to extend the TRANS-AM for gradient boosting[17] and XGBoost [18]—these additive tree models are eﬀective predictors forvarious ﬁelds [13,19–21]. TRANS-AM works well for datasets whose objective
variables are generated by the sin formula but does not work well for datasets
whose objective variables are generated by an upper unbounded formula (e.g.
exp formula). Therefore, a method that works well for upper unbounded datasetsis required for discovering new input vectors whose objective variables are larger
than a given threshold.
Acknowledgement. This research was partially supported by NAIST Big Data
Project.

References
1. Breiman, L.: Random forests. Mach Learn. 45(1), 5–32 (2001)
2. Tolomei, G., Silvestri, F., Haines, A., Lalmas, M.: Interpretable predictions of treebased ensembles via actionable feature tweaking. In: Proceedings of the 23rd ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.
465–474. ACM (2017)
3. Cao, L., Luo, D., Zhang, C.: Knowledge actionability: satisfying technical andbusiness interestingness. IJBIDM 2, 496–514 (2007)
4. Hilderman, R.J., Hamilton, H.J.: Applying objective interestingness measures indata mining systems. In: Zighed, D.A., Komorowski, J., Żytkow, J. (eds.) PKDD
2000. LNCS (LNAI), vol. 1910, pp. 432–439. Springer, Heidelberg (2000). https://doi.org/10.1007/3-540-45372-5 47

228

H. Tanaka et al.

5. Liu, B., Hsu, W.: Post-analysis of learned rules. In: AAAI/IAAI, vol. 1, pp. 828–834(1996)
6. Liu, B., Hsu, W., Ma, Y.: Pruning and summarizing the discovered associations. In:
Proceedings of the Fifth ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining, pp. 125–134. ACM (1999)
7. Cao, L., Zhang, C.: Domain-driven, actionable knowledge discovery. IEEE Intell.
Syst. 22(4) (2007)
8. Cao, L., Zhao, Y., Zhang, H., Luo, D., Zhang, C., Park, E.K.: Flexible frameworks
for actionable knowledge discovery. IEEE Trans. Knowl. Data Eng. 22(9), 1299–
1312 (2010)
9. Du, J., Hu, Y., Ling, C.X., Fan, M., Liu, M.: Eﬃcient action extraction with
many-to-many relationship between actions and features. In: van Ditmarsch, H.,
Lang, J., Ju, S. (eds.) LORI 2011. LNCS (LNAI), vol. 6953, pp. 384–385. Springer,
Heidelberg (2011). https://doi.org/10.1007/978-3-642-24130-7 29
10. Karim, M., Rahman, R.M.: Decision tree and naive bayes algorithm for classiﬁcation and generation of actionable knowledge for direct marketing. J. Softw. Eng.
Appl. 6(04), 196 (2013)
11. Yang, Q., Yin, J., Ling, C., Pan, R.: Extracting actionable knowledge from decision
trees. IEEE Trans. Knowl. Data Eng. 19(1), 43–56 (2007)
12. Yang, Q., Yin, J., Ling, C.X., Chen, T.: Postprocessing decision trees to extract
actionable knowledge. In: Third IEEE International Conference on Data Mining,
ICDM 2003, pp. 685–688. IEEE (2003)
13. Friedman, J., Hastie, T., Tibshirani, R.: The elements of statistical learning.
Springer Series in Statistics, vol. 1. Springer, New York (2001). https://doi.org/
10.1007/978-0-387-21606-5
14. Cui, Z., Chen, W., He, Y., Chen, Y.: Optimal action extraction for random forestsand boosted trees. In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 179–188. ACM (2015)15. Manindra, A., Thomas, T.: Satisﬁability problems. Technical report (2000)
16. CPLEX, I.I.: V12. 1: User’s manual for cplex. Int. Bus. Mach. Corp. 46(53), 157(2009)
17. Friedman, J.H.: Stochastic gradient boosting. Comput. Stat. Data Anal. 38(4),367–378 (2002)
18. Chen, T., Guestrin, C.: Xgboost: a scalable tree boosting system. In: Proceedingsof the 22nd ACM SIGKDD International Conference on Knowledge Discovery And
Data Mining, pp. 785–794. ACM (2016)19. Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A.,
Cook, M., Moore, R.: Real-time human pose recognition in parts from single depth
images. Commun. ACM 56(1), 116–124 (2013)
20. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2),137–154 (2004)
21. Tyree, S., Weinberger, K.Q., Agrawal, K., Paykin, J.: Parallel boosted regression
trees for web search ranking. In: Proceedings of the 20th International Conferenceon World Wide Web, pp. 387–396. ACM (2011)