Generalizing Tree Models for Improving Prediction Accuracy
Authors: Jaemin Yoo
Lee Sael
Venue:  (2021)

Abstract: Can we generalize and improve the representation power of tree models? Tree models are often favored over deep neural networks due to their interpretable structures in problems where the interpretability is required, such as in the classification of feature-based data where each feature is meaningful. However, most tree models have low accuracies and easily overfit to training data. In this work, we propose Decision Transformer Network (DTN), our highly accurate and interpretable tree model based on our generalized framework of tree models, decision transformers. Decision transformers allow us to describe tree models in the context of deep learning. Our DTN is proposed based on improving the generalizable components of the decision transformer, which increases the representation power of tree models while preserving the inherent interpretability of the tree structure. Our extensive experiments on 121 feature-based datasets show that DTN outperforms the state-of-the-art tree models and even deep neural networks.

Under review as a conference paper at ICLR 2021
GENERALIZING TREE MODELS FOR IMPROVING PRE- DICTION ACCURACY
Anonymous authors Paper under double-blind review
ABSTRACT
Can we generalize and improve the representation power of tree models? Tree models are often favored over deep neural networks due to their interpretable structures in problems where the interpretability is required, such as in the clas- sification of feature-based data where each feature is meaningful. However, most tree models have low accuracies and easily overfit to training data. In this work, we propose Decision Transformer Network (DTN), our highly accurate and inter- pretable tree model based on generalized framework of tree models, decision transformers. Decision transformers allow us to describe tree models in the con- text of deep learning. Our DTN is proposed based improving the generalizable components of the decision transformer, which increases the representation power of tree models while preserving the inherent interpretability of the tree structure. Our extensive experiments on 121 feature-based datasets show that DTN outper- forms the statc-of-the tree models and even deep neural networks.
INTRODUCTION
Can we generalize and improve the representation power of tree models? The tree models learn structure where the decision process easy to follow (Breiman et al., 1984). Due to this attractive feature of tree models, various efforts are being put to improve their performance or to utilize them as subcomponents of a deep learning model (Kontschieder et al., 2015; Shen et al., 2018).
Compared to typical deep neural networks, the main characteristic of tree models is that input data are propagated through layers without a change their representations; the internal nodes calculate the probability of an input arriving the leaf nodes. The decision process that determines the membership of a training example, i.c., the process of constructing the subsets of training data at each leaf, is the core operation in tree models, which we generalize and improve in this work.
Previous works on tree models consider the decisions at different nodes as separate operations, i.e., each node performs its own decision independently from the other nodes in the same layer, based on the arrival probability of an input x to the node. The independence of decisions allows learning to be simplified. However, in order obtain comprehensive view, the decisions of multiple nodes must be considered simultaneously. Furthermore, typical tree model with a depth of L requires 6²_ decision functions, where is a branching factor, which makes intractable to construct deep tree model.
In this work, we suggest that many tree models can be generalized what we transformer as Proposition 1. A decision transformer generalizes existing tree models by treating the decisions of each layer as a single operation involving all nodes in that layer. More specifically, the decision transformer views each layer stochastic decision (Def. 2) which linearly transforms the membership of each training data by learned stochastic matrix (Def. 1). The aggregated layer- wise view allows the decision transformer reduce the complexity of analysis to O(L), where L is the tree depth. Furthermore, formulating the tree models in a probabilistic context allows them to be understood by the deep learning framework and provides us theoretical foundation for deeply understanding the advantages and the limitations of existing tree models one framework.
the decision
The in-depth understanding of tree models decision transformers allows us to propose Decision Transformer Network (DTN), a novel architecture that inherits the interpretability of tree models while improving their representation power. DTN is extension of tree models into deep networks

---

Under review as a conference paper at ICLR 2021
that creates multiple paths between nodes based stochastic decisions. We propose two versions of DTN: DTN-D with full dense connections and DTN-S with sparse connections where the edges connected in senslity form. Our experiments show that DTN outperforms previous tree models and deep neural networks on 121 tabular datasets.
Our contributions are summarized as follows. First, we introduce generalized framework for tree models called the decision transformer and perform theoretical analysis on its generalizability and interpretability (Section 3). Second, we propose dense and sparse versions of Decision Transformer Network (DTN), our novel decision-based model that is both interpretable and accurate (Section 4). Third, we undergo extensive experiments on 121 tabular datasets to show that DTN outperforms the state-of-ther thrtthrrrrt art tree models and deep neural networks (Section 5).
RELATED WORKS
A tree model refers to a parameterized function that passes an example through its tree-structured layers without changing its representations. We categorize tree models based on whether it utilizes linear decisions over raw features or neural networks in combination with representation learning.
Tree Models with Linear Decisions Traditional tree models perform linear decisions over a raw feature. Given an example x, decision trees (DT) (Breiman et al., 1984) compare single selected element to a learned threshold at each branch to perform a decision. Soft decision trees (SDT) (Irsoy al., 2012) generalize DTs to all elements of by a logistic classifier, making probabilistic decision in each node. SDTs have been widely studied and applied to various problems due to its simplicity (Irsoy & Alpaydin. 2015; Frosst & Hinton, 2017: Linero Yang. 2018: Yoo & Sael, 2019). Deep neural decision trees (Yang et al., 2018) also extend DT's to a multi-brancher tree by splitting each feature element into multiple bins. These models provide a direct interpretation of predictions and decision processes. There are also well-known ensembles of such linear trees, such as random forests (Breiman, 2001) and XGBoost (Chen & Guestrin, 2016). These ensemble models tboost the performance of tree models at the expense of interpretability.
Tree Models with Representation Learning There recent works combining tree models with deep neural networks, which add the ability of representation learning to the hierarchical decisions of tree models. A popular approach is
abstract representations learned by convolutional neural networks (CNNs) as inputs of the decision functions of tree models (Kontschieder et al., 2015; Roy & Todorovic. 2016: Shen al., 2017: 2018; Wan et al., 2020), sometimes with multilayer percep- trons instead of CNNs (Bulò & Kontschieder, 2014). Another approach is to categorize raw exam- ples by hicrarchical decisions and then apply different classifiers to the separate clusters (Murthy et al., 2016). There are also approaches that improve the performance of deep neural networks by inserting hierarchical decisions as differentiable operations into neural network, instead of build- ing a complete tree model (Murdock et al., 2016; McGill & Perona, 2017; Brust & Denzler, 2019; Tanno et al., 2019). In this work, we focus on models having complete trees in its structure.
3 GENERALIZING TREE MODELS
We first introduce decision transformer, our unifying framework that represents tree model series of linear transformations over probability vectors. We then analyze the properties of decision transformers: the generalizability existing tree models and the interpretability.
3.1 DECISIONS TRANSFORMERS
Definition (Stochastic matrix). A matrix T is (left) stochastic if 2 O and =1 all j. A vector p is stochastic if satisfies the condition all stochastic vectors and matrices as P. i.e., we represent that T E P or E
matrix of size |p x 1. We denote the set of
Definition 2 (Stochastic decision). A function f is a stochastic decision if it linearly transforms an imput vector πt by left stochastic matrix as f(x)) = Tn, where T E P is independent of π.
Corollary 1. Given a stochastic decision f and vector n, f(π)) E P if P.P
2

---

Under review as a conference paper at ICLR 2021
dx(xx)
To
di (x)
2
X
de (x)
To
102 T₃
da (x)
10
11
12
13
14
(a) A tree model by traditional view.
(b) A tree model as decision transformer M((x).
Illustrations of a binary tree model by (a) traditional node-and-brance structure and (b) our representation as decision transformer. The traditional view treats the prediction as a sequence of decisions do, d1, dg, and dg following the path x. We understand the same model as a decision transformer represented series of linear transformations regardless of the actual path of x.
Figure I:
Based on the observation that each layer of tree model can be represented as stochastic decision that eventually learns the membership probability of an input x the leaf nodes, we propose deci- sion transformers which represent tree model as a series of stochastic decisions whose transition matrices are determined by x.
Proposition 1 (Decision transformer). parameterized classifier M is decision transformer if can be represented as
M(x: θ) = T.Tmm E ER.
(1)
where C is the number of target classes, To P is the initial decision probability, is the number of decisions, Ti P is transition matrix, and F P is classifier matrix. Ti and F are calculated from parameterized functions Ti and F, respectively, that take x as inputs:
T = Ti(x; E IR N
F(x; 0L) E RCNNL
(2)
where Ni is the size of decision vectors after the 1-th transformation. The initial decision probability mo is typically given as a hyperparameter, and thus O = 10o 0r.).
We define Tí and F separately since internal and leaf nodes have different functionality in most tree models, although they are basically the same. For example, the Ith internal layer of a typical decision tree updates a one-hot membership probability of x by stochastic matrix T,, which is generated by this case,
comparing an element of x to a learned threshold. matrix having nonzero elements at (2j.j) and (2j + 1, J, for every node j in the layer. On the other hand, the leaf layer utilizes a dense classification matrix that makes the final prediction.
a rectangular block-diage dick-dia
Figure illustrates two views of binary tree model. Figure la treats tree model as set of inde- pendent decisions that are selected based on the path of x. On the other hand, Figure 1b represents the tree series of linear transformations, based our framework of decision transformers.
3.2 GENERALIZING BINARY TREE MODELS
We now represent existing tree models decision transformers and determine their characteristics. We work with binary trees for simplicity, but the analysis can naturally extended to tree models having higher branching factors (Yang et al., 2018; Murthy et al., 2016; Tanno et al., 2019). Binary tree models can be represented with block diagonal transition matrices with nonzero values on the 2 x 1 diagonal blocks, such that each column contains most two nonzero elements that sum to one. Corollary 2 formalizes this notion.
Corollary 2. The I-th transition function Ti of a binary tree model is represented as
Ti(x) diag (1 di dx) di -
(x) 1- dx((x) ) 2
where diag(.) generates a rectangular block diagonal matrix, and de is the unit functionn of node which measures the probability of for taking the left branch node
3

---

Under review as a conference paper at ICLR 2021
Table 1: Summary of binary tree models and their representations as decision transformers: decision
trees (DT), soft decision trees (SDT), neural decision forests (NDF), deep neural decision forests (DNDF), and neural regression forests (NRF). Detailed explanations are presented in Section 3.2.
Model
di(x)
Fx)
DT (Breiman et al., 1984)
[1, x b,) > 0)
Onehot(O) (0j)
SDT (Irsoy et al., 2012)
(w, x + b,)
Categorical
NDF (Bulö & Kontschieder, 2014)
MLP,(rand(( (N;(
Categorical
DNDF (Kontschieder et al., 2015)
Categorical (0,)
NRF (Roy & Todorovic, 2016)
iN
Gaussian (0,)
Based on Corollary 2, we represent various tree models with specific forms of transition functions as summarized in Table 1. The unit decision function of decision trees (DT) selects a single feature threshold b; considering a learnable sign 8;-
of x by a one-hot vector 1, and then compares Each leave returns a one-hot vector of length C' as a prediction, where C' is the number of classes. Soft decision trees (SDT) extend DTs by a) producing probabilistic decision utilizing all features of x at each internal node and b) returning a soft categorical distribution at each Ieaf as a prediction.
The decision functions of these two models are linear with respect to the raw feature x and thus the decision processes are naturally interpretable.
Complex models that utilize deep neural networks for decision functions can also be generalized with the decision transformer. Neural decision forests (NDF) use a randomized multilayer percep- tron (MLP) as a decision function. Deep neural decision forests (DNDF) use a shared convolutional neural network (CNN) for all decisions with changing only the last fully- cully-connected layer. Neural regression forests (NRF) improve DNDFs by adopting a hierarchical CNN having different numbers of convolution operations based on the depth of node in addition to the last fullycconnected layer. Other models (Shen et al., 2017; 2018) that have an identical structure to DNDFs and NRFs.
3.3 ADVANTAGES OF USING MULTIPLE LAYERS
The transition matrices of a decision transformer in Proposition 1 can be collapsed to form a single transition matrix. However, there are two practical advantages keeping them long. First, dividing a strong black box learner into series of weak learners improves both the generalizability and inter- pretability of model. Using deep neural network as a single strong classifier is a popular approach in many domains, but is known to overfit easily especially in tabular data having insufficient training examples. We split it into multiple weak learners which linearly accumulate their representation power by matrix multiplications. Thus, it is possible to adjust the learning capacity of each weak learner individually, based on the complexity of the target problem and avoiding overfitting.
Second, the output π. from each layer is itself interpretable as probability vector that represents the property of the input x during the decision process. The previous works that combine tree models with deep neural networks focus on this advantage of hierarchical decisions, the given examples are clustered into separate bins based on their properties with respect to the target classes. This gives us novel insight into the given data, separately from the accuracy of predictions.
3.4 THEORETICAL ANALYSIS ON INTERPRETABILIE
It has been taken for granted that tree models are interpretable. However, with the integration of deep neural networks, i.e., representative black-box models, the interpretability of tree models in general becomes questionable. For a better understanding of the interpretability of tree models, we provide a theoretic description of the relationship between the interpretability of decision functions and the whole model with respect to score function for each feature element Theorem 1.
Theorem 1. Given layer of a decision transformer, assume that the transition functioo Tm of every layer m & is explainable by nonnegative score function s(-) such that s(zk, v(x)) quantifies the contribution of Ds to vxx) as a normalized score, i.e., Ee 5(Ix, v(x)) = 1. Then, the output n; of layer is explainable by the same score function s(xR, 11z for every j.
4

---

Under review as a conference paper at ICLR 2021
(a) Binary tree model.
(b) DTN-D.
(c) DTN-S.
Figure 2: A comparison between (a) a binary tree model, (b) DTN-D, and (c) DTN-S, assuming three layers each of which has four nodes. A tree model propagates an example through single path denoted by the bold lines. DTN-D uses nodes at the internal layer in the propagation, as all pairs of nodes connected. DTN-S simplifies the structure of DTN-D by imposing spatial locality, minimizing the number of parameters, still allowing a change of information between nodes.
Proof. The following equations hold for positive output values ti (x) and (2(x)):
saa u1 (x + 02 (x(x)) 11 (x))(< U1 (x)) 02 (x)s(((s (2())
(4)
s s( 11 (x) (x)) ( (x))
s(zk, 01 (x)0 (x)) Σn, s(n,, 01 (1((1_))))))))))))()) ( v2(x))
(5)
The first equation is derived from the fact that 24 contributes to the sum of values based on their sizes and the score for each value. The second equation involves normalization for all elements of x, as a multiplication does not preserve the normalized scores. Based on these properties, we derive the score s(K,, Tia (x)) for the i-th element of (x(xx) recursively as
s s(Zk, mb)
Σ
(x)( (x)) tuy (x)) (x))
(6)
x(x)
where (x) represents the element at position (i,j) of Ti(),, and =xx) is normalization constant. We prove the theorem given that the initial score is a constant as s(IR, moi) = 1/|x)
Corollary 3.
The prediction of a decision transformer for any target class is explainable by score fanction if the transition function Ti of every layer is explainable by 8.
decision transformer extends the interpretability of transition functions since they are combined by linear transformations; it is not possible to apply Equation if a nonlinear activation function or a bias term exists between transition matrices. It notable that our theoretical claim in this section should be distinguished from the structural interpretability of tree models that comes from their property that every hidden layer produces a probability vector. Corollary 3 focuses on how the interpretability of each layer relates to the interpretability of the whole model, based understanding of tree models as a decision transformer which aggregates all decisions at each layer single operation.
4 DECISION TRANSFORMER NETWORKS
Our decision transformer suggests that decision-based models that propagate an input by stochastic decisions
restricted tree structure. We propose Decision Transformer Network (DTN), an extension of tree models into deep network having fixed number of nodes at each layer. This solves the limitation of tree models that they have an exponential number of nodes with respect to tree depth, which prevents them from having large depth. Our DTN creates multiple paths from decision node to nodes at the lower layers, improving the representation power, along with its deep structure. Nevertheless, DTN is still interpretable as shown in Corollary 3.
We introduce two versions of DTN. DTN-D is a basic model having full dense connections between layers. DTN-S improves the efficiency and accuracy of DTN-D by utilizing locality- sparse connections and the spherical softmax function. Figure compares the structures of a tree model, DTN-D, and DTN-S. A tree model creates single path from node another, allowing no change of information between nodes once the branch is split. On the other hand, DTNs generate multiple paths between nodes, improving the representation power of tree models.
5

---

Under review as a conference paper at ICLR 2021
Table 2: Tested 121 datasets that are divided into three categories based on the numbers of examples.
# of Examples
# of Features
# of Labels Avg + Std
Category
# of Datasets
Min
Max
Avg
Avg + Std
Large
9
10,992
130,064
43,943
19.0 + 15.8
8.2 + 8.5
Mid
37
1,000
8,124
3,453
40.2 + 48.4
12.2 26.6
Small
75
10
990
371
24.4 = 37.9
4.1 + 3.7
All
121
10
130,064
4,555
28.8 = 40.8
6.9 = 15.5
4.1 DENSE MODEL WITH ALL PAIRWISE CONNECTIONS
Our DTN-D generates stochastic transition matrix from weighted linear function of input followed by the softmax activation. We first set To = 1, making sure that every example is fed to the root node and then distributed to multiple nodes lower layers. We then define the (i. element tas of the transition function Ti at layer as follows:
exp( w T
by x
+ bley)
(x)
Nesa k=
exp(W
buk))
where Way and buy are the learnable parameters for the index (i, at layer I. We define the classifier function F also as Equation 7, without separating and in the network.
4.2 SPARSE MODEL WITH LOCALITY-SENSIIIE DECISIONS
We propose DTN-S to improve the efficiency of DTN-D by generating sparse connections between adjacent layers. DTN-S limits the output field of each node to its W W-hop neighbors at the next layer, imposing a spatial locality between nodes; it makes each node learn its characteristics based on its position. As a result, DTN'S has ((NLD) parameters, which is N times smaller than O (N) of DTN-D, safely assuming that W is a small constant." N and D represent the number of nodes at each layer and the size of feature vectors, respectively.
At the same time, DTN-S uses the spherical softmax function (Martins & Astudillo, 2016) instead of the softmax to support generating sparse probability vectors. The softmax function at Equation 7 cannot produce a zero probability due the exponential function. On the other hand, the spherical softmax makes each element positive by the square function as spherical(z)) (z): 2// Σx &g instead of the exponential. As result, along with the locality- sensitive connections, DTN-S can effectively choose the target nodes to pass given example x by stochastic decisions.
As result, each element tii of the transition function Ti of DTN-S is defined as follows:
(xy((x) = (wTyxx + bui
Σ k=l (wTT,x + bru))² if |i -i S W otherwise 0.
(8)
EXPERIMENTS
We evaluate our proposed DTN models on tabular datasets where tree models have extensively been adopted. We use 121 datasets of the UCI Machine Learning Repository (Dua & Graff, 2017), which have been used as a benchmark dataset in recent works (Delgado et al., 2014; Olson et al., 2018). We download preprocessed datasets, scale each feature into the zero mean and unit variance, and divide each dataset into the 8:2 ratio for training and test.? 2 We categorize the datasets by the number of examples as in Table 1 into large (2 10K), medium (2 1K and < 10K), and small (<c 1K). We run each model five times for each category and report the average and standard deviation of accuracy.
We compare DTNS with existing tree models for tabular datasets, including multilayer perceptrons (MLP) that have been recently tuned for the datasets (Olson et al., 2018) that we experiment with.
We fix W to one in all of our experiments to show that even the smallest W performs well. The datasets available at http://prsooll.cinn www.:le.tsscttc.c..eeeeo wwwpss..w/.aa..ee..... htt..//e/mmmme...eaeeeee.eeeeeee www.fe/nnnee..lggddd//pr/rriiiip

---

Under review as a conference paper at ICLR 2021
Table 3: The accuracy of DTNs and competitors on four categories of datasets. The out of memory represents the shortage of GPU memory. DTNs show the highest accuracy in most categories.
Model(-layers)
Large
Medium
Small
All
MLP
91.25 + 8.85
84,51 + 13.92
79.08 + 17.77
81.65 + 16,56
DT
84.29 18.28
80.84 + 14.77
77.43 17.97
78.98 17.21
DNDT
88.63 + 6.76
78.30 = 17.38
75.23 = 16.25
77.16 16.49
SDT-10
87.23 + 14.01
80.78 + 16.44
76.88 = 17.18
78.85 = 17.00
SDT-12
88.10 + 13.31
81.69 + 15.62
77.25 + 17.24
79.41 - 16.79
SDT-14
88.63 + 12.57
82.11 + 15.46
77.49
17.39
79.73 + 16.82
SDT-16
out of memory
DTN-D (ours)
91.13 + 7.80
83.09 + 14.04
78. 19
17.36
80.65 + 16.21
DTN-S (ours)
91.40 7.98
84.22 + 13.78
79.63
16.87
81.91 15.84
MLPs have 2 hidden layers of 128 units, He-initialization (He et al., 2015), ELU activation (Clevert et al., 2016), and dropout (Srivastava et al., 2014).² For tree models, we implemented soft decision trees (SDT) (Frosst & Hinton, 2017) and deep neural decision trees (DNDT) ( (Yang et al., 2018) with PyTorch and used the scikis-learn implementation of decision trees (DT). Since DNDTs have an exponential number of parameters with respect to the number of features, we select 20 features for each dataset based on the importance scores computed by DTs. We train the models 200 epochs each using the Adam optimizer (Kingma & Ba, 2015) with the initial learning rates between 0.001 and 0.01: 0.001 in MLPs, 0.01 in SDTs and DNDTs, and 0.005 in DTNs. We ran all experiments on workstation with GTX 1080 Ti, using PyTorch for GPU computations.
5.1 COMPARISON WITH OTHER MODELS
We compare the accuracy of DTNs and competitors in Table The tree models show lower aceu- racy compared to MLPs, as they focus on the interpretability of decisions rather than the represen- tation power. Even though MLPs maximize the accuracy with a black box structure, our DTNs out- perform them in most categories of datasets, while being comparable in the medium-sized datasets. is also shown from the small standard deviations that the predictions of DTNS are robust as well as accurate, which is important advantage
off-the-shel classifier.
The table also shows the limitation of typical tree models, i.c., a large tree depth cannot be adopted due to exponential number of nodes. The accuracy of SDTs improves of layers until 14, however, deeper layers after 14 causes the out-of-memory error. As a result, the accuracy of SDTs worse than that of MLPs. DTs and DNDTs perform worse than SDTs as they use each feature independently by simply comparing its value to a learned threshold. Our proposed DTNs achieve high accuracy while maintaining interpretable as an improved tree model.
increase the number
5.2 INTERPRETABILITY OF DECISIONS
We visualize in Figure 3 the decision maps of DTN-S on the Iris dataset in the small category." 4 The dataset contains four features and three classes for classifying the class of an iris plant. The number of nodes set to 4 for clear visualization. The figures show that the examples are categorized into random clusters at the root layer, but the clusters become clearer as the number of layers increases, approaching the final classes. The blue and green classes are easily separated by the leaf classifier and thus mapped to the same cluster in Figure 3b. On the other hand, the examples of the orange class have several mixed clusters in Figure 3b, since they are located close to the green class.
'We have tested deeper layers, however, MLPs showed lower accuracies for layers deeper than 2. http://archiveiiis... www.iiieeeiss...ii..oom www...ddddmm////aaaaas///ittti
7

---

Under review as a conference paper at ICLR 2021
0-
-2-
(a) Layer O.
(b) Layer 15.
(c) Predictions.
(d) Labels.
Figure 3: The decision map at each layer of DTN-S for the Iris dataset in the small category. All feature vectors have been transformed by the principal component analysis (PCA) for visualization. Each point represents feature vector, whose color represents the node with the maximum member- ship probability. The decision maps of intermediate layers give valuable insight into the dataset.
Table 4: The accuracy of DTNs with various options of activation and sparsification functions. The spherical softmax consistently outperforms the softmax function, and locality connections show similar accuracy to dense connections even with fewer parameters.
Sparsity(-activation)
Large
Mid
Small
All
Dense-softmax (DTN-D)
91.13 + 7.80
83.09 14.04
78.19 + 17.36
80.65 16.21
Dense spnse-spherical
91.72 + 7.51
84.56 + 13.59
79.07 + 16.99
81.69 = 15.93
mofta--oftmax
90.49 + 8.67
83.28
14.26
79.11 16.92
81.23
15.91
Pruned-spherical
91.60 + 7.89
83.99 + 13.90
79.44 + 16.58
81.74 + 15.61
Local-softmax
90.60 + 8.20
83.42
14.19
79.13 + 16.81
81.30
15.89
Local-spherical (DTN-S)
91.40 + 7.98
84.22
13.78
79.63 + 16.87
81.91 15.84
5.3 ABLATION STUDY OF DTNS
We further compare various activation functions and sparsification methods of DTN in Table 4. We implement an additional baseline of sparse DTN for fair comparison, which randomly prunes the connections between adjacent layers maintain the same sparsity in DTN-S.
The accuracy of DTN improves as we adopt locality-senittie connections and spherical softmax. The spherical softmax outperforms the softmax all cases, regardless of the sparsity of connections, showing the importance of generating sparse probability vectors with a simple activation function. The locality-sensitive connections improve the accuracy of dense connections especially in the small datasets, where the efficiency and robustness are important for achieving high accuracy. The random sparse connections show generally lower accuracy than the locality- sensitivv connections, due the random nature that does not consider the positions of nodes in generating probabilities.
6 CONCLUSION
In this work, we propose Decision Transformer Network (DTN), an extension of tree models into deep network, which improves their representation power by generating multiple paths between nodes. In order to devise DTN, we first propose decision transformer, novel framework that generalizes existing tree models in the context of deep learning by formulating tree layers as stochas- tic decisions. Understanding tree models as decision transformer gives theoretical foundation for analyzing the stochastic nature of tree models regardless of the decision functions used.
Two versions of DTN are introduced: DTN-D, basic dense version, and DTN-S, sparse version having improved accuracy and efficiency. DTNN-D utilizes a series of linear transformations with the softmax activation, while DTN-S adopts locality-sensitive connections and the spherical softmax to improve the accuracy and efficiency DTN-D. We show that our DTN-S outperforms existing tree models and deep neural networks in 121 tabular datasets, demonstrating its representation power and robustness at the same time while generating interpretable decision processes.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Leo Breiman. Random forests. Mach. Learn., 45(1):5-32, 2001.
Leo Breiman, J. H. Friedman, R. A. Olshen, and C. J. Stone. Classification and Regression Trees. Wadsworth, 1984. ISBN 0-534-98053-8.
Clemens-Alexande Brust and Joachim Denzler. Integrating domain knowledge: Using hierarchies to improve deep classifiers. In ACPR, volume 12046 of Lecture Notes in Computer Science, pp- 3-16. Springer, 2019.
Samuel Rota Bulò and Peter Kontschieder. Neural decision forests for semantic image labelling, In CVPR, pp. 81-88. IEEE Computer Society, 2014. doi: 10.1 10.1109CC2R.2
Tianqi Chen and Carlos Guestrin. Xgboost: scalable tree boosting system. In KDD, pp. 785-794. ACM, 2016.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In ICLR, 2016.
Manuel Fernández Delgado, Eva Cernadas, Senén Barro, and Dinani Gomes Amorim. Do we need hundreds of classifiers to solve real world classification problems? J. Mach. Learn. Res., 15(1): 3133-3181, 2014.
Dheeru Dua and Casey Graff. UCI machine learning repository. 2017.
Nicholas Frosst and Geoffrey Hinton. Distilling a neural network into a soft decision tree. In CE 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Ozan Issoy and Ethem Alpaydin. Autoencoder trees. In ACMI, volume 45 of JMLR Workshop and Conference Proceedings, pp. 378-390. JMLR.org, 2015.
Ozan Insoy, Olcay Taner Yildiz, and Ethem Alpaydin. Soft decision trees. In 2012.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Buld. Deep neural decision forests. In ICCV, pp. 1467-1475. IEBE Computer Society, 2015.
Antonio R Linero and Yun Yang. Bayesian regression tree ensembles that adapt to smoothness and sparsity. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 80(5): 1087-1110, 2018
André F. T. Martins and Ramón Fernández Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. ICML, 2016.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural net- works. In Doina Precup and Yee Whye Teh (eds.), ICML, volume 70 of Proceedings of Machine Learning Research, pp. 2363-2372. PMLR, 2017.
Calvin Murdock, Zhen Li, Howard Zhou, and Tom Duerig. Blockout: Dynamic model selection for hierarchical deep networks. In CVPR, pp. 2583-2591. IEEE Computer Society, 2016.
Venkatesh N. Murthy, Vivek Singh, Terrence Chen, R. Manmatha, and Dorin Comaniciu. Deep decision network for multi-class image classification. In CVPR, 2016.
Matthew Olson, Abraham J. Wyner, and Richard Berk. Modern neural networks generalize on small data sets. In NeurIPS, 2018.
Anirban Roy and Sinisa Todorovic. Monocular depth estimation using neural regression forest. In CVPR, pp. 5506-5514, IEEE Computer Society, 2016.

---

Under review as a conference paper at ICLR 2021
Wei Shen, Kai Zhao, Yilu Guo, and Alan L. Yuille. Label distribution learning forests. In NIPS, pp- 834- 044-843,222 2017.
Wei Shen, Yilu Guo, Yan Wang, Kai Zhao, Bo Wang, and Alan L. Yuille. Deep regression forests for age estimation. In CVPR, pp. 2304-2313. IEEE Computer Society, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res., 15(1):
2014.
Ryutaro Tanno, Kai Arulkumaran, Daniel C. Alexander, Antonio Criminisi, and Aditya V. Nori. Adaptive neural trees. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), ICML, volume 97 of Proceedings of Machine Learning Research, pp. 6166-6175. PMLR, 2019.
Alvin Wan, Lisa Dunlap. Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph Gonzalez. NBDT: neural-backed decision trees. CoRR, abs/2004.00221, 2020.
Yongxin Yang, Irene Garcia Morillo, and Timothy M. Hospedales. Deep neural decision trees. /CMI. Workshop on Human Interpretability in Machine Learning, 2018.
Jaemin Yoo and Lee Sael. EDiT: interpreting ensemble models via compact soft decision trees. In ICDM, 2019.
10


Reviews:
[{"review_id": "kBk-XO3Eq_Y", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose Decision Transformer Networks (DTNs): a model that generalizes decision trees to deep network style decision graphs. Structurally, DTNs are similar to deep neural networks with layers of nodes operating as stochastic decision functions and which output probabilities of an input vector belonging to each sub-network. The probabilistic predictions of the decision nodes in a layer are aggregated in a \"transition matrix\" and passed to the next layer until the leaf layer is reached where the final classification decision is made. The goal of DTNs is to improve the representation power of tree models by transforming (exponentially large) trees to compact graphs without losing its interpretability power. The authors propose special scoring functions to explain the decisions made by  each layer as well as the model. On a diverse set of datasets ranging from small (<1K) to large (>10K), DTNs (dense and sparse) with softmax and spherical-softmax decision nodes, attained better accuracies than MLPs, conventional decision trees and their soft versions. \n\nThe motivation of the work is quite clear: to improve both the representation power and the interpretability of tree structured decision trees. Unfortunately,  I could not find the contribution(s) significant enough for ICLR.  Decision nodes making probabilistic predictions is not new. There is previous work that the authors cite. The cited works are mostly for tree structured models. But there are graph structured probabilistic models with superior representation capability but may lack interpretability. It seems that DTNs just use the gain in interpretability through the proposed scoring function which seems to be weak  and not generalizable. The paper is lacking in quality and clarity of presentation as well. \n\nThe architecture is not well explained. There is no defined root in DTNs, rather a set of first layer nodes which acts as the root layer, yet a single root is shown in Figure 2. Figure 2 is confusing: it is not clear what the blue nodes represent and why only certain edges are shown in bold (b and c). I understood from the description of DTNs that all nodes in a layer participate in making predictions, but the figure shows otherwise. Most of the formal definitions and descriptions in section 3 are focused on decision nodes with only two branches, left and right. But the authors don’t formally and clearly state how these could be generalized to multiple branches when formally introducing DTNs in section 4 and in experiments.  \n\nOne of the claimed contributions in the paper is the theoretical analysis of interpretability of DTNs (theorem 1). First, the notations used in the theorem are not defined, for example x_k, \\pi_{lj}, v_1(x) etc. Next, how does one interpret the score for a layer and for a decision. How does the score help to derive any meaningful explanations for a prediction? There were no experiments done to support this scoring function. The experiment on clustering is something obvious and doesn't show the benefit of using DTNs. Besides it is mentioned that the proposed scoring function does not apply to non-linear activation functions. It seems that this scoring function is weak as it will not work for other models that DTNs can be generalized to. \n\nThe question of how to train these models were completely omitted.  The authors only mentioned the activation functions used in nodes. I could not find any details on the architecture or any statistics on the size of these networks.  \n\nOther probabilistic models for discriminative tasks: \nSum-product networks: + Gens, Robert, and Pedro Domingos. \"Discriminative learning of sum-product networks.\" Advances in Neural Information Processing Systems. 2012.\n+van de Wolfshaar, J. and Pronobis, A., 2019. Deep Generalized Convolutional Sum-Product Networks for Probabilistic Image Representations. arXiv preprint arXiv:1902.06155.\nCutset networks: + Rahman, T., Jin, S. and Gogate, V., Cutset Bayesian Networks: A New Representation for Learning Rao-Blackwellised Graphical Models, in IJCAI 2019.\n+Rahman, T., Kothalkar, P., and Gogate, V. \"Cutset networks: A simple, tractable, and scalable approach for improving the accuracy of Chow-Liu trees\", ECML 2014.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "eQ46rauRZur", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose a new model that consists in transforming decision trees by seeing such models as a sequence of \"layers\" made of all nodes at the same tree depth. Each layer is then modelled as a stochastic decision (multiplication by a stochastic matrix) which can mimic the behaviour of both standard decision tree and stochastic trees (among others). Their idea is to keep the layer decomposition as it is to be able to understand/interpret each layer as pieces of the bigger picture (i.e., the interpretation of the model itself). \n\nOveral comment:\nThe approach is well motivated (comment (a)) and the paper is well-written. The most interesting feature of the proposed approach is the generalisation / unified view of several decision tree algorithms that may appear as completely different and but comes down to a mere choice of a type of decision function (kind of layers) in the proposed approach (similarly to what typically happens when building a neural network). On the other hand, this work is focused on mimicking the deep learning framework while neglecting a bit the tree-based learning framework (comments (b),(c)). Some practical details (comment (f)) or some explanation are however missing for in-depth understanding of the results (comments (c), (d), (e),(f)). Therefore, I think the paper could be accepted for the generalisation of existing approaches, the improvement of prediction accuracy (comment (d)) but the interpretability is somehow limited (comment (e)) despite a strong motivation regarding the interpretability of the proposed approach (comment (a)). \n\nMajor comments:\n(a) The motivation of focusing on the interpretability of combined tree-based and deep neural networks models is very good as these methods were most likely designed to improve the raw performances without considering the loss of interpretability.\n(b) To which extent your method relates with the tree-based learning framework? In particular, you said that \"the transition matrices of a decision transformer [...] can be collapsed to form a single transition matrix\". I totally agree that for the sake of interpretability you suggest not to consider the model as a single transition matrix but mathematically it can be rewritten as such. Therefore, how does it relates to kernel-based formulation of decision trees models? \n(c) It is unclear to me how is defined a \"contribution of x_k\". There is a large amount of working focusing on the measuring feature contributions from tree-based models. Similarly to (b), how your work relates to this? \n(d) Table 3 shows impressive results. However, it would have been interesting (maybe not better but informative at least) on the number of wins/ties/loss of your model (say the best one) wrt to the others. Indeed, given the standard deviations and the mean over many differents problems, it is not clear if your approach is always better or only partially.\n(e) I see the point of Figure 3 as showing the evolution of the decision map moving from not well separated samples to easily separable classes. However, the definition of \"random clusters\" (at the root layer for instance) is not clear as the problem is defined with three classes, I believe that at layer 0 you do not look where samples end and therefore have not idea on the predicted class (hence explaining the \"new\" colours). It is also surprising to consider \"two\" clusters at layer 0, then \"four colours\" at layer 15 while finishing with 3. Please clarify. \n(f) It would have been interesting to have more details on the practical details on the learning of your models (time, implementations, ...).", "extra_scores": {}, "extra_text": {}}, {"review_id": "VgDHR5KSLLX", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Edit: \n\nI have read the authors' response and the other reviews. I still believe that this paper is not ready for acceptance. \n\nSummary: \n\nThe authors present \"decision transformers,\" a family of classifiers of which decision trees (DTs) are a subset. They present \"Decision Transformer Networks\" (DTNs), which are a type of decision transformer where the number of nodes per layer is constant, the decision boundaries are soft, and the nodes are linked more densely than standard DTs. DTNs are experimentally compared to soft DTs, MLPs, and DTs, where the authors believe DTNs perform better than the alternatives. Also, the authors demonstrate that a decision transformer can be explained in terms of a score function if it has no non-linearities or bias terms between transition functions. \n\nReasons for score: \n\nDTNs are not shown to perform better than existing non-interpretable methods. DTNs are not shown to be interpretable in the sense that tree models are interpretable. DTNs may exhibit a favorable trade-off between accuracy and interpretability, but determining this requires defining and measuring interpretability. \n\nPros: \n\n-Decision transformers are explained very clearly and may be useful in future analysis of DT-style models.\n\n-Initial results for DTNs are favorable, so DTNs may be further refined and yield a good architecture.\n\n-The ablation study (Section 5.3) is a great inclusion which allows others to extend DTNs. \n\nCons:\n\n-The authors note that \"with the integration of deep neural networks ... the interpretability of tree models in general becomes questionable,\" but the proposed method is effectively integrating single-layer networks into its many transition matrices. The evaluated models have 16 layers with 128 nodes per layer (obtained from code; not entirely clear from paper text), use soft non-linearities, and have more connections than a tree-structured model. The paper would be improved by a clearer demonstration of the \"tree\" nature of the proposed model or how it could be interpreted in tree-like ways. \n\n-The theoretical analysis for interpretability does not apply to models \"if a nonlinear activation function or a bias term exists between transition matrices\". However, the non-linearity and bias terms have simply been moved into the transition matrices. As a result, the score function explanations are oblivious to the contents of the transition matrices. \n\n-Explanations in terms of importance values on input features are also applicable to DNNs; the benefit of a tree model is the structure. Likewise, the benefit of fewer parameters is the ability for a user to simulate a decision or more easily understand the entire model. Neither of these interactions seems possible with a DTN. The authors claim that DTNs preserve the \"inherent interpretability of the tree structure,\" though the tree structure is largely lost. One of the benefits of a hard-boundary tree is that the model's behavior over a subset of the input space can be described by a smaller (shallower) tree. Likewise, a single input traces a single path through the tree, so only one path per layer is relevant. For DTNs, the entire network is applicable to the entire input space. The network cannot be examined for a single region or for a single input without considering the entire network. This downside of DTNs is not evaluated and is largely not addressed. \n\n-The authors aim to create an interpretable method but do not measure how interpretable their method is nor define interpretability. Additional experiments on interpretability would support the authors' claims that their model is interpretable. \n\n-The experimental setup should be improved. The authors cite Olson et al. (2018) for their MLPs, but that work does not consider MLPs with 2 layers of 128 nodes (i.e., the architecture used by the authors of this work). In particular, an MLP with \"local\" connections (as in DTN-S) with depth 10 and 128 nodes per layer is similar to DTN-S and may perform better than the evaluated MLP. The way hyperparameters are selected should be made more clear to ensure that a fair comparison is made. To demonstrate the high performance of their method, the authors should compare to boosting and/or random forest approaches (not just MLPs).\n\n-Table 3 does not suggest statistically significant improvements over existing methods. \n\nQuestions During Rebuttal Period:\n\nPlease address and clarify the \"Cons\" above. In particular: How was the MLP architecture chosen? \n\nMinor Comments:\n\n-In the introduction, the authors note that DTs become wide as the number of layers increases, so deep trees cannot be constructed. However, there exist alternative constructions which limit the width of the tree. For example, all left-children can be constrained to be leaf nodes. This is akin to the limitation of the number of nodes per layer in DTNs.\n\n-Figure 1b has the input on the right and the output on the left. The figure would be clearer if the input was on the left and the output on the right (i.e., flip the figure). Very good figure, though.\n\n-It is clear from context that DTNs have a fixed number of nodes per layer, but this should be made clearer (as this is quite different from standard DTs and the decision transformer shown in Figure 1).\n\n-The width and depth of DTNs is not clearly stated in the paper.\n\n-The ablation study could be moved to the appendix in order to make room for an evaluation of interpretability. \n\n-The paper would benefit from another editing pass for grammar. \n\nSome Typos:\n\n-Section 2: \"ensemble methods tboost\" -> \"ensemble methods boost\"\n\n-Section 3.2: \"Each leave\" -> \"Each leaf\"\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "84bc8kSq6D", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I have read the author's response and decided to keep my review, confidence, and score. \n\n----\n\nSummary: this paper generalizes existing decision trees to some neural-style model. The most critical argument is that the model generalizes decision tree while maintaining interpretability. Since this is a new model, interpretability should at least be justified with the visualizations that can be judged as interpretable (let alone an objective measurement or human experiments). However, the demonstration of interpretability is far from satisfaction, so I recommend a clear rejection. See below for further details. \n\n1. One of the major motivation is using a linear number of parameters w.r.t. tree depth to construct a decision tree-style model. However, this problem has been investigated in literature but not compared theoretically or empirically at all in this paper. See the classic paper [1] and the recent paper [2]. \n\n2. The current proposition 1 does not seem useful. Any classifier $p(y | x)$ can be written as a decision transformer $F \\pi_0$ by letting $\\pi_0 = [1]$ and $F = p(y | x)$. The factorization only makes intuitive sense when each $T_i$ is restricted / interpretable. \n\n3. Interpreting a stochastically routing decision tree would inevitably involve inspecting the whole tree, since every prediction involves the whole tree. Hence, visualizing the whole tree is necessary to claim interpretability, especially for a new architecture. The only visualization in Fig. 3 is far from visualizing an interpretable model. \n\n4. The theoretical statement is not clear and rigorous. E.g., what do the authors mean by \"explainable\"?\n\n5. the proposed architectures seems to highly relevant to hierarchical mixture of experts [3], which can be trained via EM algorithms efficiently. Can the authors show similar things here?\n\n[1] Langley, Pat, and Stephanie Sage. \"Oblivious decision trees and abstract cases.\" Working notes of the AAAI-94 workshop on case-based reasoning. 1994.\n\n[2] Lee, Guang-He, and Tommi S. Jaakkola. \"Oblique Decision Trees from Derivatives of ReLU Networks.\" International Conference on Learning Representations. 2019.\n\n[3] Jordan, Michael I., and Robert A. Jacobs. \"Hierarchical mixtures of experts and the EM algorithm.\" Neural computation 6.2 (1994): 181-214.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a neural generalization of decision trees with the idea of maintaining interpretability. The approach falls a bit short on theoretical grounds. For example, the main theorem portraying interpretability isn't properly defined and some definitions appear implicitly in the proof. The view of decision trees as a sequence of soft decisions appears to need to model how the full probability distribution over the nodes propagates at each depth. A much stronger case for interpretability (rather than assuming that each T_i is interpretable) should be made if this is kept as one of the main arguments for the architecture. Interpretability of decision trees does not directly carry over to these models. 
 
Decision: Reject