Title: Permutation Decision Trees using Structural Impurity
Venue: NeurIPS 2023
Accepted: no
Keywords: Decision Tree

Effort-To-Compress

Structural Impurity

Permutation Bagging

Machine Learning
Abstract: Decision Tree is a well understood Machine Learning model that is based on minimizing impurities in the internal nodes. The most common impurity measures are \emph{Shannon entropy} and \emph{Gini impurity}. These impurity measures are insensitive to the order of training data and hence the final tree obtained is invariant to a permutation of the data. This leads to a serious limitation in modeling data instances that have order dependencies.  In this work, we use~\emph{Effort-To-Compress} (ETC) - a complexity measure, for the first time, as an impurity measure. Unlike Shannon entropy and Gini impurity, structural impurity based on ETC is able to capture order dependencies in the data, thus obtaining potentially different decision trees for different permutation of the same data instances (\emph{Permutation Decision Trees}). We then introduce the notion of {\it Permutation Bagging} achieved using permutation decision trees without the need for random feature selection and sub-sampling.  We compare the performance of the proposed permutation bagged decision trees with Random Forest. Our model does not assume independent and identical distribution of data instances. Potential applications include scenarios where a temporal order is present in the data instances.  

Abstract

Decision Tree is a well understood Machine Learning model that is based on minimizing impurities in the internal nodes. The most common impurity measures are Shannon entropy and Gini impurity . These impurity measures are insensitive to the order of training data and hence the final tree obtained is invariant to a permutation of the data. This leads to a serious limitation in modeling data instances that have order dependencies. In this work, we use Effort-To-Compress (ETC) - a complexity measure, for the first time, as an impurity measure. Unlike Shannon entropy and Gini impurity, structural impurity based on ETC is able to capture order dependencies in the data, thus obtaining potentially different decision trees for different permutation of the same data instances ( Permutation Decision Trees ). We then introduce the notion of Permutation Bagging achieved using permutation decision trees without the need for random feature selection and sub-sampling. We compare the performance of the proposed permutation bagged decision trees with Random Forest. Our model does not assume independent and identical distribution of data instances. Potential applications include scenarios where a temporal order is present in the data instances.

1 Introduction 17

The assumptions in Machine Learning (ML) models play a crucial role in interpretability, repro18

ducibility, and generalizability. One common assumption is that the dataset is independent and 19

identically distributed (iid). However, in reality, this assumption may not always hold true, as human 20

learning often involves connecting new information with what was previously observed. Psycho21

logical theories such as Primacy and Recency Effects [1], Serial Position Effect, and Frame Effect 22

suggest that the order in which data is presented can impact decision-making processes. In this work, 23

24

we have devised a learning algorithm that exhibits sensitivity to the order in which data is shuffled.

This unique characteristic imparts our proposed model with decision boundaries or decision functions 25

that rely on the specific arrangement of training data. 26

In our research, we introduce the novel use of 'Effort to Compress' (ETC) as an impurity function for 27

Decision Trees, marking the first instance of its application in Machine Learning. ETC effectively 28

measures the effort required for lossless compression of an object through a predetermined lossless 29

compression algorithm [2]. ETC was initially introduced in [3] as a measure of complexity for 30

timeseries analysis, aiming to overcome the limitations of entropy-based complexity measures. It 31

is worth noting that the concept of complexity lacks a singular, universally accepted definition. 32

In [2], complexity was explored from different perspectives, including the effort-to-describe (Shan33

non entropy, Lempel-Ziv complexity), effort-to-compress (ETC complexity), and degree-of-order 34

(Subsymmetry). The same paper highlighted the superior performance of ETC in distinguishing 35

between periodic and chaotic timeseries. Moreover, ETC has played a pivotal role in the development 36

of an interventional causality testing method called Compression-Complexity-Causality (CCC) [4]. 37

The effectivenss CCC has been tested in various causality discovery applications [5, 6, 7, 8]. ETC 38

has demonstrated good performance when applied to short and noisy time series data, leading to its 39 utilization in diverse fields such as investigating cardiovascular dynamics [9], conducting cognitive 40 research [10], and analysis of muscial compositions [11]. The same is not the case with entropy based 41

fresh perspective on its ability to capture structural impurity. Leveraging this insight, we introduce a 44 decision tree classifier that maximizes the ETC gain. It is crucial to highlight that Shannon entropy 45 and Gini impurity fall short in capturing structural impurity, resulting in an impurity measure that 46 disregards the data's underlying structure (in terms of order). The utilization of ETC as an impurity 47 measure provides the distinct advantage of generating different decision trees for various permutations 48 of data instances. Consequently, this approach frees us from the need to adhere strictly to the i.i.d. 49 assumption commonly employed in Machine Learning. Thus, by simply permuting data instances, 50

The paper is structured as follows: Section 2 introduces the Proposed Method, Section 3 presents the 52 Experiments and Results, Section 4 discusses the Limitations of the research, and Section 5 provides 53 the concluding remarks and outlines the future work. 54

55

56

57

58

59

2 Proposed Method

In this section, we establish the concept of structural impurity and subsequently present an illustrative example to aid in comprehending the functionality of ETC.

Definition: Structural impurity for a sequence S = s 0 , s 1 , . . . , s n , where s i ∈ { 0 , 1 , . . . , K } , and K ∈ Z + is the the extent of irregularity in the sequence S .

We will now illustrate how ETC serves as a measure of structural impurity. The formal definition 60 of ETC is the effort required for lossless compression of an object using a predefined lossless 61 compression algorithm. The specific algorithm employed to compute ETC is known as Non-sequential 62 Recursive Pair Substitution (NSRPS). NSRPS was initially proposed by Ebeling [12] in 1980 and 63 has since undergone improvements [13], ultimately proving to be an optimal choice [14]. Notably, 64 NSRPS has been extensively utilized to estimate the entropy of written English [15]. The algorithm 65 is briefly discussed below: Let's consider the sequence S = 00011 to demonstrate the iterative steps 66 of the algorithm. In each iteration, we identify the pair of symbols with the highest frequency and 67 replace all non-overlapping instances of that pair with a new symbol. In the case of sequence S , the 68 pair with the maximum occurrence is 00 . We substitute all occurrences of 00 with a new symbol, let's 69 say 2 , resulting in the transformed sequence 2011 . We continue applying the algorithm iteratively. 70 The sequence 2011 is further modified to become 311 , where the pair 20 is replaced by 3 . Then, the 71 sequence 311 is transformed into 41 by replacing 31 with 4 . Finally, the sequence 41 is substituted 72 with 5 . At this point, the algorithm terminates as the stopping criterion is achieved when the sequence 73 becomes homogeneous. ETC, as defined in [3], represents the count of iterations needed for the 74 NSRPS algorithm to attain a homogeneous sequence. 75

We consider the following three sequence and compute the ETC: 76

Referring to Table 1, we observe that for sequence A, the ETC, Shannon Entropy, and Gini impurity 77 all have a value of zero. This outcome arises from the fact that the sequence is homogeneous, devoid 78 of any impurity. Conversely, for sequences B, C, D, and E, the Shannon entropy and Gini impurity 79 remain constant, while ETC varies based on the structural characteristics of each sequence. Having 80 shown that the ETC captures the structural impurity of a sequence, we now define ETC Gain . ETC 81

gain is the reduction in ETC caused by partioning the data instances according to a particular attribute 82

of the dataset. Consider the decision tree structure provided in Figure 1. 83

Parent

Left Child

Right Child

The ETC Gain for the chosen parent attribute of the tree is defined as follows: 84

86

tively. The formula for ETC Gain, as given in equation 1, bears resemblance to information gain. The

87

key distinction lies in the use of ETC instead of Shannon entropy in the calculation. We now provide

the different steps in the Permutation Decision Tree algorithm. 88

89

90

Step 1: Choose an attribute to be the root node and create branches corresponding to each possible value of the attribute.

91

92

93

94

95

96

97

98

Step 2: Evaluate the quality of the split using ETC gain.

Step 3: Repeat Step 1 and Step 2 for all other attributes, recording the quality of split based on ETC gain.

Step 4: Select the partial tree with the highest ETC gain as a measure of quality.

Step 5: Iterate Steps 1 to 4 for each child node of the selected partial tree.

Step 6: If all instances at a node share the same classification (homogeneous class), stop developing that part of the tree.

3 Experiments and Results

To showcase the effectiveness of the ETC impurity measure in capturing the underlying structural 99 dependencies within the data and subsequently generating distinct decision trees for different permu100 tations of input data, we utilize the following illustrative toy example. 101

Table 2: Toy example dataset to showcase the potential of a permuted decision tree generated with a novel impurity measure known as 'Effort-To-Compress".

The visual representation of the toy example provided in Table 2 is represented in Figure 2 102

0

1

2

3

4

5

f1

0

1

2

3

4

f2

class-1

class-2

We consider the following permtation of dataset, for each of the below permutation we get distinct 103 decision tree. 104

105

106

Serial No. Permutation A: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Figure 3 represents the corresponding decision tree.

x

0

2

Class-2

x

0

4

x

1

2

Class-1

Class-2

Class-1

· Serial No Permutation B: 14, 3, 10, 12, 2, 4, 5, 11, 9, 8, 7, 1, 6, 13. Figure 4 represents the 107 corresponding decision tree. 108

x

1

2

x

0

4

Class-2

Class-1

x

0

2

Class-2

Class-1

· Serial No Permutation C: 13, 11, 8, 12, 7, 6, 4, 14, 10, 5, 2, 3, 1, 9. Figure 5 represents the 109 corresponding decision tree. 110

x

0

4

x

1

2

Class-2

Class-1

x

0

2

Class-2

Class-1

Serial No Permutation D: 3, 2, 13, 10, 11, 1, 4, 7, 6, 9, 8, 14, 5, 12. Figure 6 represents the 111 corresponding decision tree. 112

x

0

4

x

0

2

Class-2

Class-1

x

1

2

Class-2

Class-1

Serial No Permutation E: 10, 12, 1, 2, 13, 14, 8, 11, 4, 7, 9, 6, 5, 3. Figure 7 represents the 113 corresponding decision tree. 114

x

0

2

Class-2

x

1

2

x

0

4

Class-1

Class-2

Class-1

The variability in decision trees obtained from different permutations of data instances (Fig115 ures 3, 4, 5, 6,and 7) can be attributed to the ETC impurity function's ability to capture the 116 structural impurity of labels, which sets it apart from Shannon entropy and Gini impurity. Table 117 3 highlights the sensitivity of ETC to permutation, contrasting with the insensitivity of Shannon 118 entropy and Gini impurity towards data instance permutations. In the given toy example, there are six 119 class-1 data instances and eight class-2 data instances. Since Shannon entropy and Gini impurity are 120 probability-based methods, they remain invariant to label permutation. This sensitivity of ETC to 121 the structural pattern of the label motivates us to develop a bagging algorithm namely Permutation 122 Decision Forest. 123

3.1 Permutation Decision Forest

124

Permutation decision forest distinguishes itself from Random Forest by eliminating the need for 125 random subsampling of data and feature selection in order to generate distinct decision trees. Instead, 126 permutation decision forest achieves tree diversity through permutation of the data instances. The ac127 companying architecture diagram provided in Figure 8 illustrates the operational flow of permutation 128 decision forest. 129

Dataset

Permutation 1

Permutation 2

Permutation3

Permutation N

PDT 1

PDT 2

PDT 3

PDT N

Voting Scheme

Final Outcome

The architecture diagram depicted in Figure 8 showcases the workflow of the Permutation Decision 130

Forest, illustrating its functioning. Consisting of individual permutation decision trees, each tree 131

operates on a permuted dataset to construct a classification model, collectively forming a strong 132

classifier. The outcomes of the permutation decision trees are then fed into a voting scheme, where 133 the final predicted label is determined by majority votes. Notably, the key distinction between 134 the Permutation Decision Forest and Random Forest lies in their approaches to obtaining distinct 135 decision trees. While Random Forest relies on random subsampling and feature selection, Permutation 136 Decision Forest achieves diversity through permutation of the input data. This distinction is significant 137 as random feature selection in Random Forest may result in information loss, which is avoided in 138 Permutation Decision Forest. 139

140

3.2 Performance comparison between Random Forest and Permutation Decision Forest

141

142

143

144

145

We evaluate the performance of the proposed method with the following datasets: Iris [16], Breast Cancer Wisconsin [17], Haberman's Survival [18], Ionosphere [19], Seeds [20], Wine [21]. For all datasets, we allocate 80% of the data for training and reserve the remaining 20% for testing. Table 4 provides a comparison of the hyperparameters used and the test data performance as measured by macro F1-score.

In our experimental evaluations, we observed that the proposed method surpasses Random Forest 146 (F1-score = 0.56) solely for the Haberman's survival dataset (F1-score = 0.621). However, for the 147 Seeds dataset, the permutation decision forest yields comparable performance to Random Forest 148 (F1-score = 0.877). In the remaining cases, Random Forest outperforms the proposed method. 149

150

151

152

153

154

155

156

157

158

159

4 Limitations

The current framework demonstrates that the proposed method, permutation decision forest, achieves slightly lower classification scores compared to random forest. We acknowledge this limitation and aim to address it in our future work by conducting thorough testing on diverse publicly available datasets. It is important to note that permutation decision trees offer an advantage when dealing with datasets that possess a temporal order in the generation of data instances. In such scenarios, permutation decision trees can effectively capture the specific temporal ordering within the dataset. However, this use case has not been showcased in our present work. In our future endeavors, we intend to incorporate and explore this aspect more comprehensively.

5 Conclusion

In this research, we present a unique approach that unveils the interpretation of the Effort-to-Compress 160 (ETC) complexity measure as an impurity measure capable of capturing structural impurity in 161 timeseries data. Building upon this insight, we incorporate ETC into Decision Trees, resulting in the 162 introduction of the innovative Permutation Decision Tree . By leveraging permutation techniques, 163 Permutation Decision Tree facilitates the generation of distinct decision trees for varying permutations 164 of data instances. Inspired by this, we further develop a bagging method known as Permutation 165 Decision Forest , which harnesses the power of permutation decision trees. Moving forward, we are 166 committed to subjecting our proposed method to rigorous testing using diverse publicly available 167 datasets. Additionally, we envision the application of our method in detecting adversarial attacks. 168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

References

Jamie Murphy, Charles Hofacker, and Richard Mizerski. Primacy and recency effects on clicking behavior. Journal of computer-mediated communication , 11(2):522-535, 2006.

Nithin Nagaraj and Karthi Balasubramanian. Three perspectives on complexity: entropy, compression, subsymmetry. The European Physical Journal Special Topics , 226:3251-3272, 2017.

Nithin Nagaraj, Karthi Balasubramanian, and Sutirth Dey. A new complexity measure for time series analysis and classification. The European Physical Journal Special Topics , 222(3-4):847860, 2013.

Aditi Kathpalia and Nithin Nagaraj. Data-based intervention approach for complexity-causality measure. PeerJ Computer Science , 5:e196, 2019.

SY Pranay and Nithin Nagaraj. Causal discovery using compression-complexity measures. Journal of Biomedical Informatics , 117:103724, 2021.

Vikram Ramanan, Nikhil A Baraiya, and SR Chakravarthy. Detection and identification of nature of mutual synchronization for low-and high-frequency non-premixed syngas combustion dynamics. Nonlinear Dynamics , 108(2):1357-1370, 2022.

Aditi Kathpalia, Pouya Manshour, and Milan Paluš. Compression complexity with ordinal patterns for robust causal inference in irregularly sampled time series. Scientific Reports , 12(1):1-14, 2022.

Harikrishnan NB, Aditi Kathpalia, and Nithin Nagaraj. Causality preserving chaotic transformation and classification using neurochaos learning. Advances in Neural Information Processing Systems , 35:2046-2058, 2022.

Karthi Balasubramanian, K Harikumar, Nithin Nagaraj, and Sandipan Pati. Vagus nerve stimulation modulates complexity of heart rate variability differently during sleep and wakefulness. Annals of Indian Academy of Neurology , 20(4):403, 2017.

Vasilios K Kimiskidis, Christos Koutlis, Alkiviadis Tsimpiris, Reetta Kälviäinen, Philippe Ryvlin, and Dimitris Kugiumtzis. Transcranial magnetic stimulation combined with eeg reveals covert states of elevated excitability in the human epileptic brain. International journal of neural systems , 25(05):1550018, 2015.

Abhishek Nandekar, Preeth Khona, MB Rajani, Anindya Sinha, and Nithin Nagaraj. Causal analysis of carnatic music compositions. In 2021 IEEE International Conference on Electronics, Computing and Communication Technologies (CONECCT) , pages 1-6. IEEE, 2021.

Werner Ebeling and Miguel A Jiménez-Montaño. On grammars, complexity, and information measures of biological macromolecules. Mathematical Biosciences , 52(1-2):53-71, 1980.

Miguel A Jiménez-Montaño, Werner Ebeling, Thomas Pohl, and Paul E Rapp. Entropy and complexity of finite sequences as fluctuating quantities. Biosystems , 64(1-3):23-32, 2002.

Dario Benedetto, Emanuele Caglioti, and Davide Gabrielli. Non-sequential recursive pair substitution: some rigorous results. Journal of Statistical Mechanics: Theory and Experiment , 2006(09):P09011, 2006.

Peter Grassberger. Data compression and entropy estimates by non-sequential recursive pair substitution. arXiv preprint physics/0207023 , 2002.

R. A. FISHER. The use of multiple measurements in taxonomic problems. Annals of Eugenics , 7(2):179-188, 1936.

WNick Street, William H Wolberg, and Olvi L Mangasarian. Nuclear feature extraction for breast tumor diagnosis. In Biomedical image processing and biomedical visualization , volume 1905, pages 861-870. SPIE, 1993.

Shelby J Haberman. The analysis of residuals in cross-classified tables. Biometrics , pages 215 205-220, 1973. 216

Vincent G Sigillito, Simon P Wing, Larrie V Hutton, and Kile B Baker. Classification of radar 217 returns from the ionosphere using neural networks. Johns Hopkins APL Technical Digest , 218 10(3):262-266, 1989. 219

Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. 220

Michele Forina, Riccardo Leardi, Armanino C, and Sergio Lanteri. PARVUS: An Extendable 221 Package of Programs for Data Exploration . 01 1998. 222

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed the permutation decision trees method, which uses Effort-To-Compress as the impurity measure to model the order dependencies of data instances, and extended the proposed permutation decision tree to a variant of random forest. They also did some experiments to compare the performance of the proposed methods with random forests.\n\nSTRENGTHS:\nThe proposed structural impurity can actually capture the order dependencies of data instances, as shown in the examples in Table 1.\n\nWEAKNESSES:\nThe paper exhibits several weaknesses, which are outlined below:\n\n1. Insufficient clarity regarding the chosen setting: The authors' intended focus appears to be on time series data; however, the task discussed pertains to multi-class classification, which is an i.i.d. setting. The authors are recommended to formalize the problem setup.\n2. Inconsistent use of notation: The paper demonstrates inconsistencies in notation usage. For instance, the features presented in Table 3 are denoted as $f_{1}, f_{2}$, whereas in Figures 3-7, they are represented as $x_{0}, x_{1}$.\n3. Unfair experimental setup and insignificant results: The experiment setup lacks fairness, and the obtained results do not exhibit statistical significance. In the only out-performing dataset, the random forest model employed only one tree, while the proposed permutation decision forest utilized five trees, indicating an apparent unfairness in the comparison. Furthermore, the hyperparameters' n_estimators vary across different datasets, which is deemed unreasonable.\n\nQUESTIONS:\n1. Which type of tasks does this paper care about? See Weaknesses #1.\n2. If the concerned tasks are order-sensitive, why can the data instances be casually permuted when generating a forest?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations in Section 4.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn traditional decision tree algorithms such as CART and C4.5, impurity measures are used to split internal nodes. The paper proposes a decision tree induction method by using effort-to-compress (ETC) measure, which can capture order dependencies in the data. With ETC’s ability to capture order dependencies, permuting the data can result in different trees, thereby constructing a forest without the need for bagging. This proposed decision tree induction method can be used for datasets with temporal order.\n\nSTRENGTHS:\nThe paper uses ETC as a new impurity measure for constructing decision trees. Since ETC is sensitive to the order of data points, the tree built using this measure may be well-suited for temporal datasets. And it also provides a different way for constructing diverse trees and thereby getting a forest. Overall, the paper is clearly written and easy to follow.\n\nWEAKNESSES:\n- What about the bias and variance in the permutation decision forest? Random forest uses bagging and random feature selection to make trees in the forest uncorrelated, thereby reducing variance. But trees in the permutation forest are not uncorrelated. Using the ensemble of these correlated trees may not reduce variance. \n\n- In the toy example, some leaf paths are shown in different trees. I am wondering if there will be a significant number of duplicated leaf paths within the permutation forest. \n\n- Experiments only show the comparison between random forest and permutation tree forest in terms of F1-score. How about other evaluation metrics, e.g. misclassification loss? The results don’t show that the proposed method outperforms random forest. And there is no comparison between the performance of single trees, such as CART vs. ETC tree.\n\nQUESTIONS:\nI am wondering how the hyperparameters are chosen in the experiments. Random forest tends to have lower depth but more estimators, while permutation tree forest has deeper trees with fewer estimators.\n\nLIMITATIONS:\nThe authors have addressed the limitations of the paper and propose future directions.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper \"Permutation Decision Trees using Structural Impurity\" introduces a novel split criterion for the training of decision trees that also takes the order of labels inside the training data into account. This way, to obtain a forest, one only needs to shuffle the data before training individual trees. Moreover, the novel split criterion supposedly works better for data that includes (temporal) dependencies, although der are no experiments to support this claim.\n\nSTRENGTHS:\n- I think the idea of tackling non-iid data with a novel split criterion is nice, and Effort-To-Compress as an impurity measure seems like a good choice\n\nWEAKNESSES:\n- The experimental evaluation is very weak. The authors compare their method on 6 small real-world datasets and one artificial dataset and compare it only against Random Forests. Moreover, their method seems to be worse compared to RF. Hyperparameters are also incomparable, as the RF uses smaller trees than their method although it is well-known that RF benefits more from larger trees. In addition, the number of estimators changes for every experiment. There is no clear experimental protocol, and the authors do not use random repetitions and/or cross-validation but resort to a single train/test split. The experimental evaluation is hence borderline useless and can only be seen as a first test-experiment.  \n- The paper contains limited valuable information. While the Effort-To-Compress (ETC) measure seems to be of central interest here, the authors do not present a formal mathematical explanation of it. They mention the NSRPS algorithm to compute ETC, but also do not explain it mathematically, and only offer a single example. A thorough mathematical explanation and the typical explanations of the notation (Model function f(x), samples X, labels Y, etc.) is missing entirely. \n- The authors deal with the case in which the order of samples is important. This is completely against the typical IID assumption we have in Machine Learning. Unfortunately, the authors neither discuss this (certainly interstring) difference in more detail nor do they really present any real-world example of this. \n- A dedicated Related Work section is missing, although there is plenty of space left in the paper. The authors decided to waste roughly two pages by printing different DTs, which does not add any new information to the paper. This space would have been used better to highlight related work or pinpoint the novelty of this work in more detail. \n- Eq. (1) and Tab. 4 do not fit the page width\n\nQUESTIONS:\n- Are there any datasets and/or real-world applications you are aware of in which the order of samples matter? What are the implications for \"classical\" IID ML here?\n\nLIMITATIONS:\nThe authors acknowledge that their method is worse compared to the state of the art and intend to perform more testing. As this paper presents a novel method I don't see any immediate negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel in Decision Tree literature splitting criteria based on Effort To Compress (ETC) gain. Use of this criteria is justified by a desire to work with data that doesn't conform to i.i.d. assumption about the generating distribution. There is an experiment on a synthetic data that shows that different decision trees are generated when different orderings of the data are used for training. A permutation voting forest is introduced, that allows using random permutations of full data to obtain multiple different decision trees for use in the final ensemble of trees. There is an evaluation of Permutation Voting Forest against regular random forests on multiple real world datasets that however show slightly lower results when using proposed method.\n\nSTRENGTHS:\n- The paper opens a novel line of research about using Decision Trees for modeling data, that comes in a sequence and does not follow i.i.d. assumption. \n- There is a novel application of ETC measure as a splitting criteria in decision trees.\n- A generalization of the proposed model: Permutation Decision Forest is introduced, that uses a novel idea of shuffling the data in the context of a splitting criteria that generates different trees for different permutations of the training data.\n\nWEAKNESSES:\n- It is noted that usage of ETC allows to get rid of i.i.d. assumption. But this claim needs more thorough theoretical analysis. If we want to keep sequential structure of the data, the sequence still gets destroyed upon split: split does not split examples in a consecutive way; some examples may go to the left split, then some to the right, then again to the left part of the split and so on. So, new left and right sequences after the split will have completely different properties. Considering an example from introduction, where ETC is used: musical compositions. Splitting the musical composition according to some feature, like presence of some range of frequencies at a given moment, will result in an unpleasant music on both sides of the split, because instead of hearing half of the musical composition, we will hear a \"fractal\" - small parts of original sequence with small gaps inbetween, that got assigned to left or right parts of the split\n- Related to the previous point: at the testing phase there is no \"memory\" in the model, and the model still predicts elements by looking at them one by one. So, shuffling the testing set will result in exactly same predictions. Can we say that the problem of non-i.i.d. distribution is solved, if the behavior on the testing set is equal to the behaviour of the i.i.d. models?\n- Testing of regular Decision Trees with proposed splitting criteria on real data is needed (only the forests were tested on real data, but proposed forests work differently due to the proposed shuffling of the input data, so regular trees must be evaluated separately as well). It would be nice to both test on regular datasets (that are not sequentially ordered, like the datasets from section 3.2), and also to find at least some example real datasets where ordering is important, and where proposed model (regular permutation decision tree) is both practically and theoretically better than the baseline decision tree models.\n- In section 3.2 a more thorough experimental design would be more convincing. (a) If we compare proposed model to the baseline, why hyperparameters are different for same dataset? If hyperparameters tuning was done, that should be thoroughly described. (b) Experiments should be run several times on different train-test splits and mean scores and standard deviations should be reported to allow fair comparison in the presense of noise.\n\nQUESTIONS:\n- The proposed decision tree and forest work very differently, because forest does internal data shuffling, which destroys ordering of the original input data. Shall we consider two variants of the forest: one with shuffling, and one without (where ETC splitting is used, but instead of shuffling we do regular feature and example subsets selection)? Then the forest with shuffling may be compared to original Random Forest (because both are not using sequential information); and the Forest without shuffling may be compared to the single proposed permutation decision tree.\n\n- Minor note: typo on line 41: \"muscial\"\n\nLIMITATIONS:\nLimitations are well described in the paper, which is good. Since there were identified significant limitations in terms of accuracy of the proposed permutation decision forests (that may also affect proposed single permutation decision trees), it would be more convincing to include additional experiments that will clarify the extent of such limitations right away without deferring them to the future work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}