# Forgetful Forests: Data Structures for Machine Learning on Streaming Data under Concept Drift
## 5 Results
### Datasets  
We use four real datasets and two synthetic datasets to test the performance of our forgetful methods against the state-of-the-art incremental algorithms. The forest cover type, phishing, and power supply datasets suffer from frequent and mild concept drifts, the electricity dataset suffers from frequent and drastic concept drift, and the two synthetic datasets suffer from gradual and abrupt concept drifts, respectively. Thus, our test datasets cover a variety of concept drift scenarios.  
* The forest cover type (ForestCover) [20] dataset captures images of forests for each \(30*30\) m cell determined from the US Forest Service (USFS) Region 2 Resource Information System (RIS) data. Each increment consists of \(400\) image observations. The task is to infer the forest type. This dataset suffers from concept drift because later increments have different mappings from input image to forest type than earlier ones. For this dataset, the accuracy first increases and then flattens out after the first \(24,000\) data items have been observed, out of \(581,102\) data items.
* The electricity [21] dataset describes the price and demand of electricity. The task is to forecast the price trend in the next \(30\) min. Each increment consists of data from one day. This data suffers from concept drift because of market and other external influences. For this dataset, the accuracy never stabilizes, so we start measuring accuracy after the first increment, which is after the first \(49\) data items have arrived, out of \(36,407\) data items.
* Phishing [22] contains \(11,055\) web pages accessed over time, some of which are malicious. The task is to predict which pages are malicious. Each increment consists of \(100\) pages. The tactics of phishing purveyors get more sophisticated over time, so this dataset suffers from concept drift. For this dataset, the accuracy flattens out after the first \(500\) data items have arrived.
* Power supply [23] contains three years of power supply records of an Italian electrical utility, comprising \(29,928\) data items. Each data item contains two features, which are the amount of power supplied from the main grid and the amount of power transformed from other grids. Each data item is labelled with the hour of the daywhen it was collected (from 0 to 23). The task is to predict the label from the power measurements. Concept drift arises because of season, weather, and the differences between working days and weekends. Each increment consists of 100 data items, and the accuracy flattens out after the first 1000 data items have arrived.
* The two synthetic datasets are from [16]. Both are time-ordered and are designed to suffer from concept drift over time. One, called **Gradual**, has \(41,000\) data points. Gradual is characterized by complete label changes that happen gradually over 1000 data points at three single points, and \(10,000\) data items between each concept drift. Another dataset, called **Abrupt**, has \(40,000\) data points. It undergoes complete label changes at three single points, with \(10,000\) data items between each concept drift. Each increment consists of 100 data points. Unlike the datasets that were used in Section 4, these datasets contain only four features, two of which are binary classes without noise, and the other two are sparse values generated by \(sin(x)\) and \(sin^{-1}(y)\), where \(x\) and \(y\) are the uniformly generated random numbers. For both datasets, the accuracy flattens out after 1000 data items have arrived.  
To measure the statistical stability in the face of the noise caused by the randomized setting of the initial seeds, we test all decision trees and random forests six times with different seeds and record the mean values with a 95% confidence interval for time consumption, accuracy, and F1 score.  
The following experiments are performed on an Intel Xeon Platinum 8268 24C 205W 2.9 GHz Processor with 200 gigabytes of memory, Intel, Santa Clara, CA, USA.