## 7 Experimental Evaluation
### Streaming ML vs. Batch ML/DL  
The objective of this section is to compare the proposed streaming ML approach against three common batch-based scenarios, representing how data engineers would train and update their batch models in production. The difference between the scenarios lies in how engineers perform the updates over time, where one would: (1 st) train once but never update the model; (2 nd) periodically retrain the model from scratch; and (3 rd) retrain and incrementally update the model. To generate the scenarios, we split the data of each trace into ten equi-size segments (or _buckets_). For illustration, the buckets could represent different periods, e.g., ten consecutive hours. For the **1 st scenario**, where the engineer would train once but never update the model, we train the batch model with the first bucket and test with the remaining buckets. For the **2 nd scenario**, where the engineer would periodically retrain the model from scratch, we first train with the data of bucket #1 and test on the data of bucket #2. Then, we train a new model with the data of bucket #2 and test with the data of bucket #3, and so on. Thus, we train a model with bucket \(i\) and test with bucket \(i+1\), without updating the model or aggregating the training data. For the **3 rd scenario**, we train incrementally with the data from all the previous buckets and update the model. Thus, we first train a model with bucket #1 and test with bucket #2. Then, we retrain a model with the aggregated buckets #1 and #2 and test on bucket #3, and so on. Thus, we train a model with buckets #1 to \(i\) and test with bucket \(i+1\). For the streaming scenario, prequential evaluation is used to test and train the model, with the  
Figure 6: Original and predicted _Next Offset_ values for Trace 11 with optimized configuration over selected features. Figure 7: Original and predicted _Next File Hotness_ values for Trace 9 with optimized configuration over selected features. results averaged and reported per bucket for a fair comparison. The same training are used in all scenarios, containing the selected features described in Section 4. We use Weka [71] (v3.8.6) to create batch ML models that correspond to our streaming models, namely _Random Forest Regressor_ (_RFR_) for regression and _Decision Tree J48_ (_DT_) for classification. These two batch models are used in prior related work that proposed ML-based cache policies [33], while the comparison between these streaming and batch-based algorithms has been performed in other domains as well [56; 58]. Hyperparameter tuning was performed for all models, and the selected parameters are listed in Table 6. These experiments are run on a server with a 24-core 2.2 GHz CPU and 128 GB RAM. For each scenario, we collected the runtime and MAE/F1 score (for regression/classification) for every bucket used to test the model, leading to nine data points. Figure 8 presents the average F1 score and Figure 9 presents the total runtime (for testing and training) for all test buckets when predicting the Next Offset Class for the batch DT and streaming scenarios. The 1 st scenario has the worst F1 score across most traces because the workload access patterns in many traces change drastically over time. For example, the 1 st scenario achieves very low F1 scores for Trace 4. This trace begins with sequences of small random read operations and then switches to longer sequential read and write operations, driving the model to wrongly predict many sequential offsets as random. The 2 nd and 3 rd scenarios achieve much better F1 scores as they are (re) trained over time but at the expense of being, on average, 4\(\times\) and 13\(\times\) slower than the 1 st scenario, respectively. The 3 rd scenario exhibits a slightly better F1 score than the 2 nd scenario in some cases but at the cost being 3.8\(\times\) slower than the 2 nd scenario due to the increased training data size for each subsequent bucket.