## 7 Experimental Evaluation
### Streaming ML vs. Batch ML/DL
In comparison, the streaming models exhibit almost the same high F1 scores as the 3 rd scenario across the traces. Interestingly, the 3 rd batch scenario achieves a higher F1 score for 7 out of the 17 traces (2.6% on average), while streaming achieves a higher F1 score for the other 10 traces (4.9% on average). Overall, streaming ML achieves a 1.8% higher F1 score while being, on average, 7.7\(\times\) faster than the 3 rd scenario, affirming its high performance in predicting the next offset efficiently. The batch experiments were repeated by replacing the RFR and DT algorithms with deep neural network models from the Weka DeepLearning4j (v1.7.2) framework [72]. In particular, we developed _Long Short-Term Memory_ (LSTM) _Recurrent Neural Network_ (RNN) models, as proposed in [17; 30], to make the same predictions. The RNN models were configured with 25 backward and 25 forward backpropagations through time, an LSTM layer with the Sigmoid gate activation function and the Rectified Linear Unit (ReLU) activation function, and an output layer with the softmax activation function and the multi-class, cross-entropy loss function (MCXENT) as an optimization objective. The Adam \begin{table}
\begin{tabular}{l c c} \hline \hline
**Decision Tree J48 (DT)** & **Next Offset Class** & **Next File** \\ \hline Pruning confidence & 0.10 & 0.20 \\ Num folds for reduced error pruning & 3 & 3 \\ Min number of instances per leaf & 2 & 2 \\ \hline
**Random Forest Regressor (RFR)** & **Next Offset** & **Next File** \\  & & **Hotness** \\ \hline Number of iterations & 40 & 30 \\ Size of each bag & 100 & 100 \\ Min variance for split & 0.001 & 0.001 \\ Min number of instances per leaf & 1 & 1 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Selected hyperparameter values for Decision Tree J48 (DT) and Random Forest Regressor (RFR) after hyperparameter tuning. optimizer, with a mean decay of 0.9 and a var decay of 0.999, was used as the updater, while Stochastic Gradient Descent (SGD), with a learning rate of 0.001, was used as the updater for the bias. Due to the excessive running times required for training the models (and the lack of a GPU), we present only the results from the first 12 traces. Note that lacking a GPU is a realistic testing environment since storage systems are typically not equipped with GPU devices and even lack significant CPU capabilities. Hence, even inference from deep-learning models can be prohibitively expensive in storage systems [47]. Figure 10 presents the average F1 score and Figure 11 the runtime when predicting the _Next Offset Class_ for the batch RNN and streaming scenarios. The trends and results for the F1 score are very similar to the ones produced using DTs and discussed above: the 1 st scenario can sometimes lead to very low prediction performance, while the 3 rd scenario offers the best predictions for the batch-based experiments. In general, we did not observe any significant differences in F1 scores between DTs and RNNs in our experiments. In terms of runtime, training and using RNNs are 2-3 orders of magnitude slower than using DTs and, on average, 4000\(\times\) slower than our streaming ML models, revealing that RNNs are impractically slow to be used within storage systems in production. Figure 8: F1 score for the three batch scenarios using DTs against streaming ML for the _Next Offset Class_. Figure 10: F1 score for the three batch scenarios using RNN (LSTM) against streaming ML for the _Next Offset Class_. Figure 9: Runtime for the three batch scenarios using DTs against streaming ML for the _Next Offset Class_. Figure 12 presents the variation of the F1 score along the buckets for Trace 8 when predicting the Next File Hotness Class. Note that each bucket corresponds to approximately 3.7 min as the duration of the entire trace is 37.2 min (recall Table 1). For the 1 st scenario, we observe a high variation in the F1 score, with a sudden drop of 20% from bucket #2 to bucket #3 (from 0.97 to 0.77) for both DT and RNN models.