## 3 Materials and Methods
### Workflow Scheduling  
The prediction model described in Section 3.2 can be exploited to forecast the amount of memory that will be needed to execute a given task on a target computing node and its duration, based on the task features listed in Section 3.1. These predictions are then used within the scheduling strategy described in the following, whose goal is to avoid swapping to disk due to memory saturation in order to improve application performance and makespan through a better use of in-memory computing. The results discussed below refer to a static scheduling problem, as the scheduling plan is generated before the execution. In typical static scheduling, the workflow system has to predict the execution load of each task accurately, using heuristic-based methods [21]. Likewise, in the proposed method, the execution load of each task of a given workflow is predicted by the model trained on past executions. Moreover, we investigated how workflow tasks can be scheduled and run on a single computing node, but this approach can be easily generalized to a multi-node scenario. For example, a data-intensive workflow can be decomposed into multiple sub-workflows to be run on different computing nodes according to their features and data locality. Each sub-workflow is scheduled locally to the assigned node using the proposed strategy. In the IIWM, we modeled the scheduling problem as an offline Bin Packing (BP). This is a well-known problem, widely used for resource and task management or scheduling, such as load balancing in mobile cloud computing architectures [22], energy-efficient execution of data-intensive applications in clouds [23], DAGs' real-time scheduling in heterogeneous clusters [24], and task scheduling in multiprocessor environments [25]. Its classical formulation is as follows [26]. Let \(n\) be the number of items, \(w_{j}\) the weight of the \(j\)-th item, and \(c\) the capacity of each bin: the goal is to assign each item to a bin without exceeding the capacity \(c\) and minimizing the number of used bins. The problem is \(\mathcal{NP}\)-complete, and much effort went into finding fast algorithms with near-optimal solutions. We adapted the classical problem to our purposes as follows:  
Figure 2: Meta learner regression estimates for the different target variables. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **RMSE** & **MAE** & **Adjusted \(R^{2}\)** & **Pearson Correlation** \\ \hline Storage Memory & 108.23 & 26.66 & 0.96 & 0.98 \\ Execution Memory & 312.60 & 26.30 & 0.91 & 0.95 \\ Duration & 4443.17 & 2003.70 & 0.95 & 0.98 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Evaluation metrics on the test set. * An item is a task to be executed. * A bin identifies a stage, i.e., a set of tasks that can be run in parallel. * The capacity of a bin is the maximum amount \(C\) of available memory in a computing node. When assigning a task to a stage \(s\in\mathcal{S}\), its residual available memory is indicated with \(C_{s}\). * The weight of an item is the memory occupancy estimated by the prediction model. In the case of the Spark testbed, it is the maximum of the execution and storage memory, in order to model a peak in the unified memory. As concerns the estimated execution time, it is used for selecting the stage to be assigned when memory constraints hold for multiple stages. With respect to the classical BP problem, two changes were introduced:
* All workflow tasks have to be executed, so the capacity of a stage may still be exceeded if a task takes up more memory than the available one. * The assignment of a task \(t\) to a stage \(s\) is subject to dependency constraints. Hence, if a dependency exists between \(t_{i}\) and \(t_{j}\), then the stage of \(t_{i}\) has to be executed before the one of \(t_{j}\). To solve the BP problem, modeled as described above, in order to produce the final scheduling plan, we used the first fit decreasing algorithm, which assigns tasks sorted in non-increasing order of weight.