## 6 Results and Discussion
### Comparison with the State of the Art  
We compare our scheduling policy with the state-of-the-art scheduling policies stated in Table 3.  
**Differences in scheduling decision making between our work and the state of the art.** We should consider the similarities and differences in scheduling decision making between our work and the state of the art.  
First, their scheduling decisions can be changed in the subsequent round. Their results can thus be compared with ours under the same conditions, which is the experiments with the \(cd\) label.  
Second, in a real cluster, the training performance remains unknown until a job initiates running for a few epochs, thus preventing the scheduler from making the optimal decision. In this work, we employ the model-similarity-based scheduling policy, which gives us an estimate to overcome this limitation. However, the previous works to which we compare provide the actual throughput recorded in advance to the scheduler, enabling the scheduler to make decisions based on predetermined information. Acting on perfect data that should not be known is not a realistic hypothesis and might skew the results in their favour.  
Third, their optimisation focuses on the throughput of a set of jobs. This implies that any jobs fitting the available resources can run, irrespective of their arrival time. This approach may impact job waiting times, as jobs can commence training and then be paused later. In contrast, our optimisation priorisises job arrival time to align with a typical real cluster.  
**Comparison of several evaluation metrics.** Figure 7 compares the makespan of our experiments to the state of the art in three scenarios of job density (range of arrival time 0-15, 0-7 minutes, and all jobs arrive simultaneously), with rescheduling every 3 min. Our work demonstrates superior performance, outperforming the state of the art in makespan across all scenarios. Remarkably, our work exhibits a significant decrease in makespan, particularly in scenarios where job arrivals span 0-7 min and when all jobs arrive simultaneously.  
Figure 8a-c compare job completion time, waiting time, and GPU utilisation of our experiments to the state of the art. Our work outperforms the state of the art across all evaluation metrics. A substantial improvement is observed in job completion time and GPU utilisation, while waiting time exhibits a modest improvement.  
The observed improvement in our work can be attributed to taking into account several influential factors regarding the training performance, including GPU architecture, the number of GPUs used, the DL model, and the DL hyperparameter (batch size). The state-of-the-art methods show awareness of the diverse training performance arising from the DL model and GPU architecture combined, but overlook the number of GPUs and batch size in making scheduling decisions.  
Figure 8: Comparison of several evaluation metrics of our experiments to the state of the art in the scenario that all jobs arrive simultaneously, with rescheduling every three minutes. The blue and green lines represent our experiments, that is, _similarity_gpu_ngpu_and _similarity_gpu_ngpu_bs_, respectively. The other lines represent the state of the art.  
Figure 7: Comparison of makespan of our experiments to the state of the art in three scenarios of job density with rescheduling every three minutes. The blue and green lines represent our experiments.