## 6 Results and Discussion
### Comparison among Our Experiments  
For conciseness, the results of the different experiments presented are labelled with abbreviations. The experiments are over three allocation variables: GPU architecture, number of GPUs used in training, and batch size. They are abbreviated as _gpu_, _ngpu_, and _bs_, respectively. The experiments described in Section 5.2 are represented by _similarity_gpu_ (Experiment 1), _similarity_gpu_ngpu_ (Experiment 2), and _similarity_gpu_ngpu_bs_ (Experiment 3). Furthermore, the experiments in which the scheduling decisions can be changed for the next round are denoted as _cd_, and the experiments integrating job packing are denoted as _jp_. **Comparison of makespan on three different aspects. ** Figure 5 illustrates the makespan comparison across three key aspects of our experiments. These aspects include varying time per iteration (columns), three job density scenarios (ranges of arrival time in rows),and the increasing number of jobs (x-axis). The makespan generally rises due to two factors: an increased number of jobs necessitating more time for completion and the gradual delay of job arrivals. In the comparison of job density scenarios, the makespan slope varies across arrival time ranges. The makespan of _similarity_gpu_ remains consistently high while _similarity_gpu_ngpu_ and _similarity_gpu_ngpu_bs_ show comparable makespans, with a noticeable difference observed when all jobs arrive simultaneously (range of arrival time of 0 min). In this scenario, the makespan of _similarity_gpu_ngpu_bs_ is slightly lower than that of _similarity_gpu_ngpu_. In considering time per iteration for rescheduling, one can observe an increase in both makespan and time per iteration as the interval between rescheduling grows. This is due to potential GPU architecture idleness when a job finishes during a round, and rescheduling does not occur until the end of that round. In our study, rescheduling every three minutes yields the lowest makespan, although there is a slight gap between rescheduling every three and six minutes. **Improvement of implementing a round-based mechanism. ** Figure 6a-d compare various evaluation metrics between two experiments (_similarity_gpu_ngpu_ and _similarity_gpu_ngpu_bs_) in the scenario that all jobs arrive simultaneously, with rescheduling every three minutes. The results of the _similarity_gpu_ experiment are excluded from this figure, as they do not show any significant improvement. The lack of improvement arises from the scheduler focusing exclusively on selecting the appropriate GPU architecture for jobs while overlooking the impact of the number of GPUs used, which also playing a pivotal role in training performance. Figure 5: Comparison of makespan among experiments on three different aspects. Incorporating the model-similarity-based scheduling policy with a round-based mechanism, as illustrated in experiments _similarity_gpu_ngpu_cd_ and _similarity_gpu_ngpu_bs_cd_, results in a reduction in makespan and an increase in GPU utilisation compared to experiments without the allowance to change the scheduling decision. Furthermore, the average job completion time decreases, particularly with a certain number of jobs (25 and 50 jobs, in our case). Despite these advantages, there is a slight delay in the queue. The experiments of _similarity_gpu_ngpu_cd_ and _similarity_gpu_ngpu_bs_cd_ with 25 jobs show a reduction in makespan by approximately 33% and 29%, respectively, compared to their counterparts without the round-based mechanism that allows changing the scheduling decision in each round (i.e., without the _cd_ label). The improvement in GPU utilisation is approximately 24% and 26%, respectively, compared to their counterparts without the round-based mechanism. Also, the average job completion time also decreases by  
Figure 6: Comparison of several evaluation metrics among experiments in the scenario that all jobs arrive simultaneously with rescheduling every three minutes. The blue represents the experiment of _similarity_gpu_ngpu_.