## 4 Scheduling Policy
### The Model-Similarity-Based Scheduling Policy  
The model-similarity-based scheduling policy employs a similarity measurement approach. This measurement is computed by comparing a job against reference jobs  
\begin{table}
\begin{tabular}{c l} \hline \hline
**Notations** & **Definitions** \\ \hline \(j\) & A current job \\ \(J\) & Active jobs in the queue \\ \(c_{i}\) & Model configurations, \(i=1,\ldots,n\) \\ \(Sj\) & Jobs that are scheduled in current round \\ \(FJ\) & A set of jobs recorded in the system \\ \(ff\) & A specific recorded job in the system \\ \(G\) & GPU architectures in the cluster \\ \(n\_gpu\) & Number of total GPU availability \\ \(g\) & Specific GPU architecture \\ \(ng\) & Number of a specific GPU \\ \(m\) & The closest reference to the given model \(j\) \\ \(S\) & Results of computing similarity between a job \(j\) and jobs \(FJ\) \\ \(sch_{m}\) & Suggestion for scheduling a job \(j\) based on the closest reference to the given model \\ \(jp_{m}\) & Potential job combinations based on the closest reference to the given model \\ \(T\) & Total processing time of jobs \(J\) \\ \(arr\) & Arrival time \\ \(st\) & Start execution time \\ \(wl\) & Workload of GPU architecture \\ \hline \hline \end{tabular}
\end{table}
Table 1: Notations and their definitions.  
Figure 4: Throughput of sharing an Nvidia A100 GPU architecture for a VGG16 model and other models. The red line represents the baseline throughput for VGG16 training alone.  
recorded in a database. Numerous approaches exist to measure the similarity between objects, such as Euclidean distance [23], Manhattan distance [24], and cosine similarity [25]. Specifically, cosine similarity is applied to measure the similarity of the model in this context due to its suitability for handling multi-dimensional data. The model similarity is based on DL characteristics, such as the number of layers and model parameters.  
Each job is represented by a set of model characteristics. Let a job \(j\) be a tuple \((c_{1},c_{2},\dots,c_{n})\) of the model characteristics. The current job \(j\) is compared to other jobs \(\textit{fj}\in F\). The similarity of \((j,\textit{fj})\) is defined as the similarity of two objects in a multidimensional space. It is determined by the following:  
\[\begin{split} similarity(j,\textit{fj})&=cos( \theta)=\frac{j\cdot\textit{fj}}{\|j\|\|\textit{fj}\|}\\ &=\frac{\sum_{i=1}^{n}f_{i}\cdot\textit{fj}_{i}}{\sqrt{\sum_{i=1 }^{n}f_{i}^{2}\sqrt{\sum_{i=1}^{n}f_{i}^{2}}}}\\ \end{split} \tag{3}\]  
where _fj_ denotes a reference job recorded in the system that is currently compared to the current job.  
The procedure of computing the similarity between a job \(j\) and a set of reference jobs in the database \(F\)J is described in Algorithm 1.  
```
1:\(j\leftarrow(c_{1},c_{2},\dots,c_{n})\)
2:for each\(\textit{fj}\in F\)do
3:\(S_{(\textit{fj})}\leftarrow similarity(j,\textit{fj})\)
4:endfor
```  
**Algorithm 1** Algorithm of computing model similarity.  
After discovering the closest reference to the given model, a scheduler has the training information of the reference model on several GPU architectures. This information includes GPU architecture, the corresponding number of GPUs used in training, and the batch size. They are then organised in descending order of throughput and supplied to the scheduler to make a scheduling decision based on current cluster availability.