## 5 Prototypical Validation and Case Study
### Variability4 TOSCA: An Extension of the TOSCA Standard
#### 5.1.2 Extending TOSCA for VDMM
In addition, 1000 Relationship Templates and are generated, which connect one Node Templates and the _next_ Node Templates and which have a condition assigned, which evaluates to true. Furthermore, the 1000 Node Templates and are generated, which have a condition assigned that evaluates to false. Additional 1000 Relationship Templates are generated which connect the Node Templates with a condition assigned which evaluates to false. In total, 4000 templates are generated consisting of 2000 Node Templates and 2000 Relationship Templates. After resolving the variability, only the 1000 Node Templates along with the 1000 Relationship Templates are between them are

The time for transforming the Variability4 TOSCA model into an internal data structure, resolving the variability, running the consistency check, and transforming the Topology Template into a TOSCA Simple Profile in YAML Version 1.3 model is measured. Thereby, the Variability4 TOSCA model is already loaded as JSON in memory. To remove outliers, the median of 10 test runs is used. The benchmark results of all in-memory tests are given in Table 1 (see tests 1-7). A Topology Template consisting of 40 templates (20 Node Templates and 20 Relationship Templates) is processed within 0.295 ms, a Topology Template consisting of 40,000 templates (20,000 Node Templates and 20,000 Relationship Templates)within 137.353 ms. As seen by the median per template column in Table 1 and Figure 6 on the left, the time used for processing scales slightly non-linear. In another test set, we additionally measured the time which is needed to read and write the Topology Template files which also includes YAML parsing. These tests take significantly longer mainly due to the parsing of YAML. The benchmark results of these tests are given in Table 1 (see tests 8-14). A Topology Template consisting of 40 templates is processed within 3.335 ms, a Topology Template consisting of 40,000 templates within 11.345 s. The file of the Topology Template consisting of 400,000 templates has a size of 14.270 MB and consists of 350,010 lines. As a result, the processing scales worse (see Table 1 and Figure 6 on the right). Such Topology Templates are already huge, possibly representing a single deployment consisting of thousands of servers and their hosted components along with databases. The required time for processing is negligible in comparison to the time required to actually provision such huge topologies. Based on our experience, creating, for example, a MySQL5 DBMS on GCP takes several minutes. The tests have been executed on commodity hardware; to be specific, on Windows 10, an i7-6700K CPU 4x 4.00 GHz, 2133 MHz DDR4 RAM, and NVMe M.2 SSD. Due to the single-threaded nature of NodeJs, only a single core is used. The performance could be further improved by using better hardware, threading, or a more efficient language, such as Go. Refactoring the algorithm could also improve the performance. However, at the current stage, we prefer understandability over performance. \begin{table}
\begin{tabular}{l r r r r r r} \hline \hline
**Test** & **Seed** & **Templates** & **Median** & **Median/Template** & **File Size** & **File Lines** \\ \hline
1 & 10 & 40 & 0.295 ms & 0.007 ms & n/a & n/a \\
2 & 250 & 1000 & 4.045 ms & 0.004 ms & n/a & n/a \\
3 & 500 & 2000 & 4.329 ms & 0.002 ms & n/a & n/a \\
4 & 1000 & 4000 & 7.383 ms & 0.002 ms & n/a & n/a \\
5 & 2500 & 10,000 & 29.436 ms & 0.003 ms & n/a & n/a \\
6 & 5000 & 20,000 & 63.307 ms & 0.003 ms & n/a & n/a \\
7 & 10,000 & 40,000 & 137.353 ms & 0.003 ms & n/a & n/a \\ \hline
8 & 10 & 40 & 3.335 ms & 0.083 ms & 14 kB & 360 \\
9 & 250 & 1000 & 39.033 ms & 0.039 ms & 351 kB & 8760 \\
10 & 500 & 2000 & 88.504 ms & 0.044 ms & 704 kB & 17,510 \\
11 & 1000 & 4000 & 225.444 ms & 0.056 ms & 1.409 MB & 35,010 \\
12 & 2500 & 10,000 & 947.011 ms & 0.095 ms & 3.553 MB & 87,510 \\
13 & 5000 & 20,000 & 3.185 s & 0.159 ms & 7.125 MB & 175,010 \\
14 & 10,000 & 40,000 & 11.345 s & 0.284 ms & 14.270 MB & 350,010 \\ \hline \hline \end{tabular}
\end{table}