# CMS: A Continuous Machine-Learning and Serving Platform for Industrial Big Data
## 5 Case Study and Experimental Results
### Performance Evaluation
#### 5.3.2 Multitenancy with Isolation  
The subsection tests the multitenancy performance with different types of resource isolation configurations. A LSTM model _trainer_ is used for the superheater temperature prediction and its corresponding _modelet_. This model is updated with comparably small amount of historical data, about 626 MB. The response time of the _modelet_ during a _trainer_' execution is collected. This test was conducted in a setting with four vCores and 144 GB memory accessible from the _trainer_ and _modelet_. Three different isolation configurations are tested: (1) The complete isolation puts the _trainer_ and the model server over the two different sets of vCores (two for each); (2) The no-isolation made no restriction on the CPU vCores. This means that the _trainer_ and server would compete for the four vCores. (3) In the partial isolation, the assigned CPU vCores of _trainer_ and server overlap with one vCore. Each one is assigned with three vCores.  
As shown in Figure 5a, the _trainer_ execution starts at about 15 min, indicated by the black dot line. For the complete isolation scene, the training process takes about 18 min and finished at about 33 mins, indicated by the green dashed line. Compared to the stage without training task, the CPU utilization increases about 138%. During the whole simulation period, the prediction delay keeps at about 102 ms, without significant changes. The complete isolation guarantees the exclusive CPU usage, so the prediction response time keeps at about 102 ms during the whole simulation period. With respect to the no-CPU isolation configuration, the prediction and training services are bound to exactly the same four CPU vCores. After the training service is started, the CPU usage increases dramatically to about 194%. This configuration achieves the least training time, about 13 min (the red dotted line). However, due to the CPU competition, the prediction delay was significantly increased by about 300 ms. In a partially isolated environment, the prediction and training services are bound to three CPU vCores with one overleaping. After the training service was started, the CPU occupancy rate was increased by about 180%, and the training time took about 15 min. After the model training is started, due to the duplicated CPU vCore's occupation, the prediction delay increases by about 28 ms. For the memory usage, the training service uses about 10% of memory (about 14 G) in all three scenarios, as no memory isolation is used.  
Figure 5: Average predictive delay and resource utilization in different isolation environments. (**a**) Prediction delay; (**b**) CPU Utilization; (**c**) Memory Utilization.  
As seen from simulation results, although the configuration with no isolation has the highest CPU usage and shortest model training time, it greatly increases the prediction delay. The complete isolation configuration achieves the best protection of the model prediction, however, at the cost of the lowest CPU usage and the longest training time. In comparison, the partial isolation configuration achieves a little increase in prediction time (28 ms) and almost the same CPU usage and training time as the no-isolation configuration. Thus, from this perspective, the partial isolation is the best. However, how to best assign the resources to the different types of services remains a challenge and is one of our future works.