# Demystifying MLOps and Presenting a Recipe for the Selection of Open-Source Tools
## 5 Tooling in MLOps
### 3.3 Model's Performance Monitoring (MPM)
#### 5.1.6 General Requirements  
Other than the ML-specific requirements, there are some additional elements that could play an important role in the selection of tools. In this section, we define four key points which are also used commonly for other software domains.  
GR-1: Scalability  
The scalability of supportive tools for data science projects is a basic requirement. Depending on the specific task, various aspects like the required processing power, storage amount, or service latency must be flexible.  
GR-2: APIs  
By controlling a tool via its served APIs, many tasks can be realized programmatically, potentially strengthening the robustness of the MLOps pipeline by automating specific tasks and eliminating errors of manually utilization or configuration of tools.  
\begin{table}
\begin{tabular}{p{142.3 pt} p{284.5 pt}} \hline \hline
**Metric** & **Remarks** \\ \hline Output distribution & The distribution shift of the predicted outputs between data from the training and production phase indicates model degradation. Metrics such as the Population stability index or Divergence index could help in alerting when such shifts happen. Especially in time-series-based datasets, this metric is essential in order to determine the credibility of existing models on the new data [19]. \\ \hline Model’s performance & This particular metric is dependent on the task at hand. It is useful for checking whether the current model is good enough or not. Examples are Auc score, precision, recall, FPR, TPR, and Confusion Matrix for classification tasks. Such metrics are difficult to calculate as there might be long latency between the output and true label. In general, a model’s performance can be evaluated by its accuracy (e.g., right predictions), gains (comparison to other models), and accreditation (credibility of resulting predictions) [19]. \\ \hline Feature importance change & The model’s performance might not degrade, but there could be a change in feature importance based on which the predictions are made. This could be an indication of drift and possibly a point where data should be collected again for retraining. \\ \hline Concept drift & Concept drift is a major source of model performance degradation. It represents the pattern which the model has learned in the past is not valid anymore. For a comparative discussion of active and passive concept drift detection in distributed environments, we refer to the work in [20]. \\ \hline Bias/Fairness & The trained model could be biased towards one class or group. Accuracy difference (AD) can be measured for each class to check for model bias. Another possibility is that the model can be evaluated on specific slices of data for obtaining an in-depth review of the model’s performance. \\ \hline Numerical stability & Numeric values that are invalid or wrong are possibly learned during model training. These do not directly trigger any explicit errors but can produce significantly wrong predictions in production. Checking for numerical stability ensures robustness. \\ \hline \hline \end{tabular}
\end{table}
Table 4: The different metrics that can be used for monitoring ML systems.  
GR-3: Tool Maturity Level  
The reliability of tools in production environments is a vital requirement, as errors may dramatically impact business goals. Therefore, a maturity assessment of potential tools (e.g., well-tested, widely spread utilization, etc.) should be carried out beforehand in any case.  
GR-4: User Friendliness  
As many actors are active within the MLOps workflow, non-technical personal must be supported by user-friendly tools (e.g., self-explaining or intuitive user interfaces).