# Evaluation Methodology for Circular and Resilient Information Systems
## 3 Results
### CRIS System evaluation of functionality performance  
To evaluate the system as a whole, various qualitative and quantitative approaches will be utilised. Starting from mathematical modelling approaches, availability and reliability will be evaluated. **Reliability:** This KPI relates to the likelihood that a system or system component is functional at a specific moment in time under a particular set of environmental conditions. The formula to calculate the KPI is as follows:  
\[R(t)=\mathrm{P}(\mathrm{T}>\mathrm{t})=1-F(t) \tag{16}\]  
where \(R(t)\) is the reliability, \(F(t)\) is the failure probability, and \(\mathrm{T}\) is the time to failure. **Availability**: This KPI relates to a component's/system's readiness for immediate use at any given time. The formula to calculate the KPI is as follows:  
\[\mathrm{Availability}=\mathrm{Total\ time}-\mathrm{Downtime}/\mathrm{Full\ time}\times 100\% \tag{17}\]  
Extending the overarching CRIS system-wide KPIs, each module, respectively, will be evaluated in specific metrics that reflect its scope of application and expected outcomes, as presented in Table 2.  
Extending the evaluation of the CRIS relevant to its performance and the performance of comprising modules, the evaluation methodology also includes dimensions that relate to \begin{table}
\begin{tabular}{c c c} \hline \hline
**Component** & **Supporting Processes** & **KPIs** \\ \hline \multirow{6}{*}{Dashboard/UI} & Viewing of VN-specific KPIs. & \multirow{6}{*}{Number of users designing their own Dashboard.} \\  & Adding, deleting users, etc. & \\ \cline{1-1}  & Execution of 3 rd-party services. & \\ \cline{1-1}  & Monitoring of internal processes. & \\ \cline{1-1}  & Monitoring of value network. & \\ \hline \multirow{6}{*}{\begin{tabular}{c} Sustainability \\ Balanced \\ Scoreccard \\ \end{tabular} } & Viewing of current status relevant to sustainability. & \multirow{6}{*}{\begin{tabular}{c} The number of metrics in terms of environment, \\ society, economy, growth, governance, and \\ pilot-specific pillars will be determined according \\ to pilots’ preferences per use case. \\ \end{tabular} } \\ \hline \multirow{6}{*}{\begin{tabular}{c} Analytics \\ Service \\ \end{tabular} } & Pilot specific for energy consumption & \multirow{6}{*}{\begin{tabular}{c} Reliability of the service to perform real-time \\ analysis when new data are incorporated into the \\ system. Forecasting accuracy (mean absolute error, \\ root mean squared error, bias), anomaly detection \\ precision/recall/F1 score (perceived ability of the \\ system to correctly identify anomalies), system \\ performance (processing time per dataset, system \\ throughput). \\ \hline \multirow{2}{*}{\begin{tabular}{c} Blockchain \\ \end{tabular} } & Pilot specific for collaboration establishment & Number of hashed contracts processed, transaction \\  & support in a trustworthy environment & latency. \\ \hline \multirow{2}{*}{Digital Twins Modeler} & Creation and updating of Digital Twins. & Number of Digital Twins modelled. \\ \hline \multirow{2}{*}{Value Network Modeler} & Creating, viewing, and acting on the value & \multirow{2}{*}{\begin{tabular}{c} Number of companies introduced in the value \\ network; number of companies depicted in the \\ value network. \\ \end{tabular} } \\  & \begin{tabular}{c} Pilot specific for creation of optimisation of the \\ production planning processes. \\ \end{tabular} & \multirow{2}{*}{\begin{tabular}{c} Number of production plans generated. \\ produced. \\ \end{tabular} } \\ \hline \multirow{2}{*}{\begin{tabular}{c} Guidelines for \\ Certification \\ \end{tabular} } & Evaluation of the CRIS systems’ compliance for & \multirow{2}{*}{\begin{tabular}{c} Number of guidelines per value network \\ produced. \\ \end{tabular} } \\  & \
\begin{tabular}{c} Development of the process models introduced \\ in the CRIS. \\ \end{tabular} & \multirow{2}{*}{
\begin{tabular}{c} Number of control variables for the entire value \\ chain. \\ \end{tabular} } \\ \hline \hline \end{tabular}
\end{table}