# Evaluation Methodology for Circular and Resilient Information Systems
## 4 Discussion  
This research aims to identify and form an evaluation approach for information systems based on emerging technologies, particularly circular and resilient information systems. In the course of the CRIS design and through the involvement of the pilots in the requirement elicitation, a series of evaluation criteria were identified that will enable the validation of the system post-application. Initially, it was identified that a CRIS should be evaluated in a two-fold manner, i.e., its capacity to assist the pilots in their utilisation of SRM through their collaboration in the new circular supply chain (pilot KPIs' evaluation), as well as its capacity to constantly and consistently perform its tasks (technical evaluation). Relevant to the evaluation of the pilots' KPIs, a series of metrics were identified that the system should monitor, evaluate, and present to the user. These KPIs were inherently different from pilot to pilot, as the production contexts (and SRM) were different in nature. In particular, for CFRP waste, the identified KPIs related to the optimisation of the SRM (prepreg) requalification process and its subsequent utilisation will lead to fewer scraps being discarded as they are used in new composite materials. Therefore, it was identifiedthat both in-factory as well as network-wide KPIs should be utilised by the CRIS to evaluate the overall effectiveness of its application for the pilots in the circular supply chain. For WEEE for magnets, the identification KPIs related to the optimisation of SRM (bonded NdFeB, Sr-ferrite) use in new magnet production. Similar to the previous pilot, both in-factory and network-wide KPIs should be utilised by the CRIS. Lastly, in citrus processing waste for juice by-products, the identified KPIs related to the optimisation of the process of the transformation of the SRM (e.g., molasses, d-limonene, and COD), as well as the evaluation of the subsequent business-related outcomes (e.g., revenues from produced animal feed). In this case, the circularity potential is within the factory and, as such, the CRIS should only enable monitoring of KPIs that relate to a single partner rather than network-wide KPIs. It is thus evident that the CRIS system should be evaluated in (a) its ability to monitor varied KPIs relevant to pilot-specific contexts--in order to enable the factories to operate in a resilient manner--as well as (b) its ability to monitor collaboratively produced KPIs from different factories in an interconnected factory. The latter is in order to enable the factories of the network to operate in a circular manner and close the loop of SRM utilisation. Relevant to the technical evaluation of the CRIS system, results indicate that, as the system consists of various components and services that are naturally classified as established (e.g., simulation) as well as emerging (e.g., Digital Twins), the system should have a three-fold evaluation approach, i.e., component evaluation, system evaluation, and, lastly, user evaluation. For component evaluation, the examined components included the Dashboard, the Sustainability Balanced Scorecard, and the involved services (analytics, simulation, optimisation, and blockchain), as well as the services that enabled the formation of circular chains (value network modeller) and the twining of the individual factories (Digital Twins modeller), each of which has its own metrics that the evaluation will be based upon. For the integrated CRIS system, a number of evaluation approaches and metrics available from the established IS, RIS, and CIS evaluation literature were identified as applicable for CRIS evaluation, with the most prominent being the reliability and availability of the system.