# Framing Algorithm-Driven Development of Sets of Objectives Using Elementary Interactions
## 5 Functional Core Component Validation Study
### Results  
In total, 2200 questions were answered, and 12 iterations were performed at a frequency of two iterations a week. The SOO achieved after 12 iterations are depicted in Figure 8. Although the SOO apparently appears to be incomplete, similarities to further SOOs for assessing the sustainability of water infrastructures, (e.g., [46]) are observable. The apparent incompleteness applies to objectives, criteria, and indicators and is reflected in unmet validity and stability measures, missing indicators, or unassigned criteria. A bottom-up approach was used to derive the objectives from the criteria. In the workflow of the study, the criteria were determined first. Then, with the help of the EI type 7, _Common Name_, an attempt was made to derive an objective from various criteria. This attempt had, within two iterations, no results.  
Due to the limited time of the study, the objective, _ecological objectives_, was provided to the SOO model manually, so that criteria could be assigned. Further gaps, (e.g., no indicators for the four criteria in the right columns of Figure 7) in the SOO are due to the pursuit of specific research aspects, since only a finite number of EIs (limited by the number of participants) could be answered per iteration.  
Additionally, the study allowed for experiences with validity measures. Heuristically, criteria validation requires a confirmation rate of 75% or more, and at least 10 answers. This validity measure was applied for criteria and indicators and helped to identify the SOO depictured in Figure 8.  
**Challenges.** Various challenges were observed during the study, which are described hereafter.  
_Unclear names_. Challenges occurred when the names of the criteria were not clear. As a result, some participants were overwhelmed by the EI type 2, _Confirm_, and were unable to answer the question. Two measures have been taken. Firstly, the option \"I don't know\" was added. Secondly, definitions for a criterion were requested; a criterion cannot be defined  
Figure 8: Validation study result: preliminary set of objectives (SOO).  
only by supplying a name. Using the EI type 4, _Choose set-based_, the participants were then able to determine a suitable definition for a criterion.  
_Stakeholder-specific questions._ All participants received the same questions without any differentiation according to the stakeholder group to which they belonged. According to informal feedback, domain-specific questions were too demanding for some participants, especially those who identified themselves as end-users. Therefore, the participants should be assigned to stakeholder groups and the questions should be stakeholder group-specific. Further, some participants complained about the repetition of the questions.  
_Question designs._ Various question designs were used in the study. For the EI _Identify duplicates_, for example, in addition to the original Yes/No variant, the question of how far both elements overlap on a 7-point Likert scale was also raised. No clear results could be found, it seems that participant-type dependent preferences exist.  
**Dependencies of EI types.** In general, dependencies between the uses of the EIs were clarified and confirmed in the validation study, such as that only after a criterion is confirmed, can participants be asked for possible indicators for that criterion. In consequence, the description schema of EIs requires the naming of prerequisites for the application of the EI. Further, state models, (e.g., containing the states _Criterion suggested_, _Criterion confirmed_, _Criterion rejected_) and state transition diagrams for SOO elements would be beneficial for the platform concept description.