# Framing Algorithm-Driven Development of Sets of Objectives Using Elementary Interactions
## 5 Functional Core Component Validation Study
### Study Design  
The aim of the study is to validate the core components without having them developed in software. Consequently, the study was largely conducted manually. Figure 5 shows the data storage involved and the core component validation process. The first data storage is the pool of questions in the learning management system, Moodle [69], shown in Figure 5, item (2). The answers to the questions are additionally stored in Moodle. The SOO and supporting data, called the SOO model, are stored in a spreadsheet file, as shown in Figure 5, item (5). The steps of the iterative workflow employed in the validation study are:Step 1: Generating EIs: EIs are mainly derived from the current state of the SOO. This step is performed manually by the study leader. The current state of the SOO model is used as input. The output of this step is a set of questions that are used as EIs. In each turn, a set of questions was generated (typically 10-20) manually by the study leader. Step 2: The questions generated in Step 1 are added to a question pool provided by moodle. This step is performed manually by the study leader. Step 3: The questions are made available to the participants via the moodle _Test_ activity (Figure 6). Participants are then required to answer a minimum of 10 questions but can additionally answer as many questions as they wish to answer. Step 4: The answers to the questions are used to update the SOO model. To do this, the study leader extracts the answers to the questions from Moodle and aggregates them into the SOO model, following aggregation rules. The manual aggregation includes the correction of spelling errors, which might be more complex in an automated algorithm. Step 5: In Step 5, a new version of the SOO model will be made available. The new version of the SOO model is then the basis for the next iteration, i.e., the workflow is started again with Step 1. Step 6: The workflow, in particular Steps 1 and 4, was accompanied by supervision aimed at improving the workflow. After each iteration, a decision is made about whether changes should be made in Steps 1 and 4, especially regarding the rules to generate EIs, and the rules for aggregating the answers into the SOO model. While the platform concept employs the concept of a continuous EI stream and model aggregation, for practical reasons, the study followed an iterative workflow. The iterative approach allows the algorithms for generating EIs and for model aggregation to be applied to a larger number of EIs at a time. This is useful for the manual handling of these steps, while software-based automated processing allows continuous processing after each EI  
Figure 5: Workflow of core component validation. Figure 6: Example screenshot of the interface for conducting elementary interactions provisionally as part of the case study (moodle _Test_ activity). The SOO model updated in Step 5 was then used as the baseline for generating a set of EIs for the next round. Figure 7 shows an excerpt of the SOO model's suggested criteria and their validity measures. The underlying spreadsheet is available as a digital supplement (Table S1) to this article. For the study, the assessment goal of _sustainability of water infrastructure_ was chosen, as many potential participants had at least basic expertise in this field. Additionally, the authors were involved in the development of another SOO for this goal, conceivably serving as a reference [46]. **Participants**. Participants were recruited from the scientific staff of a chair of urban wastewater management (n = 12) and from the acquaintances of the study leader (n = 14). Altogether, 26 participants were involved, voluntarily, and did not receive incentives in the study. Before entering the validation study, the participants received a written introduction to the study, its purpose, and the tasks to be performed. At the end of the study-more than six weeks after the beginning, 18 participants were still active.