## 3 Results and Discussion
### Interactive Learning/Human-In-The-Loop  
Machine learning is applied to develop algorithms that can learn from data and improve the knowledge base without human intervention to solve different problems such as pattern recognition, classification, etc. in real time. As mentioned, it is not always possible to achieve high performance without involving humans. Here, interactive learning, i.e., including human-in-the-loop, can be useful to improve the performance of the learning process. One of the main objectives of an IDIVS is to apply feedback from the user in effective ways to improve results, as illustrated in Figure 4. Different types of user feedback can be applied to IDIVSs. The feedback can be explicit, e.g., the user provides input to the IDIVS through a user interface, or implicit, e.g., the user turns on a light switch, providing feedback to the IDIVS that it was too dark. Furthermore, there might be many alternatives regarding when users provide labels, i.e., different triggers such as the place and time of interaction [25]. A taxonomy of different interactive learning strategies where different types of triggers were identified is presented by [PERSON] et al. [21]. The taxonomy distinguishes between when the system is more active in the feedback process, as in active learning, and when the user is more proactive, as in machine teaching. The classification of triggers includes the following for active learning: Uncertainty, i.e., the ML acknowledges the uncertainty of the classification; Time, i.e., the system asks the user for a label at a given time interval; and Random, where the user is asked randomly. The classification of triggers for machine teaching includes the following: Error, i.e., the user provides a label when the system is wrong; State change, i.e., the user provides a label when the state of the environment changes; Time, the user provides a label when the user acknowledges a time interval has passed; User factors, i.e., other factors the user may consider for providing a label. We also acknowledge that hybrid versions of the approaches presented exist, such that the user may be asked to provide labels both based on uncertainty and random. Importantly, when designing systems with an IDIVS, we advise considering a service design that allows for both active learning and machine teaching. Hence, this requires that a system of IDIVS and services, such that the user is able to observe the system output as well as the environment, receive requests from the system for feedback, e.g., correct labels, and have efficient means of allowing the user to provide feedback, both on request and pro-actively. The IDIVS suggested capability of supporting self-assessment of the accuracy of its output typically comes into play when an active learning strategy, such as the Uncertainty strategy, is to be used, in which the user is requested to provide labels when the uncertainty is high. The choice of which particular learning strategy (or hybrid) to use, as introduced above, depends, of course, on the particular application and on the type of user who will be present. Further, the chosen interactive approaches for learning can significantly influence the quality of the classification [22]. As an illustration, please see Figure 5, which clearly indicates that the interactive learning strategy influences performance. The graph shows the average performance of three different learning algorithms, Naive Bayes (NB), k-Nearest Neighbour (k-NN), and Support Vector Machine (SVM) [22]. In this case, the learning strategy appears to influence the performance much more than the choice of learning algorithms. The labelling budget expresses the number of correct labels provided to the system in relation to the number of data instances streamed. Note, however, that the different learning strategies may imply different amounts of effort for the users. For instance, the Error strategy requires the user to constantly monitor both the system output and the environment, looking for discrepancies.