## 4 Dyversify Architecture
### Event/Anomaly Detection
#### 4.6.4 Event Detection: Expert-Rules
After user interaction, the label and semantic type of the event are updated and the updated event is resubmitted to the event Kafka topic where it can be picked up again by other services. The dashboard is described in more detail in other works. [PERSON] et al. describe the dashboard in the context of Dyversify and include a demo movie ([[https://www.dropbox.com/s/lhg7v5](https://www.dropbox.com/s/lhg7v5) wz09 ffvun/Dyversify%20 demo.avi?dl=0]([https://www.dropbox.com/s/lhg7v5](https://www.dropbox.com/s/lhg7v5) wz09 ffvun/Dyversify%20 demo.avi?dl=0), (accessed on 3 October 2021)) in their work [66]. [PERSON] et al. describe the interaction between the dashboard and Obelisk in more detail for a different, industrial IoT case [43]. ### Monitoring

Ensuring components are behaving as expected is of vital importance both during development and in production settings. Monitoring systems give insight into the underlying system to ensure this is the case. Even if different teams do not know the specifics of components maintained by other teams, it is straightforward to interpret metrics such as CPU usage, message throughput, or Kafka messages to quickly validate that components are behaving well. As a monitoring solution, we compared the TICK stack and Prometheus. TICK (Telegraf, InfluxbDB, Chronograf, and Kapacitor) is a set of open-source tools that can be combined together or used separately to collect, store, visualize, and manipulate time-seriesdata. It is developed by InfluData, mostly known for its time-series database InfluxDB. The second candidate, Prometheus is a fully open-source monitoring solution inspired by Google Borg Monitor [67]. It was initially developed by SoundCloud and later donated to the Cloud Native Computing Foundation, which also houses Kubernetes, fluentd, Helm, Envoy, and others. Each Prometheus server is standalone, ensuring correct functioning of Prometheus even when other parts of the infrastructure are broken. While both monitoring systems have the same capabilities, we chose Prometheus as a monitoring solution because it collects metrics in a pull-based manner, has a more streamlined data store, has a less verbose query language, and found it easier to configure for small setups. Monitoring solutions such as Prometheus collect metrics can be broadly divided into three categories: service metrics (e.g., input or error rate), resource metrics (e.g., CPU usage or network I/O), and events (e.g., alerts or configuration changes). Some metrics like cache hits or database locks cannot be put in any of these categories but may still prove to be useful in representing the operability of the system. Existing methodologies can prove good starting grounds for deciding which metrics to collect. The USE-method [68] is a system-agnostic methodology that focuses on the resource utilization, saturation, and errors. Another methodology comes from the Google Site-Reliability Engineering (SRE) team; they formulated four golden signals: latency, traffic, errors, and saturation [67]. Finally, the RED-method (rate, errors, duration) was created by [PERSON], a former Google SRE employee, and focuses on microservice monitoring. All of the metrics above were made available in a Grafana dashboard. Still, we found the Kafka consumer lag and latest Kafka messages the most useful metrics during service development. We also experimented with a fully automatic rule-based scaling mechanism for the microservices, based on the number of incoming messages, the number of consumed messages, the average consumer lag, and the derivation thereof. This approach proved successful, but it took some time before the system reached a stable state after rescaling, so specific configuration for every use case would be required in order to be truly efficient.