## 4 Dyversify Architecture
### Event/Anomaly Detection
#### 4.6.4 Event Detection: Expert-Rules
## 5 Evaluation Results

To evaluate the complete architecture, from sensor value ingestion up to the visualization of detected events in the dashboard, the end-to-end methodology was evaluated for the derivation of highly accurate anomalies and their interpretable causes from sensor monitoring streams by enabling adaptive, context-aware anomaly detection, and root cause analysis through the fusion of (semantic) knowledge- and data-driven techniques. This methodology and pipeline was validated using the [PERSON] use case. All methods, either machine learning based, semantic-based, or fused data analysis for anomaly detection and root cause analysis, have local response times below 1 s. The dynamic detection and match-making of visualizations that fit with sensors and/or occurring events remained below 1.3 s for the user-driven visualization and below 1.6 s for the anomaly-driven visualization. It is logical that anomaly-driven visualization takes on average slightly longer to complete than user-driven visualization, because the dashboard must suggest multiple visualization widgets at once instead of assisting in the visualization options for just one widget. As a result, the full chain from sensor data ingestion up to visualization is achieved in less than 5 s.

## 6 Lessons Learned

After this overview of the architecture of the Dyversify stack, we now discuss lessons we learned while developing and testing our stack. ### Scalability Requirements

Scalability is achieved by up- or downscaling the instances of components based on the workload and having each instance process a subset of the incoming data. When these components and their data flow adhere to certain guidelines, the complexity can be greatly reduced. First, each instance should be able to work independently, so there is no need for synchronization between all instances. Ideally, the input data also forms disjoint logical groups. In our proof-of-concept, we wanted to perform event detection across all ventilated rooms per building, so the data had to be partitioned in such a way that all streams originating from a single building were assigned to the same Kafka partition. Here, we encountered an issue for the MP-events component which also uses user feedback. The event topic contains the patterns that need to be tracked, while the measurement topic contains the actual data being tested. The issue originates from the need to process two Kafka topics at the same time (with the event topic having priority), which is not supported by Kafka. We solved this by interweaving the reads from both topics, though this introduces a timing-dependent non-deterministic behavior that might be undesired for some use cases. A second requirement for reactive scalability is that each instance should have a short startup (ideally around or less than one minute) and perform a clean and fast shutdown when it receives a termination signal. Here, we experienced many problems with the MP-outliers service, which determines anomalies by referencing one year of historic data. Because of the way different device/metric pairs are interwoven on the measurement topic, all relevant historic data will be loaded before the detector achieves normal operating speed. As fetching historic data could take several minutes for a single sensor, this resulted in an effective startup time of 30 to 60 min before the service could start processing at normal speed. We observed cases where data fetching for a single sensor took exceptionally long, up to 12 min. These occurrences triggered the mechanism in Kafka to detect stalled processors, which terminated these processors and redistributed the workload, effectively re-triggering the issue. We solved the cascade problem by using manual partition assignment and starting the instances one at a time to avoid all instances querying Obelisk at the same moment. The long data fetches could be solved by caching the historic datasets on a shared drive, avoiding the need for lengthy data retrieval queries, and by increasing Kafka timeouts through configuration.