ZW21-Day2-Using a Forest to Explore the Logs  Automation, Analytics & AI w/Zeek Logs at UC Davis
Channel: Zeek
Date: 2021-12-09

Using a Forest to Explore the Logs: Automation, Analytics and AI with Zeek Logs at UC Davis - Jeff Rowe

Cyber-security operations at a large public university are a challenge due to open access policies and the volume and variety of information flows that result. Network data collection at the campus border is a foundational capability supporting a variety of university SecOps workflows, but making sense out of large volumes of Zeek network traffic logs is daunting. The UC Davis SOC has developed and deployed automation, analytics and AI to leverage the power of these logs, managing alerts, classifying problematic data flows, and identifying evolving threats specific to UC Davis. We automate alert management by filtering and correlating across Zeek log streams to create investigation packages. These include all data, occurring before and after the alert, that are relevant to alert dispensation. We employ the isolation forest machine learning algorithm to detect unusual behavior hidden in encrypted data flows. This generates alerts for anomalous outbound SSL, RDP and SSH connections from the Zeek conn log, which has successfully identified serious security incidents. We have also developed a novel threat intelligence generation process that identifies threat indicators specific to UC Davis. Our Zeek intel logs average 1 million alerts per day; we build a fully tuned random forest model from these data to learn a UC Davis-specific threat profile which has produced highly accurate classification results, and practical, actionable security alerts.