1. System Structure
 - Monitoring Server: 
+ Operating System: Ubuntu 22.04
+ IP Address: 192.168.197.128 
+ Functions: 
•	Prometheus Server (version 2.37.9): Collects and stores data from monitored servers. 
•	Node Exporter (version:1.6.1): Collects metrics from the monitoring server itself. 
•	Alertmanager (version 0.26.0): Manages and sends alert notifications based on established rules. 
•	Grafana (version 11.2.0): Creates dashboards and displays data from Prometheus.
- Monitored Server: 
+ Operating System: Kali Linux 
+ IP Address: 192.168.197.135 
+ Function: Node Exporter (version:1.6.1): Collects server metrics and provides them to Prometheus.
2. Data Flow
- Prometheus Server on Ubuntu machine (192.168.197.128) collects data from: 
+ Node Exporter on Ubuntu (itself) 
+ Node Exporter on Kali machine (192.168.197.135)
- Alertmanager is integrated with Prometheus to manage alerts.
- Grafana connects to Prometheus to retrieve data and create visual dashboards.
3. Model Operation
a. Data collection from Kali server: 
- Node Exporter is installed on Kali Linux (192.168.197.135). 
- Node Exporter collects system metrics (CPU, memory, disk, network, etc.) and exposes them via a default HTTP endpoint on port 9100. 
- Prometheus Server on Ubuntu accesses the Node Exporter endpoint on Kali to collect data.
b. Data collection from Ubuntu server: 
- On Ubuntu (192.168.197.128), Node Exporter is also installed to collect metrics of this server itself. - Prometheus Server collects data from this local Node Exporter.
c. Prometheus Server: 
- Installed on Ubuntu (192.168.197.128). 
- Periodically (based on scrape_interval) sends requests to Node Exporters to collect data. 
- Stores collected data in Prometheus' time-series database. 
- Evaluates alert rules based on collected data.
d. Alertmanager: 
- Installed on Ubuntu (192.168.197.128) and integrated with Prometheus. 
- Receives alerts from Prometheus when alert conditions are triggered (e.g., CPU overload, low memory, etc.). 
- Processes alerts (grouping, deduplication, etc.) and sends notifications to configured communication channels (email, Slack, etc.).
e. Grafana: 
- Installed on Ubuntu (192.168.197.128). 
- Connects to Prometheus Server as a data source. 
- Creates and displays visual dashboards based on data from Prometheus. 
- Allows users to customize and create various charts, tables, and panels. 
- Supports alerting and notifications based on configured thresholds.
4. Data Processing and Collected Metrics
a. Data Processing: 
- Node Exporter on each server collects system metrics and exposes them via HTTP in metric format. - Prometheus Server connects to Node Exporters to fetch this data, then stores and processes it. 
- Data is stored as time-series, allowing for historical querying and graph visualization.
b. Collected Metrics: 
- CPU: 
+ CPU usage (user, system, idle, iowait, etc.) 
+ Load average 
- Memory (RAM): 
+ Total capacity, used capacity, free capacity 
+ Swap information 
- Disk: 
+ Disk capacity (total, used, free) 
+ Number of read/write operations 
+ I/O utilization 
- Network: 
+ Network traffic (upload, download) 
+ Number of packets sent/received 
+ Network errors 
- System Information: 
+ Uptime 
+ Process and thread information 
+ System service status
5. Detailed Operation
a. Node Exporter: 
- Lightweight application with minimal impact on system performance. 
- Provides metrics in a format Prometheus understands. 
- Operates as an HTTP server, listening on default port 9100. 
- Uses collectors to gather information from various system sources.
b. Prometheus Server: 
- Periodically sends HTTP requests to targets (Node Exporters) to scrape metrics. 
- Processes and stores collected data in a local time-series database. 
- Provides PromQL query language for searching and aggregating data. 
- Evaluates alert rules and sends alerts to Alertmanager when necessary.
c. Alertmanager: 
- Receives alerts from Prometheus Server. 
- Groups, filters, and routes alerts based on configured rules. 
- Handles alert deduplication and manages silences. 
 Sends notifications via configured channels like email, Slack, PagerDuty, etc.
d. Grafana: 
- Connects to Prometheus Server via API to retrieve data. 
- Uses PromQL for complex data queries. 
- Creates and manages dashboards with various panels (graphs, tables, gauges, etc.). 
- Supports annotations to mark important events on graphs. 
- Allows setting up alerts based on conditions defined on panels. 
- Supports user authentication and access control for dashboards. 
- Provides dashboard sharing and data export capabilities.
Instead of installing on the virtual machine, I installed on the Windows docker and added the tools CADVISOR DVWA Loki Promtail Redis Pushgateway and a AI model discovered SQL and XSS attacks. Based on the introduction above or rewrite the additional and add additional tools