Project: SkyWalking (Observability and Analysis / Observability)
File: backend-nginx-monitoring.md

The is a lua library that can be used with Nginx to collect metrics and expose them on a separate web page. To use this library, you will need Nginx with or directly . SkyWalking leverages OpenTelemetry Collector to transfer the metrics to and into the . collects metrics from Nginx and expose them to an endpoint. OpenTelemetry Collector fetches metrics from the endpoint expose above via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter. The SkyWalking OAP Server parses the expression with to filter/calculate/aggregate and store the results. Collect Nginx metrics and expose the following four metrics by . For details on metrics definition, refer to . histogram: nginxhttplatency gauge: nginxhttpconnections counter: nginxhttpsize_bytes counter: nginxhttprequests_total Set up . For details on Prometheus Receiver in OpenTelemetry Collector, refer to . Config SkyWalking . SkyWalking observes the status, payload, and latency of the Nginx server, which is cataloged as a `LAYER: Nginx` `Service` in the OAP and instances would be recognized as `LAYER: Nginx` `instance`. About `LAYER: Nginx` `endpoint`, it depends on how precision you want to monitor the nginx. We do not recommend expose every request path metrics, because it will cause explosion of metrics endpoint data. You can collect host metrics: ``` http { logbylua_block { metricbytes:inc(tonumber(ngx.var.requestlength), {"request", ngx.var.host}) metricbytes:inc(tonumber(ngx.var.bytessend), {"response", ngx.var.host}) metric_requests:inc(1, {ngx.var.status, ngx.var.host}) metriclatency:observe(tonumber(ngx.var.requesttime), {ngx.var.host}) } } ``` or grouped urls and upstream metrics: ``` upstream backend { server ip:port; } server { location /test { default_type application/json; return 200 '{"code": 200, "message": "success"}'; logbylua_block { metricbytes:inc(tonumber(ngx.var.requestlength), {"request", "/test/"}) metricbytes:inc(tonumber(ngx.var.bytessend), {"response", "/test/"}) metric_requests:inc(1, {ngx.var.status, "/test/"}) metriclatency:observe(tonumber(ngx.var.requesttime), {"/test/"}) } } location /test_upstream { proxy_pass http://backend; logbylua_block { metricbytes:inc(tonumber(ngx.var.requestlength), {"request", "upstream/backend"}) metricbytes:inc(tonumber(ngx.var.bytessend), {"response", "upstream/backend"}) metric_requests:inc(1, {ngx.var.status, "upstream/backend"})

{"upstream/backend"}) } } } ``` | Monitoring Panel | Unit | Metric Name | Catalog | Description | Data Source | |-||--|||--| | HTTP Request Trend | | meternginxservicehttprequests | Service | The increment rate of HTTP requests | nginx-lua-prometheus | | HTTP Latency | ms | meternginxservicehttplatency | Service | The increment rate of the latency of HTTP requests | nginx-lua-prometheus | | HTTP Bandwidth | KB | meternginxservice_bandwidth | Service | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus | | HTTP Connections | | meternginxservicehttpconnections | Service | The avg number of the connections | nginx-lua-prometheus | | HTTP Status Trend | | meternginxservicehttpstatus | Service | The increment rate of the status of HTTP requests | nginx-lua-prometheus | | HTTP Status 4xx Percent | % | meternginxservicehttp4xxrequestsincrement / meternginxservicehttprequests_increment | Service | The percentage of 4xx status of HTTP requests | nginx-lua-prometheus | | HTTP Status 5xx Percent | % | meternginxservicehttp5xxrequestsincrement / meternginxservicehttprequests_increment | Service | The percentage of 4xx status of HTTP requests | nginx-lua-prometheus | | Monitoring Panel | Unit | Metric Name | Catalog | Description | Data Source | |||-|-||--| | HTTP Request Trend | | meternginxinstancehttprequests | Instance | The increment rate of HTTP requests | nginx-lua-prometheus | | HTTP Latency | ms | meternginxinstancehttplatency | Instance | The increment rate of the latency of HTTP requests | nginx-lua-prometheus | | HTTP Bandwidth | KB | meternginxinstance_bandwidth | Instance | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus | | HTTP Connections | | meternginxinstancehttpconnections | Instance | The avg number of the connections | nginx-lua-prometheus | | HTTP Status Trend | | meternginxinstancehttpstatus | Instance | The increment rate of the status of HTTP requests | nginx-lua-prometheus | | HTTP Status 4xx Percent | % | meternginxinstancehttp4xxrequestsincrement / meternginxinstancehttprequests_increment | Instance | The percentage of 4xx status of HTTP requests | nginx-lua-prometheus | | HTTP Status 5xx Percent | % | meternginxinstancehttp5xxrequestsincrement / meternginxinstancehttprequests_increment | Instance | The percentage of 4xx status of HTTP requests | nginx-lua-prometheus | | Monitoring Panel | Unit | Metric Name | Catalog | Description | Data Source | |-||-|-||-| | HTTP Request Trend | | meternginxendpointhttprequests | Endpoint | The increment rate of HTTP requests | nginx-lua-prometheus | | HTTP Latency | ms | meternginxendpointhttplatency | Endpoint | The increment rate of the latency of HTTP requests | nginx-lua-prometheus | | HTTP Bandwidth | KB | meternginxendpoint_bandwidth | Endpoint | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus | | HTTP Status Trend | | meternginxendpointhttpstatus | Endpoint | The increment rate of the status of HTTP requests | nginx-lua-prometheus | | HTTP Status 4xx Percent | % | meternginxendpointhttp4xxrequestsincrement / meternginxendpointhttprequests_increment | Endpoint | The percentage of 4xx status of HTTP requests | nginx-lua-prometheus | | HTTP Status 5xx Percent | % | meternginxendpointhttp5xxrequestsincrement / meternginxendpointhttprequests_increment | Endpoint | The percentage of 4xx status of HTTP requests | nginx-lua-prometheus | You can customize your own metrics/expression/dashboard panel. The metrics definition and expression rules are found in `/config/otel-rules/nginx-service.yaml, /config/otel-rules/nginx-instance.yaml, /config/otel-rules/nginx-endpoint.yaml`. The Nginx dashboard panel configurations are found in `/config/ui-initialized-templates/nginx`. SkyWalking leverages or other log agents for collecting access log and error log of Nginx. fluentbit agent collects access log and error log from Nginx. fluentbit agent sends data to SkyWalking OAP Server using native meter APIs via HTTP. The SkyWalking OAP Server parses the expression with to parse/extract and store the results. Install . Config fluent bit with fluent-bit.conf, refer to . Error Log monitoring provides monitoring of the error.log of the Nginx server.