Repository: apache/skywalking (language: python)

Issue:
[Feature] Support Nginx monitoring.
### Search before asking

- [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar feature requirement.


### Description

Nginx is a popular HTTP and reverse proxy server. I was already doing this work last weekend, please assign this task to me, more details and progress I will sync here.

### Use case

_No response_

### Related issues

_No response_

### Are you willing to submit a pull request to implement this on your own?

- [X] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)



Discussion / hints:
Are you using both metrics and log analysis?
Just metrics analysis, expose by [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus), but it need some extra programming in nginx.conf:
```
http {
    resolver local=on ipv6=off;

    lua_shared_dict prometheus_metrics 10M;
    # lua_package_path "/path/to/nginx-lua-prometheus/?.lua;;";

    init_worker_by_lua_block {
      prometheus = require("prometheus").init("prometheus_metrics")

      metric_bytes = prometheus:counter(
        "nginx_http_size_bytes", "Total size of HTTP", {"type", "route"})
      metric_requests = prometheus:counter(
        "nginx_http_requests_total", "Number of HTTP requests", {"status", "route"})
      metric_latency = prometheus:histogram(
        "nginx_http_latency", "HTTP request latency", {"route"})
      metric_connections = prometheus:gauge(
        "nginx_http_connections", "Number of HTTP connections", {"state"})
    }

    server {
        listen 8080;

        location /test {
          default_type application/json;
          return 200  '{"code": 200, "message": "success"}';

          log_by_lua_block {
            metric_bytes:inc(tonumber(ngx.var.bytes_received), {"request", "/test/**"})
            metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
            metric_requests:inc(1, {ngx.var.status, "/test/**"})
            metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
          }
        }

        location /test_404 {
          default_type application/json;
          return 404  '{"code": 404, "message": "not found"}';

          log_by_lua_block {
            metric_bytes:inc(tonumber(ngx.var.bytes_received), {"request", "/test_404/**"})
            metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test_404/**"})
            metric_requests:inc(1, {ngx.var.status, "/test_404/**"})
            metric_latency:observe(tonumber(ngx.var.request_time), {"/test_404/**"})
          }
        }

        location /test_500 {
          default_type application/json;
          return 500  '{"code": 500, "message": "internal error"}';

          log_by_lua_block {
            metric_bytes:inc(tonumber(ngx.var.bytes_received), {"request", "/test_500/**"})
            metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test_500/**"})
            metric_requests:inc(1, {ngx.var.status, "/test_500/**"})
            metric_latency:observe(tonumber(ngx.var.request_time), {"/test_500/**"})
          }
        }
    }

    server {
      listen 9145;
      location /metrics {
        content_by_lua_block {
          metric_connections:set(ngx.var.connections_reading, {"reading"})
          metric_connections:set(ngx.var.connections_waiting, {"waiting"})
          metric_connections:set(ngx.var.connections_writing, {"writing"})
          prometheus:collect()
        }
      }
    }
}
```

I compared it with [nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) and [nginx-vts-exporter](https://github.com/hnlq715/nginx-vts-exporter), this lua library can provide more metrics and user can freely define their endpoint grouping rule.
[nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter) works well with nginx plus, but in nginx oss it expose poor metrics.

[nginx-vts-exporter](https://github.com/hnlq715/nginx-vts-exporter) requires rebuild nginx with [vts-module](https://github.com/vozlt/nginx-module-vts), alough [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus) requires extra lua module too, I think it is more widely used by user.
Thanks for the explanation. I just wanted to ask about [nginx-prometheus-exporter](https://github.com/nginxinc/nginx-prometheus-exporter).

__

So, you are proposing to use [nginx-vts-exporter](https://github.com/hnlq715/nginx-vts-exporter)?
About logs, I was trying to ask, whether we should check whether there are some typically error logs to indicate something, we could put them on the LAL and dashboard.
I propose to use [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus).

About error logs analysis, I need more research, once I get some clues I will share with you here.
> I propose to use [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus).

About this, this requires adding more metrics through LUA scripts for every route rule, right?

> > I propose to use [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus).
> 
> About this, this requires adding more metrics through LUA scripts for every route rule, right?

It can be added in both http or location section, it depended on how precision user want monitoring endpoint metrics, may be just `/**` in http section, or more precision `user/**`, `user/1`, `user/2` in location section.
> > > I propose to use [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus).
> > 
> > 
> > About this, this requires adding more metrics through LUA scripts for every route rule, right?
> 
> It can be added in both http or location section, it depended on how precision user want monitoring endpoint metrics, may be just `/**` in http section, or more precision `user/**`, `user/1`, `user/2` in location section.

Yes, that was I mean. OK, then we need a good documentation to explain.
> > > > I propose to use [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus).
> > > 
> > > 
> > > About this, this requires adding more metrics through LUA scripts for every route rule, right?
> > 
> > 
> > It can be added in both http or location section, it depended on how precision user want monitoring endpoint metrics, may be just `/**` in http section, or more precision `user/**`, `user/1`, `user/2` in location section.
> 
> Yes, that was I mean. OK, then we need a good documentation to explain.

Ok, thanks.
Error Log Count metric is extracted from nginx `error.log`, this may be meaningful to user.
 
![image](https://github.com/apache/skywalking/assets/16346644/e90dca5a-3c3b-4799-8cb4-d8310f60ad6c)


Yes, monitoring output error level logs totally make sense.

Fix patch:
diff --git a/.github/workflows/skywalking.yaml b/.github/workflows/skywalking.yaml
index 12d60d7d10d1..488588ac1492 100644
--- a/.github/workflows/skywalking.yaml
+++ b/.github/workflows/skywalking.yaml
@@ -611,6 +611,8 @@ jobs:
           - name: Zipkin BanyanDB
             config: test/e2e-v2/cases/zipkin/banyandb/e2e.yaml
 
+          - name: Nginx
+            config: test/e2e-v2/cases/nginx/e2e.yaml
           - name: APISIX metrics
             config: test/e2e-v2/cases/apisix/otel-collector/e2e.yaml
           - name: Exporter Kafka
diff --git a/docs/en/changes/changes.md b/docs/en/changes/changes.md
index 99a6978f11b9..679064c6550f 100644
--- a/docs/en/changes/changes.md
+++ b/docs/en/changes/changes.md
@@ -42,6 +42,7 @@
 * Change the DataCarrier IF_POSSIBLE strategy to use ArrayBlockingQueue implementation.
 * Change the policy of the queue(DataCarrier) in the L1 metric aggregate worker to IF_POSSIBLE mode. 
 * Add self-observability metric `metrics_aggregator_abandon` to count the number of abandon metrics.
+* Support Nginx monitoring.
 
 #### UI
 
diff --git a/docs/en/setup/backend/backend-nginx-monitoring.md b/docs/en/setup/backend/backend-nginx-monitoring.md
new file mode 100644
index 000000000000..696aa93db863
--- /dev/null
+++ b/docs/en/setup/backend/backend-nginx-monitoring.md
@@ -0,0 +1,141 @@
+# Nginx monitoring
+## Nginx performance from nginx-lua-prometheus
+The [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus) is a lua library that can be used with Nginx to collect metrics 
+and expose them on a separate web page. 
+To use this library, you will need Nginx with [lua-nginx-module](https://github.com/openresty/lua-nginx-module) or directly [OpenResty](https://openresty.org/).
+
+SkyWalking leverages OpenTelemetry Collector to transfer the metrics to [OpenTelemetry receiver](opentelemetry-receiver.md) and into the [Meter System](./../../concepts-and-designs/meter.md).
+
+### Data flow
+1. [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus) collects metrics from Nginx and expose them to an endpoint.
+2. OpenTelemetry Collector fetches metrics from the endpoint expose above via Prometheus Receiver and pushes metrics to SkyWalking OAP Server via OpenTelemetry gRPC exporter.
+3. The SkyWalking OAP Server parses the expression with [MAL](../../concepts-and-designs/mal.md) to filter/calculate/aggregate and store the results.
+
+### Set up
+1. Collect Nginx metrics and expose the following four metrics by [nginx-lua-prometheus](https://github.com/knyar/nginx-lua-prometheus). For details on metrics definition, refer to [here](../../../../test/e2e-v2/cases/nginx/nginx.conf).
+- histogram: nginx_http_latency
+- gauge: nginx_http_connections
+- counter: nginx_http_size_bytes
+- counter: nginx_http_requests_total
+
+2. Set up [OpenTelemetry Collector ](https://opentelemetry.io/docs/collector/getting-started/#docker). For details on Prometheus Receiver in OpenTelemetry Collector, refer to [here](../../../../test/e2e-v2/cases/nginx/otel-collector-config.yaml).
+3. Config SkyWalking [OpenTelemetry receiver](opentelemetry-receiver.md).
+
+### Nginx Monitoring
+
+SkyWalking observes the status, payload, and latency of the Nginx server, which is cataloged as a `LAYER: Nginx` `Service` in the OAP and instances would be recognized as `LAYER: Nginx` `instance`.
+
+About `LAYER: Nginx` `endpoint`, it depends on how precision you want to monitor the nginx.
+We do not recommend expose every request path metrics, because it will cause explosion of metrics endpoint data.
+
+You can collect host metrics:
+```
+http {
+  log_by_lua_block {
+      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", ngx.var.host})
+      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", ngx.var.host})
+      metric_requests:inc(1, {ngx.var.status, ngx.var.host})
+      metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.host})
+  }
+}
+```
+or grouped urls and upstream metrics:
+```
+upstream backend {
+  server ip:port;
+}
+
+server {
+
+  location /test {
+    default_type application/json;
+    return 200  '{"code": 200, "message": "success"}';
+  
+    log_by_lua_block {
+      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
+      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
+      metric_requests:inc(1, {ngx.var.status, "/test/**"})
+      metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
+    }
+  }
+  
+  location /test_upstream {
+  
+    proxy_pass http://backend;
+  
+    log_by_lua_block {
+      metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "upstream/backend"})
+      metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "upstream/backend"})
+      metric_requests:inc(1, {ngx.var.status, "upstream/backend"})
+      metric_latency:observe(tonumber(ngx.var.request_time), {"upstream/backend"})
+    }
+  }
+}
+```
+
+#### Nginx Service Supported Metrics 
+| Monitoring Panel        | Unit | Metric Name                                                                                   | Catalog | Description                                          | Data Source                    |
+|-------------------------|------|-----------------------------------------------------------------------------------------------|---------|------------------------------------------------------|--------------------------------|
+| HTTP Request Trend      |      | meter_nginx_service_http_requests                                                             | Service | The increment rate of HTTP requests                  | nginx-lua-prometheus           |
+| HTTP Latency            | ms   | meter_nginx_service_http_latency                                                              | Service | The increment rate of the latency of HTTP requests   | nginx-lua-prometheus           |
+| HTTP Bandwidth          | KB   | meter_nginx_service_bandwidth                                                                 | Service | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus           |
+| HTTP Connections        |      | meter_nginx_service_http_connections                                                          | Service | The avg number of the connections                    | nginx-lua-prometheus           |
+| HTTP Status Trend       |      | meter_nginx_service_http_status                                                               | Service | The increment rate of the status of HTTP requests    | nginx-lua-prometheus           |
+| HTTP Status 4xx Percent | %    | meter_nginx_service_http_4xx_requests_increment / meter_nginx_service_http_requests_increment | Service | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |
+| HTTP Status 5xx Percent | %    | meter_nginx_service_http_5xx_requests_increment / meter_nginx_service_http_requests_increment | Service | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |
+
+#### Nginx Instance Supported Metrics
+| Monitoring Panel          | Unit | Metric Name                                                                                     | Catalog  | Description                                          | Data Source                    |
+|---------------------------|------|-------------------------------------------------------------------------------------------------|----------|------------------------------------------------------|--------------------------------|
+| HTTP Request Trend        |      | meter_nginx_instance_http_requests                                                              | Instance | The increment rate of HTTP requests                  | nginx-lua-prometheus           |
+| HTTP Latency              | ms   | meter_nginx_instance_http_latency                                                               | Instance | The increment rate of the latency of HTTP requests   | nginx-lua-prometheus           |
+| HTTP Bandwidth            | KB   | meter_nginx_instance_bandwidth                                                                  | Instance | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus           |
+| HTTP Connections          |      | meter_nginx_instance_http_connections                                                           | Instance | The avg number of the connections                    | nginx-lua-prometheus           |
+| HTTP Status Trend         |      | meter_nginx_instance_http_status                                                                | Instance | The increment rate of the status of HTTP requests    | nginx-lua-prometheus           |
+| HTTP Status 4xx Percent   | %    | meter_nginx_instance_http_4xx_requests_increment / meter_nginx_instance_http_requests_increment | Instance | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |
+| HTTP Status 5xx Percent   | %    | meter_nginx_instance_http_5xx_requests_increment / meter_nginx_instance_http_requests_increment | Instance | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus           |
+
+#### Nginx Endpoint Supported Metrics
+| Monitoring Panel        | Unit | Metric Name                                                                                     | Catalog  | Description                                          | Data Source          |
+|-------------------------|------|-------------------------------------------------------------------------------------------------|----------|------------------------------------------------------|----------------------|
+| HTTP Request Trend      |      | meter_nginx_endpoint_http_requests                                                              | Endpoint | The increment rate of HTTP requests                  | nginx-lua-prometheus |
+| HTTP Latency            | ms   | meter_nginx_endpoint_http_latency                                                               | Endpoint | The increment rate of the latency of HTTP requests   | nginx-lua-prometheus |
+| HTTP Bandwidth          | KB   | meter_nginx_endpoint_bandwidth                                                                  | Endpoint | The increment rate of the bandwidth of HTTP requests | nginx-lua-prometheus |
+| HTTP Status Trend       |      | meter_nginx_endpoint_http_status                                                                | Endpoint | The increment rate of the status of HTTP requests    | nginx-lua-prometheus |
+| HTTP Status 4xx Percent | %    | meter_nginx_endpoint_http_4xx_requests_increment / meter_nginx_endpoint_http_requests_increment | Endpoint | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus |
+| HTTP Status 5xx Percent | %    | meter_nginx_endpoint_http_5xx_requests_increment / meter_nginx_endpoint_http_requests_increment | Endpoint | The percentage of 4xx status of HTTP requests        | nginx-lua-prometheus |
+
+### Customizations
+You can customize your own metrics/expression/dashboard panel.
+
+The metrics definition and expression rules are found in `/config/otel-rules/nginx-service.yaml, /config/otel-rules/nginx-instance.yaml, /config/otel-rules/nginx-endpoint.yaml`.
+
+The Nginx dashboard panel configurations are found in `/config/ui-initialized-templates/nginx`.
+
+## Collect nginx access and error log
+SkyWalking leverages [fluentbit](https://fluentbit.io/) or other log agents for collecting access log and error log of Nginx.
+
+### Data flow
+1. fluentbit agent collects access log and error log from Nginx.
+2. fluentbit agent sends data to SkyWalking OAP Server using native meter APIs via HTTP.
+3. The SkyWalking OAP Server parses the expression with [LAL](../../concepts-and-designs/lal.md) to parse/extract and store the results.
+
+### Set up
+1. Install [fluentbit](https://docs.fluentbit.io/manual/installation/docker).
+2. Config fluent bit with fluent-bit.conf, refer to [here](../../../../test/e2e-v2/cases/nginx/fluent-bit.conf).
+
+### Error Log Monitoring
+Error Log monitoring provides monitoring of the error.log of the Nginx server.
+
+#### Supported Metrics
+| Monitoring Panel         | Metric Name                          | Catalog  | Description                               | Data Source |
+|--------------------------|--------------------------------------|----------|-------------------------------------------|-------------|
+| Service Error Log Count  | meter_nginx_service_error_log_count  | Service  | The count of log level of nginx error.log | fluent bit  |
+| Instance Error Log Count | meter_nginx_instance_error_log_count | Instance | The count of log level of nginx error.log | fluent bit  |
+
+### Customizations
+You can customize your own metrics/expression/dashboard panel.
+
+The log collect and analyse rules are found in `/config/lal/nginx.yaml`, `/config/log-mal-rules/nginx.yaml`.
+
+The Nginx dashboard panel configurations are found in `/config/ui-initialized-templates/nginx`.
\ No newline at end of file
diff --git a/docs/menu.yml b/docs/menu.yml
index eed8f06f8217..a01ed1d4f66f 100644
--- a/docs/menu.yml
+++ b/docs/menu.yml
@@ -239,6 +239,8 @@ catalog:
             path: "/en/setup/service-agent/browser-agent"
           - name: "Gateway Monitoring"
             catalog:
+              - name: "Nginx Monitoring"
+                path: "/en/setup/backend/backend-nginx-monitoring"
               - name: "APISIX Monitoring"
                 path: "/en/setup/backend/backend-apisix-monitoring"
               - name: "AWS API Gateway"
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
index 46cf71f52a07..629292cdb1ad 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/analysis/Layer.java
@@ -205,7 +205,12 @@ public enum Layer {
     /**
      * A scalable, fault-tolerant, and low-latency storage service optimized for real-time workloads.
      */
-    BOOKKEEPER(33, true);
+    BOOKKEEPER(33, true),
+
+    /**
+     * Nginx is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server.
+     */
+    NGINX(34, true);
 
     private final int value;
     /**
diff --git a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
index 686619c428e4..f8e5e7797e3c 100644
--- a/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
+++ b/oap-server/server-core/src/main/java/org/apache/skywalking/oap/server/core/management/ui/template/UITemplateInitializer.java
@@ -72,6 +72,7 @@ public class UITemplateInitializer {
         Layer.KAFKA.name(),
         Layer.PULSAR.name(),
         Layer.BOOKKEEPER.name(),
+        Layer.NGINX.name(),
         "custom"
     };
     private final UITemplateManagementService uiTemplateManagementService;
diff --git a/oap-server/server-starter/src/main/resources/application.yml b/oap-server/server-starter/src/main/resources/application.yml
index 4dcf16e28ebd..f9caeff0c78b 100644
--- a/oap-server/server-starter/src/main/resources/application.yml
+++ b/oap-server/server-starter/src/main/resources/application.yml
@@ -119,7 +119,7 @@ core:
     searchableTracesTags: ${SW_SEARCHABLE_TAG_KEYS:http.method,http.status_code,rpc.status_code,db.type,db.instance,mq.queue,mq.topic,mq.broker}
     # Define the set of log tag keys, which should be searchable through the GraphQL.
     # The max length of key=value should be less than 256 or will be dropped.
-    searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level}
+    searchableLogsTags: ${SW_SEARCHABLE_LOGS_TAG_KEYS:level,http.status_code}
     # Define the set of alarm tag keys, which should be searchable through the GraphQL.
     # The max length of key=value should be less than 256 or will be dropped.
     searchableAlarmTags: ${SW_SEARCHABLE_ALARM_TAG_KEYS:level}
@@ -245,8 +245,8 @@ agent-analyzer:
 log-analyzer:
   selector: ${SW_LOG_ANALYZER:default}
   default:
-    lalFiles: ${SW_LOG_LAL_FILES:envoy-als,mesh-dp,mysql-slowsql,pgsql-slowsql,redis-slowsql,k8s-service,default}
-    malFiles: ${SW_LOG_MAL_FILES:""}
+    lalFiles: ${SW_LOG_LAL_FILES:envoy-als,mesh-dp,mysql-slowsql,pgsql-slowsql,redis-slowsql,k8s-service,nginx,default}
+    malFiles: ${SW_LOG_MAL_FILES:"nginx"}
 
 event-analyzer:
   selector: ${SW_EVENT_ANALYZER:default}
@@ -340,7 +340,7 @@ receiver-otel:
   selector: ${SW_OTEL_RECEIVER:default}
   default:
     enabledHandlers: ${SW_OTEL_RECEIVER_ENABLED_HANDLERS:"otlp-metrics,otlp-logs"}
-    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*"}
+    enabledOtelMetricsRules: ${SW_OTEL_RECEIVER_ENABLED_OTEL_METRICS_RULES:"apisix,nginx/*,k8s/*,istio-controlplane,vm,mysql/*,postgresql/*,oap,aws-eks/*,windows,aws-s3/*,aws-dynamodb/*,aws-gateway/*,redis/*,elasticsearch/*,rabbitmq/*,mongodb/*,kafka/*,pulsar/*,bookkeeper/*"}
 
 receiver-zipkin:
   selector: ${SW_RECEIVER_ZIPKIN:-}
diff --git a/oap-server/server-starter/src/main/resources/lal/nginx.yaml b/oap-server/server-starter/src/main/resources/lal/nginx.yaml
new file mode 100644
index 000000000000..d6c50dd4c0fd
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/lal/nginx.yaml
@@ -0,0 +1,60 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+rules:
+  - name: nginx-access-log
+    layer: NGINX
+    dsl: |
+      filter {
+        if (tag("LOG_KIND") == "NGINX_ACCESS_LOG") {
+          text {
+            regexp $/.+ \"(?<request>.+)\" (?<status>\d{3}) .+/$
+          }
+
+          extractor {
+            if (parsed.status) {
+              tag 'http.status_code': parsed.status
+            }
+          }
+
+          sink {
+          }
+        }
+      }
+  - name: nginx-error-log
+    layer: NGINX
+    dsl: |
+      filter {
+        if (tag("LOG_KIND") == "NGINX_ERROR_LOG") {
+          text {
+            regexp $/(?<time>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?<level>.+)].*/$
+          }
+
+          extractor {
+            tag level: parsed.level
+            timestamp parsed.time as String, "yyyy/MM/dd HH:mm:ss"
+
+            metrics {
+              timestamp log.timestamp as Long
+              labels level: parsed.level, service: log.service, service_instance_id: log.serviceInstance
+              name "nginx_error_log_count"
+              value 1
+            }
+          }
+
+          sink {
+          }
+        }
+      }
diff --git a/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml b/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml
new file mode 100644
index 000000000000..380b43bd46b6
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/log-mal-rules/nginx.yaml
@@ -0,0 +1,36 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+metricPrefix: meter_nginx
+metricsRules:
+  - name: service_error_log_count
+    exp: nginx_error_log_count.sum(['level','service']).downsampling(SUM).service(['service'], Layer.NGINX)
+  - name: instance_error_log_count
+    exp: nginx_error_log_count.sum(['level','service','service_instance_id']).downsampling(SUM).instance(['service'],['service_instance_id'], Layer.NGINX)
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml
new file mode 100644
index 000000000000..cb51df9a4021
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-endpoint.yaml
@@ -0,0 +1,49 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'nginx-monitoring' }" # The OpenTelemetry job name
+expPrefix: tag({tags -> tags.service = 'nginx::' + tags.service})
+expSuffix: endpoint(['service'],['route'], Layer.NGINX)
+metricPrefix: meter_nginx_endpoint
+metricsRules:
+  - name: http_requests
+    exp: nginx_http_requests_total.sum(['service','route']).rate('PT1M')
+  - name: http_latency
+    exp: nginx_http_latency.sum(['type','le','service','route']).histogram().histogram_percentile([50,70,90,99])
+  - name: http_bandwidth
+    exp: nginx_http_size_bytes.sum(['type','service','route']).rate('PT1M')
+  - name: http_status
+    exp: nginx_http_requests_total.sum(['status','service','route']).rate('PT1M')
+  - name: http_requests_increment
+    exp: nginx_http_requests_total.sum(['service','route']).increase('PT1M')
+  - name: http_4xx_requests_increment
+    exp: nginx_http_requests_total.tagMatch("status", "400|401|403|404|405").sum(['service','route']).increase('PT1M')
+  - name: http_5xx_requests_increment
+    exp: nginx_http_requests_total.tagMatch("status", "500|502|503|504").sum(['service','route']).increase('PT1M')
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml
new file mode 100644
index 000000000000..b72a4697068f
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-instance.yaml
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'nginx-monitoring' }" # The OpenTelemetry job name
+expPrefix: tag({tags -> tags.service = 'nginx::' + tags.service})
+expSuffix: instance(['service'],['service_instance_id'], Layer.NGINX)
+metricPrefix: meter_nginx_instance
+metricsRules:
+  - name: http_requests
+    exp: nginx_http_requests_total.sum(['service','service_instance_id']).rate('PT1M')
+  - name: http_latency
+    exp: nginx_http_latency.sum(['type','le','service','service_instance_id']).histogram().histogram_percentile([50,70,90,99])
+  - name: http_bandwidth
+    exp: nginx_http_size_bytes.sum(['type','service','service_instance_id']).rate('PT1M')
+  - name: http_connections
+    exp: nginx_http_connections.sum(['state','service','service_instance_id'])
+  - name: http_status
+    exp: nginx_http_requests_total.sum(['status','service','service_instance_id']).rate('PT1M')
+  - name: http_requests_increment
+    exp: nginx_http_requests_total.sum(['service','service_instance_id']).increase('PT1M')
+  - name: http_4xx_requests_increment
+    exp: nginx_http_requests_total.tagMatch("status", "400|401|403|404|405").sum(['service','service_instance_id']).increase('PT1M')
+  - name: http_5xx_requests_increment
+    exp: nginx_http_requests_total.tagMatch("status", "500|502|503|504").sum(['service','service_instance_id']).increase('PT1M')
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml
new file mode 100644
index 000000000000..29dac9072ae4
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/otel-rules/nginx/nginx-service.yaml
@@ -0,0 +1,51 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This will parse a textual representation of a duration. The formats
+# accepted are based on the ISO-8601 duration format {@code PnDTnHnMn.nS}
+# with days considered to be exactly 24 hours.
+# <p>
+# Examples:
+# <pre>
+#    "PT20.345S" -- parses as "20.345 seconds"
+#    "PT15M"     -- parses as "15 minutes" (where a minute is 60 seconds)
+#    "PT10H"     -- parses as "10 hours" (where an hour is 3600 seconds)
+#    "P2D"       -- parses as "2 days" (where a day is 24 hours or 86400 seconds)
+#    "P2DT3H4M"  -- parses as "2 days, 3 hours and 4 minutes"
+#    "P-6H3M"    -- parses as "-6 hours and +3 minutes"
+#    "-P6H3M"    -- parses as "-6 hours and -3 minutes"
+#    "-P-6H+3M"  -- parses as "+6 hours and -3 minutes"
+# </pre>
+filter: "{ tags -> tags.job_name == 'nginx-monitoring' }" # The OpenTelemetry job name
+expPrefix: tag({tags -> tags.service = 'nginx::' + tags.service})
+expSuffix: service(['service'] , Layer.NGINX)
+metricPrefix: meter_nginx_service
+metricsRules:
+  - name: http_requests
+    exp: nginx_http_requests_total.sum(['service']).rate('PT1M')
+  - name: http_latency
+    exp: nginx_http_latency.sum(['type','le','service']).histogram().histogram_percentile([50,70,90,99])
+  - name: http_bandwidth
+    exp: nginx_http_size_bytes.sum(['type','service']).rate('PT1M')
+  - name: http_connections
+    exp: nginx_http_connections.sum(['state','service'])
+  - name: http_status
+    exp: nginx_http_requests_total.sum(['status','service']).rate('PT1M')
+  - name: http_requests_increment
+    exp: nginx_http_requests_total.sum(['service']).increase('PT1M')
+  - name: http_4xx_requests_increment
+    exp: nginx_http_requests_total.tagMatch("status", "400|401|403|404|405").sum(['service']).increase('PT1M')
+  - name: http_5xx_requests_increment
+    exp: nginx_http_requests_total.tagMatch("status", "500|502|503|504").sum(['service']).increase('PT1M')
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml b/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
index 3ae556d8895b..ba7d4789509e 100644
--- a/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/menu.yaml
@@ -143,6 +143,11 @@ menus:
     description: API gateway is an API management tool that sits between a client and a collection of backend services
     i18nKey: gateway
     menus:
+      - title: Nginx
+        layer: NGINX
+        description: Provide Nginx monitoring through OpenTelemetry's Prometheus Receiver.
+        documentLink: https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-nginx-monitoring/
+        i18nKey: gateway_nginx
       - title: APISIX
         layer: APISIX
         description: Provide APISIX monitoring through OpenTelemetry's Prometheus Receiver.
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json
new file mode 100644
index 000000000000..4d53d0d025ac
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-endpoint.json
@@ -0,0 +1,166 @@
+[
+  {
+    "id": "nginx-Endpoint",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 0,
+          "w": 8,
+          "h": 13,
+          "i": "0",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Request Trend (requests per second)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_endpoint_http_requests"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 8,
+          "y": 0,
+          "w": 8,
+          "h": 13,
+          "i": "1",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Latency (ms)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_endpoint_http_latency"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 16,
+          "y": 0,
+          "w": 8,
+          "h": 13,
+          "i": "2",
+          "type": "Widget",
+          "metricMode": "Expression",
+          "expressions": [
+            "meter_nginx_endpoint_http_bandwidth/1024"
+          ],
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ],
+          "widget": {
+            "title": "HTTP Bandwidth (KB)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          }
+        },
+        {
+          "x": 0,
+          "y": 13,
+          "w": 8,
+          "h": 13,
+          "i": "3",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Status Trend (requests per second)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_endpoint_http_status"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 8,
+          "y": 13,
+          "w": 8,
+          "h": 13,
+          "i": "4",
+          "type": "Widget",
+          "metricMode": "Expression",
+          "expressions": [
+            "(meter_nginx_endpoint_http_4xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100"
+          ],
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ],
+          "graph": {
+            "type": "Area",
+            "opacity": 0.4,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "HTTP Status 4xx Percent Per Minute (%)"
+          }
+        },
+        {
+          "x": 16,
+          "y": 13,
+          "w": 8,
+          "h": 13,
+          "i": "5",
+          "type": "Widget",
+          "metricMode": "Expression",
+          "expressions": [
+            "(meter_nginx_endpoint_http_5xx_requests_increment/meter_nginx_endpoint_http_requests_increment)*100"
+          ],
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ],
+          "graph": {
+            "type": "Area",
+            "opacity": 0.4,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "HTTP Status 5xx Percent Per Minute (%)"
+          }
+        }
+      ],
+      "layer": "NGINX",
+      "entity": "Endpoint",
+      "name": "Nginx-Endpoint",
+      "id": "nginx-Endpoint",
+      "isRoot": false
+    }
+  }
+]
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json
new file mode 100644
index 000000000000..c86c9d1a69cc
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-instance.json
@@ -0,0 +1,214 @@
+[
+  {
+    "id": "nginx-Instance",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 0,
+          "w": 6,
+          "h": 13,
+          "i": "0",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Request Trend (requests per second)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_service_http_requests"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 6,
+          "y": 0,
+          "w": 6,
+          "h": 13,
+          "i": "1",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Latency (ms)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_service_http_latency"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 12,
+          "y": 0,
+          "w": 6,
+          "h": 13,
+          "i": "2",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Bandwidth (KB/s)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_service_http_bandwidth/1024"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 18,
+          "y": 0,
+          "w": 6,
+          "h": 13,
+          "i": "4",
+          "type": "Widget",
+          "expressions": [
+            "meter_nginx_service_http_connections"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ],
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "HTTP Connections"
+          }
+        },
+        {
+          "x": 0,
+          "y": 13,
+          "w": 6,
+          "h": 13,
+          "i": "5",
+          "type": "Widget",
+          "widget": {
+            "title": "HTTP Status Trend (requests per second)"
+          },
+          "graph": {
+            "type": "Line",
+            "step": false,
+            "smooth": false,
+            "showSymbol": true,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "expressions": [
+            "meter_nginx_service_http_status"
+          ],
+          "metricMode": "Expression",
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ]
+        },
+        {
+          "x": 6,
+          "y": 13,
+          "w": 6,
+          "h": 13,
+          "i": "6",
+          "type": "Widget",
+          "metricMode": "Expression",
+          "expressions": [
+            "(meter_nginx_instance_http_4xx_requests_increment/meter_nginx_instance_http_requests_increment)*100"
+          ],
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ],
+          "graph": {
+            "type": "Area",
+            "opacity": 0.4,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "HTTP Status 4xx Percent Per Minute (%)"
+          }
+        },
+        {
+          "x": 12,
+          "y": 13,
+          "w": 6,
+          "h": 13,
+          "i": "7",
+          "type": "Widget",
+          "metricMode": "Expression",
+          "expressions": [
+            "(meter_nginx_service_http_5xx_requests_increment/meter_nginx_service_http_requests_increment)*100"
+          ],
+          "typesOfMQE": [
+            "TIME_SERIES_VALUES"
+          ],
+          "graph": {
+            "type": "Area",
+            "opacity": 0.4,
+            "showXAxis": true,
+            "showYAxis": true
+          },
+          "widget": {
+            "title": "HTTP Status 5xx Percent Per Minute (%)"
+          }
+        },
+        {
+          "x": 18,
+          "y": 13,
+          "w": 6,
+          "h": 13,
+          "i": "8",
+          "type": "Widget",
+          "metricMode": "Expression",
+          "expressions": [
+            "meter_nginx_instance_error_log_count"
+          ],
+          "graph": {
+            "type": "Bar",
+            "showBackground": true
+          },
+          "widget": {
+            "title": "Error Log Count (Per Minute)"
+          },
+          "typesOfMQE": [
+            "UNKNOWN"
+          ]
+        }
+      ],
+      "layer": "NGINX",
+      "entity": "ServiceInstance",
+      "name": "Nginx-Instance",
+      "id": "nginx-Instance",
+      "isRoot": false
+    }
+  }
+]
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-root.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-root.json
new file mode 100644
index 000000000000..d47041c1e53f
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-root.json
@@ -0,0 +1,47 @@
+[
+  {
+    "id": "nginx-root",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 3,
+          "w": 24,
+          "h": 36,
+          "i": "0",
+          "type": "Widget",
+          "graph": {
+            "type": "ServiceList",
+            "dashboardName": "Nginx-Service",
+            "fontSize": 12,
+            "showXAxis": false,
+            "showYAxis": false,
+            "showGroup": true
+          }
+        },
+        {
+          "x": 0,
+          "y": 0,
+          "w": 24,
+          "h": 3,
+          "i": "1",
+          "type": "Text",
+          "graph": {
+            "fontColor": "theme",
+            "backgroundColor": "theme",
+            "content": "Provide Nginx monitoring through OpenTelemetry's Prometheus Receiver",
+            "fontSize": 14,
+            "textAlign": "left",
+            "url": "https://skywalking.apache.org/docs/main/next/en/setup/backend/backend-nginx-monitoring/"
+          }
+        }
+      ],
+      "id": "nginx-root",
+      "layer": "NGINX",
+      "entity": "All",
+      "name": "Nginx-Root",
+      "isRoot": true,
+      "path": "/Gateway/Nginx"
+    }
+  }
+]
\ No newline at end of file
diff --git a/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
new file mode 100644
index 000000000000..c2f8f57a3710
--- /dev/null
+++ b/oap-server/server-starter/src/main/resources/ui-initialized-templates/nginx/nginx-service.json
@@ -0,0 +1,294 @@
+[
+  {
+    "id": "nginx-service",
+    "configuration": {
+      "children": [
+        {
+          "x": 0,
+          "y": 0,
+          "w": 24,
+          "h": 39,
+          "i": "0",
+          "type": "Tab",
+          "children": [
+            {
+              "name": "Overview",
+              "children": [
+                {
+                  "x": 0,
+                  "y": 0,
+                  "w": 6,
+                  "h": 13,
+                  "i": "0",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "HTTP Request Trend (requests per second)"
+                  },
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "meter_nginx_service_http_requests"
+                  ],
+                  "metricMode": "Expression",
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ]
+                },
+                {
+                  "x": 6,
+                  "y": 0,
+                  "w": 6,
+                  "h": 13,
+                  "i": "1",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "HTTP Latency (ms)"
+                  },
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "meter_nginx_service_http_latency"
+                  ],
+                  "metricMode": "Expression",
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ]
+                },
+                {
+                  "x": 12,
+                  "y": 0,
+                  "w": 6,
+                  "h": 13,
+                  "i": "2",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "HTTP Bandwidth (KB/s)"
+                  },
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "meter_nginx_service_http_bandwidth/1024"
+                  ],
+                  "metricMode": "Expression",
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ]
+                },
+                {
+                  "x": 18,
+                  "y": 0,
+                  "w": 6,
+                  "h": 13,
+                  "i": "4",
+                  "type": "Widget",
+                  "expressions": [
+                    "meter_nginx_service_http_connections"
+                  ],
+                  "metricMode": "Expression",
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ],
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "HTTP Connections"
+                  }
+                },
+                {
+                  "x": 0,
+                  "y": 13,
+                  "w": 6,
+                  "h": 13,
+                  "i": "5",
+                  "type": "Widget",
+                  "widget": {
+                    "title": "HTTP Status Trend (requests per second)"
+                  },
+                  "graph": {
+                    "type": "Line",
+                    "step": false,
+                    "smooth": false,
+                    "showSymbol": true,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "expressions": [
+                    "meter_nginx_service_http_status"
+                  ],
+                  "metricMode": "Expression",
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ]
+                },
+                {
+                  "x": 6,
+                  "y": 13,
+                  "w": 6,
+                  "h": 13,
+                  "i": "6",
+                  "type": "Widget",
+                  "metricMode": "Expression",
+                  "expressions": [
+                    "(meter_nginx_service_http_4xx_requests_increment/meter_nginx_service_http_requests_increment)*100"
+                  ],
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ],
+                  "graph": {
+                    "type": "Area",
+                    "opacity": 0.4,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "HTTP Status 4xx Percent Per Minute (%)"
+                  }
+                },
+                {
+                  "x": 12,
+                  "y": 13,
+                  "w": 6,
+                  "h": 13,
+                  "i": "7",
+                  "type": "Widget",
+                  "metricMode": "Expression",
+                  "expressions": [
+                    "(meter_nginx_service_http_5xx_requests_increment/meter_nginx_service_http_requests_increment)*100"
+                  ],
+                  "typesOfMQE": [
+                    "TIME_SERIES_VALUES"
+                  ],
+                  "graph": {
+                    "type": "Area",
+                    "opacity": 0.4,
+                    "showXAxis": true,
+                    "showYAxis": true
+                  },
+                  "widget": {
+                    "title": "HTTP Status 5xx Percent Per Minute (%)"
+                  }
+                },
+                {
+                  "x": 18,
+                  "y": 13,
+                  "w": 6,
+                  "h": 13,
+                  "i": "8",
+                  "type": "Widget",
+                  "metricMode": "Expression",
+                  "expressions": [
+                    "meter_nginx_service_error_log_count"
+                  ],
+                  "graph": {
+                    "type": "Bar",
+                    "showBackground": true
+                  },
+                  "widget": {
+                    "title": "Error Log Count (Per Minute)"
+                  },
+                  "typesOfMQE": [
+                    "UNKNOWN"
+                  ]
+                }
+              ]
+            },
+            {
+              "name": "Instance",
+              "children": [
+                {
+                  "x": 0,
+                  "y": 0,
+                  "w": 24,
+                  "h": 35,
+                  "i": "0",
+                  "type": "Widget",
+                  "metricMode": "Expression",
+                  "expressions": [
+                    ""
+                  ],
+                  "typesOfMQE": [
+                    ""
+                  ],
+                  "graph": {
+                    "type": "InstanceList",
+                    "dashboardName": "Nginx-Instance",
+                    "fontSize": 12
+                  }
+                }
+              ]
+            },
+            {
+              "name": "Route",
+              "children": [
+                {
+                  "x": 0,
+                  "y": 0,
+                  "w": 24,
+                  "h": 35,
+                  "i": "0",
+                  "type": "Widget",
+                  "metricMode": "Expression",
+                  "expressions": [
+                    ""
+                  ],
+                  "typesOfMQE": [
+                    ""
+                  ],
+                  "graph": {
+                    "type": "EndpointList",
+                    "dashboardName": "Nginx-Endpoint",
+                    "fontSize": 12,
+                    "showXAxis": false,
+                    "showYAxis": false
+                  }
+                }
+              ]
+            },
+            {
+              "name": "Log",
+              "children": [
+                {
+                  "x": 0,
+                  "y": 0,
+                  "w": 24,
+                  "h": 35,
+                  "i": "0",
+                  "type": "Log"
+                }
+              ]
+            }
+          ]
+        }
+      ],
+      "layer": "NGINX",
+      "entity": "Service",
+      "name": "Nginx-Service",
+      "id": "nginx-service",
+      "isRoot": false
+    }
+  }
+]
\ No newline at end of file


Test patch:
diff --git a/test/e2e-v2/cases/nginx/Dockerfile.nginx b/test/e2e-v2/cases/nginx/Dockerfile.nginx
new file mode 100644
index 000000000000..75db26438d2c
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/Dockerfile.nginx
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM openresty/openresty:1.17.8.2-5-alpine-fat
+
+RUN luarocks install nginx-lua-prometheus
+RUN mkdir -p /var/log/nginx/
\ No newline at end of file
diff --git a/test/e2e-v2/cases/nginx/docker-compose.yml b/test/e2e-v2/cases/nginx/docker-compose.yml
new file mode 100644
index 000000000000..64d57a363299
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/docker-compose.yml
@@ -0,0 +1,72 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: '2.1'
+
+services:
+  oap:
+    extends:
+      file: ../../script/docker-compose/base-compose.yml
+      service: oap
+    ports:
+      - 12800
+    networks:
+      - e2e
+
+  nginx:
+    build:
+      context: .
+      dockerfile: Dockerfile.nginx
+    volumes:
+      - ./nginx.conf:/var/nginx/conf.d/nginx.conf
+      - /tmp/skywalking-logs/nginx:/var/log/nginx
+    entrypoint: ['bash', '-c', 'sleep 5 && openresty -c /var/nginx/conf.d/nginx.conf']
+    depends_on:
+      oap:
+        condition: service_healthy
+    ports:
+      - 8080
+    expose:
+      - 9145
+    networks:
+      - e2e
+
+  fluent-bit:
+    image: fluent/fluent-bit:1.7
+    volumes:
+      - /tmp/skywalking-logs/nginx:/var/log/nginx
+      - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
+      - ./fluent-bit-script.lua:/fluent-bit/etc/fluent-bit-script.lua
+    depends_on:
+      - oap
+      - nginx
+    networks:
+      - e2e
+
+  otel-collector:
+    image: otel/opentelemetry-collector:0.50.0
+    command: [ "--config=/etc/otel-collector-config.yaml" ]
+    volumes:
+      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
+    depends_on:
+      - oap
+      - nginx
+    expose:
+      - 55678
+    networks:
+      e2e:
+
+networks:
+  e2e:
diff --git a/test/e2e-v2/cases/nginx/e2e.yaml b/test/e2e-v2/cases/nginx/e2e.yaml
new file mode 100644
index 000000000000..16c0488089af
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/e2e.yaml
@@ -0,0 +1,96 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# This file is used to show how to write configuration files and can be used to test.
+
+setup:
+  env: compose
+  file: docker-compose.yml
+  timeout: 20m
+  init-system-environment: ../../script/env
+  steps:
+    - name: set PATH
+      command: export PATH=/tmp/skywalking-infra-e2e/bin:$PATH
+    - name: install yq
+      command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh yq
+    - name: install swctl
+      command: bash test/e2e-v2/script/prepare/setup-e2e-shell/install.sh swctl
+
+trigger:
+  action: http
+  interval: 3s
+  times: 10
+  url: http://${nginx_host}:${nginx_8080}/test
+  method: POST
+
+verify:
+  # verify with retry strategy
+  retry:
+    # max retry count
+    count: 20
+    # the interval between two retries, in millisecond.
+    interval: 3s
+  cases:
+    # service cases
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql service ls
+      expected: expected/service.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_requests --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_latency --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_bandwidth --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_connections --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_status --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_http_requests_increment --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_service_error_log_count --service-name=nginx::e2e-test
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql logs list --service-name=nginx::e2e-test
+      expected: expected/logs.yml
+
+    # service instance cases
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql instance list --service-name=nginx::e2e-test
+      expected: expected/service-instance.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_instance_http_requests --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/metrics-has-value.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_instance_http_latency --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_instance_http_bandwidth --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_instance_http_status --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_instance_http_requests_increment --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/metrics-has-value.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_instance_error_log_count --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql logs list --service-name=nginx::e2e-test --instance-name=e2e-test-instance
+      expected: expected/logs.yml
+
+    # service endpoint cases
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql endpoint list --service-name=nginx::e2e-test
+      expected: expected/service-endpoint.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_endpoint_http_requests --service-name=nginx::e2e-test --endpoint-name=/test/**
+      expected: expected/metrics-has-value.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_endpoint_http_latency --service-name=nginx::e2e-test --endpoint-name=/test/**
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_endpoint_http_bandwidth --service-name=nginx::e2e-test --endpoint-name=/test/**
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_endpoint_http_status --service-name=nginx::e2e-test --endpoint-name=/test/**
+      expected: expected/metrics-has-value-label.yml
+    - query: swctl --display yaml --base-url=http://${oap_host}:${oap_12800}/graphql metrics exec --expression=meter_nginx_endpoint_http_requests_increment --service-name=nginx::e2e-test --endpoint-name=/test/**
+      expected: expected/metrics-has-value.yml
\ No newline at end of file
diff --git a/test/e2e-v2/cases/nginx/expected/logs.yml b/test/e2e-v2/cases/nginx/expected/logs.yml
new file mode 100644
index 000000000000..6ea07a4b7584
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/logs.yml
@@ -0,0 +1,34 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+errorreason: null
+logs:
+{{- contains .logs }}
+- servicename: nginx::e2e-test
+  serviceid: {{ b64enc "nginx::e2e-test" }}.1
+  serviceinstancename: e2e-test-instance
+  serviceinstanceid: {{ b64enc "nginx::e2e-test" }}.1_{{ b64enc "e2e-test-instance" }}
+  endpointname: null
+  endpointid: null
+  traceid: null
+  timestamp: {{ gt .timestamp 0 }}
+  contenttype: TEXT
+  content: {{ .content }}
+  tags:
+  {{- contains .tags }}
+  - key: LOG_KIND
+    value: {{ .value }}
+  {{- end }}
+{{- end }}
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value-label.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label.yml
new file mode 100644
index 000000000000..a94c3c4ba00c
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value-label.yml
@@ -0,0 +1,35 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels:
+        {{- contains .metric.labels }}
+        - key: _
+          value: {{ .value }}
+        {{- end}}
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/nginx/expected/metrics-has-value.yml b/test/e2e-v2/cases/nginx/expected/metrics-has-value.yml
new file mode 100644
index 000000000000..dc71f566321a
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/metrics-has-value.yml
@@ -0,0 +1,31 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+type: TIME_SERIES_VALUES
+results:
+  {{- contains .results }}
+  - metric:
+      labels: []
+    values:
+      {{- contains .values }}
+      - id: {{ notEmpty .id }}
+        value: {{ .value }}
+        traceid: null
+      - id: {{ notEmpty .id }}
+        value: null
+        traceid: null
+      {{- end}}
+  {{- end}}
+error: null
diff --git a/test/e2e-v2/cases/nginx/expected/service-endpoint.yml b/test/e2e-v2/cases/nginx/expected/service-endpoint.yml
new file mode 100644
index 000000000000..796c7ab57735
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/service-endpoint.yml
@@ -0,0 +1,19 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains .}}
+- id: {{ b64enc "nginx::e2e-test" }}.1_{{ b64enc "/test/**" }}
+  name: /test/**
+{{- end}}
diff --git a/test/e2e-v2/cases/nginx/expected/service-instance.yml b/test/e2e-v2/cases/nginx/expected/service-instance.yml
new file mode 100644
index 000000000000..24ffd2b61b52
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/service-instance.yml
@@ -0,0 +1,22 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains . }}
+- id: {{ notEmpty .id }}
+  name: e2e-test-instance
+  instanceuuid: {{ notEmpty .instanceuuid }}
+  attributes: []
+  language: UNKNOWN
+{{- end }}
diff --git a/test/e2e-v2/cases/nginx/expected/service.yml b/test/e2e-v2/cases/nginx/expected/service.yml
new file mode 100644
index 000000000000..354d3687b70d
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/expected/service.yml
@@ -0,0 +1,24 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+{{- contains . }}
+- id: {{ b64enc "nginx::e2e-test" }}.1
+  name: nginx::e2e-test
+  group: nginx
+  shortname: e2e-test
+  layers:
+    - NGINX
+  normal: true
+{{- end }}
\ No newline at end of file
diff --git a/test/e2e-v2/cases/nginx/fluent-bit-script.lua b/test/e2e-v2/cases/nginx/fluent-bit-script.lua
new file mode 100644
index 000000000000..c02b0a97e753
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/fluent-bit-script.lua
@@ -0,0 +1,36 @@
+--
+-- Licensed to the Apache Software Foundation (ASF) under one or more
+-- contributor license agreements.  See the NOTICE file distributed with
+-- this work for additional information regarding copyright ownership.
+-- The ASF licenses this file to You under the Apache License, Version 2.0
+-- (the "License"); you may not use this file except in compliance with
+-- the License.  You may obtain a copy of the License at
+--
+--     http://www.apache.org/licenses/LICENSE-2.0
+--
+-- Unless required by applicable law or agreed to in writing, software
+-- distributed under the License is distributed on an "AS IS" BASIS,
+-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+-- See the License for the specific language governing permissions and
+-- limitations under the License.
+--
+
+function rewrite_access_log(tag, timestamp, record)
+    local newRecord = {}
+    newRecord["layer"] = "NGINX"
+    newRecord["service"] = "nginx::e2e-test"
+    newRecord["serviceInstance"] = "e2e-test-instance"
+    newRecord["body"] = { text = { text = record.log } }
+    newRecord["tags"] = {data={{key="LOG_KIND", value="NGINX_ACCESS_LOG"}}}
+    return 1, timestamp, newRecord
+end
+
+function rewrite_error_log(tag, timestamp, record)
+    local newRecord = {}
+    newRecord["layer"] = "NGINX"
+    newRecord["service"] = "nginx::e2e-test"
+    newRecord["serviceInstance"] = "e2e-test-instance"
+    newRecord["body"] = { text = { text = record.log } }
+    newRecord["tags"] = {data={{key="LOG_KIND", value="NGINX_ERROR_LOG"}}}
+    return 1, timestamp, newRecord
+end
\ No newline at end of file
diff --git a/test/e2e-v2/cases/nginx/fluent-bit.conf b/test/e2e-v2/cases/nginx/fluent-bit.conf
new file mode 100644
index 000000000000..73b0538fa632
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/fluent-bit.conf
@@ -0,0 +1,48 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+[SERVICE]
+    Flush          5
+    Daemon         Off
+    Log_Level      warn
+[INPUT]
+    Name           tail
+    Tag            access
+    Path           /var/log/nginx/access.log
+[INPUT]
+    Name           tail
+    Tag            error
+    Path           /var/log/nginx/error.log
+[FILTER]
+    Name           lua
+    Match          access
+    Script         fluent-bit-script.lua
+    Call           rewrite_access_log
+[FILTER]
+    Name           lua
+    Match          error
+    Script         fluent-bit-script.lua
+    Call           rewrite_error_log
+[OUTPUT]
+    Name            stdout
+    Match           *
+    Format          json
+[OUTPUT]
+    Name            http
+    Match           *
+    Host            oap
+    Port            12800
+    URI             /v3/logs
+    Format          json
\ No newline at end of file
diff --git a/test/e2e-v2/cases/nginx/nginx.conf b/test/e2e-v2/cases/nginx/nginx.conf
new file mode 100644
index 000000000000..c7fb37abb6b9
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/nginx.conf
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+worker_processes  1;
+daemon off;
+error_log  /var/log/nginx/error.log notice;
+
+events {
+    worker_connections 1024;
+}
+
+http {
+    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
+                      '$status $body_bytes_sent "$http_referer" '
+                      '"$http_user_agent" "$http_x_forwarded_for"';
+
+    access_log  /var/log/nginx/access.log  main;
+
+    lua_shared_dict prometheus_metrics 10M;
+    # lua_package_path "/path/to/nginx-lua-prometheus/?.lua;;";
+
+    init_worker_by_lua_block {
+      prometheus = require("prometheus").init("prometheus_metrics")
+
+      metric_bytes = prometheus:counter(
+        "nginx_http_size_bytes", "Total size of HTTP", {"type", "route"})
+      metric_requests = prometheus:counter(
+        "nginx_http_requests_total", "Number of HTTP requests", {"status", "route"})
+      metric_latency = prometheus:histogram(
+        "nginx_http_latency", "HTTP request latency", {"route"})
+      metric_connections = prometheus:gauge(
+        "nginx_http_connections", "Number of HTTP connections", {"state"})
+    }
+
+    server {
+        listen 8080;
+
+        location /test {
+          default_type application/json;
+          return 200  '{"code": 200, "message": "success"}';
+
+          log_by_lua_block {
+            metric_bytes:inc(tonumber(ngx.var.request_length), {"request", "/test/**"})
+            metric_bytes:inc(tonumber(ngx.var.bytes_send), {"response", "/test/**"})
+            metric_requests:inc(1, {ngx.var.status, "/test/**"})
+            metric_latency:observe(tonumber(ngx.var.request_time), {"/test/**"})
+          }
+        }
+    }
+
+    server {
+      listen 9145;
+      location /metrics {
+        content_by_lua_block {
+          metric_connections:set(ngx.var.connections_reading, {"reading"})
+          metric_connections:set(ngx.var.connections_waiting, {"waiting"})
+          metric_connections:set(ngx.var.connections_writing, {"writing"})
+          prometheus:collect()
+        }
+      }
+    }
+}
diff --git a/test/e2e-v2/cases/nginx/otel-collector-config.yaml b/test/e2e-v2/cases/nginx/otel-collector-config.yaml
new file mode 100644
index 000000000000..eec88fb6c13c
--- /dev/null
+++ b/test/e2e-v2/cases/nginx/otel-collector-config.yaml
@@ -0,0 +1,44 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+receivers:
+  prometheus:
+    config:
+      scrape_configs:
+        - job_name: 'nginx-monitoring'
+          scrape_interval: 10s
+          metrics_path: "/metrics"
+          static_configs:
+            - targets: ['nginx:9145']
+              labels:
+                service: e2e-test
+                service_instance_id: e2e-test-instance
+processors:
+  batch:
+
+exporters:
+  otlp:
+    endpoint: oap:11800
+    tls:
+      insecure: true
+service:
+  pipelines:
+    metrics:
+      receivers:
+        - prometheus
+      processors:
+        - batch
+      exporters:
+        - otlp
\ No newline at end of file