Repository: bentoml/BentoML

Issue:
feature: allow user to specify URLs to exclude from tracing
### Feature request

Currently, if tracing is enabled, traces are collected / sampled from all routes. There should be a way to disable tracing on certain routes, especially when a deployment on Kubernetes may have liveness and readiness probes pinging the deployment frequently. Since we cannot disable tracing on the liveness/readiness probe routes, many redundant traces get collected.

For example, you can see below that a lot of the traces collected are of the liveness and readiness probe, and only one of them has a useful trace for the predict route.
![image](https://user-images.githubusercontent.com/12453748/181473003-79c7c9e5-9fa9-4b15-8b71-c727d4cbb5c2.png)

For other opentelemetry instrumentations, we can use an environment variable like `OTEL_PYTHON_FASTAPI_EXCLUDED_URLS` to specify routes to exclude from tracing, we should be able to do something similar.

### Motivation

_No response_

### Other

_No response_


Discussion:
I think your proposal makes sense. We would love to have you contribute this feature 🚀 
Alright, will try to work on it

Resolving patch:
diff --git a/bentoml/_internal/configuration/containers.py b/bentoml/_internal/configuration/containers.py
--- a/bentoml/_internal/configuration/containers.py
+++ b/bentoml/_internal/configuration/containers.py
@@ -116,6 +116,7 @@ def _is_ip_address(addr: str) -> bool:
         "tracing": {
             "type": Or(And(str, Use(str.lower), _check_tracing_type), None),
             "sample_rate": Or(And(float, lambda i: i >= 0 and i <= 1), None),
+            "excluded_urls": Or(str, None),
             Optional("zipkin"): {"url": Or(str, None)},
             Optional("jaeger"): {"address": Or(str, None), "port": Or(int, None)},
         },
@@ -408,6 +409,15 @@ def tracer_provider(
         else:
             return provider
 
+    @providers.SingletonFactory
+    @staticmethod
+    def tracing_excluded_urls(
+        excluded_urls: t.Optional[str] = Provide[config.tracing.excluded_urls],
+    ):
+        from opentelemetry.util.http import parse_excluded_urls
+
+        return parse_excluded_urls(excluded_urls)
+
     # Mapping from runner name to RunnerApp file descriptor
     remote_runner_mapping = providers.Static[t.Dict[str, str]]({})
     plasma_db = providers.Static[t.Optional["ext.PlasmaClient"]](None)
diff --git a/bentoml/_internal/server/runner_app.py b/bentoml/_internal/server/runner_app.py
--- a/bentoml/_internal/server/runner_app.py
+++ b/bentoml/_internal/server/runner_app.py
@@ -134,7 +134,7 @@ def client_request_hook(span: Span, _scope: t.Dict[str, t.Any]) -> None:
         middlewares.append(
             Middleware(
                 otel_asgi.OpenTelemetryMiddleware,
-                excluded_urls=None,
+                excluded_urls=BentoMLContainer.tracing_excluded_urls.get(),
                 default_span_details=None,
                 server_request_hook=None,
                 client_request_hook=client_request_hook,
diff --git a/bentoml/_internal/server/service_app.py b/bentoml/_internal/server/service_app.py
--- a/bentoml/_internal/server/service_app.py
+++ b/bentoml/_internal/server/service_app.py
@@ -218,7 +218,7 @@ def client_request_hook(span: Span, _scope: dict[str, t.Any]) -> None:
         middlewares.append(
             Middleware(
                 otel_asgi.OpenTelemetryMiddleware,
-                excluded_urls=None,
+                excluded_urls=BentoMLContainer.tracing_excluded_urls.get(),
                 default_span_details=None,
                 server_request_hook=None,
                 client_request_hook=client_request_hook,


Test patch: