Repository: bentoml/BentoML
Pull request: feat: Adaptive batching size histogram metrics 
## What does this PR address?
<!--
Thanks for sending a pull request!

Congrats for making it this far! Here's a 🍱 for you. There are still a few steps ahead.

Please make sure to read the contribution guidelines, then fill out the blanks below before requesting a code review.

Name your Pull Request with one of the following prefixes, e.g. "feat: add support for PyTorch", to indicate the type of changes proposed. This is based on the [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/#summary).
  - feat: (new feature for the user, not a new feature for build script)
  - fix: (bug fix for the user, not a fix to a build script)
  - docs: (changes to the documentation)
  - style: (formatting, missing semicolons, etc; no production code change)
  - refactor: (refactoring production code, eg. renaming a variable)
  - perf: (code changes that improve performance)
  - test: (adding missing tests, refactoring tests; no production code change)
  - chore: (updating grunt tasks etc; no production code change)
  - build: (changes that affect the build system or external dependencies)
  - ci: (changes to configuration files and scripts)
  - revert: (reverts a previous commit)

Describe your changes in detail. Attach screenshots here if appropriate.

Once you're done with this, someone from BentoML team or community member will help review your PR (see "Who can help review?" section for potential reviewers.). If no one has reviewed your PR after a week have passed, don't hesitate to post a new comment and ping @-the same person. Notifications sometimes get lost 🥲.
-->

This PR introduces three changes. Fixes [#1499](https://github.com/bentoml/BentoML/issues/1499).

1. Adds a histogram metric for adaptive batching size. The histogram used is an exponential histogram from 1 to `max_batch_size`. The assumption made here is that the adaptive batching size histogram buckets are filled more evenly with exponential bounds.

![Screen Shot 2022-08-12 at 22 49 57](https://user-images.githubusercontent.com/861225/184470659-1a07b83b-63a9-4e0c-9141-bdc6258270df.png)

2. In addition, this PR also adds a configuration for generating request duration histogram buckets. Currently, request duration buckets use the [default buckets](https://github.com/prometheus/client_python/blob/master/prometheus_client/metrics.py#L544) from prometheus client. With the max bucket size at 10, the histogram is unable to track latency greater 10 seconds. This change adds three keys for configuring the duration histogram.
    - min: the smallest upper bound
    - max: the largest upper bound
    - factor: the exponential factor used to generate the buckets

    For example, with `min=0.005`, `max=10.0`, and `factor=2`, buckets generated is `(0.005, 0.01, 0.02, 0.04, 0.08, 0.16, 
    0.32, 0.64, 1.28, 2.56, 5.12, 10.0, INF)`. We also assume request duration follows a long tail pattern.

3. Lastly, this PR fixes the access log to output request duration in the correct time unit. Previously, the access outputs latency in milliseconds but the value is tracked in seconds. With this PR, the value is displayed accurately.

    Before:
    ```
    2022-08-12T22:51:10-0700 [INFO] [api_server:5] 127.0.0.1:64649 (scheme=http,method=POST,path=/classify,type=application/json,length=132) (status=200,type=application/json,length=18) 0.003ms (trace=270554840034756658801751152705387155180,span=17828550978915614742,sampled=0)
    ```
    After:
    ```
    2022-08-12T22:51:10-0700 [INFO] [api_server:5] 127.0.0.1:64649 (scheme=http,method=POST,path=/classify,type=application/json,length=132) (status=200,type=application/json,length=18) 3.458ms (trace=270554840034756658801751152705387155180,span=17828550978915614742,sampled=0)
    ```


## Before submitting:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
<!--- If you plan to update documentation or tests in follow-up, please note -->
- [ ] Does the Pull Request follow [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/#summary) naming? Here are [GitHub's
guide](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) on how to create a pull request.
- [ ] Does the code follow BentoML's code style, both `make format` and `make lint` script have passed ([instructions](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md#style-check-auto-formatting-type-checking))?
- [ ] Did you read through [contribution guidelines](https://github.com/bentoml/BentoML/blob/main/CONTRIBUTING.md#ways-to-contribute) and follow [development guidelines](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md#start-developing)?
- [ ] Did your changes require updates to the documentation? Have you updated
  those accordingly? Here are [documentation guidelines](https://github.com/bentoml/BentoML/tree/main/docs) and [tips on writting docs](https://github.com/bentoml/BentoML/tree/main/docs#writing-documentation).
- [ ] Did you write tests to cover your changes?

## Who can help review?

Feel free to tag members/contributors who can help review your PR.
<!--
Feel free to ping any of the BentoML members for help on your issue, but don't ping more than three people 😊.
If you know how to use git blame, that is probably the easiest way.

Team members that you can ping:
- @parano
- @yubozhao
- @bojiang
- @ssheng
- @aarnphm
- @sauyon
- @larme
- @yetone
- @jjmachan
-->


Problem statement:
Add more debug/info logs runner indicating adaptive batching logic
**Is your feature request related to a problem? Please describe.**
<!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->

Currently there is not too much visibility for the user to visualize or to understand how BentoML's micro-batching works. having more logs around how BentoML makes a decision and what is happening in the batching layer should help with that.

cc @bojiang 

**Describe the solution you'd like**
<!-- A clear and concise description of what you want to happen. -->

**Describe alternatives you've considered**
<!-- A clear and concise description of any alternative solutions or features you've considered. -->

**Additional context**
<!-- Add any other context or screenshots about the feature request here. -->

Hints from issue discussion:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

In debug mode, RunnerApp should:

* log the batch size for each run
* log the latency for each run

Commit message: format

Patch to review:
diff --git a/bentoml/_internal/configuration/containers.py b/bentoml/_internal/configuration/containers.py
index a0d92c77013..baf9b52cebf 100644
--- a/bentoml/_internal/configuration/containers.py
+++ b/bentoml/_internal/configuration/containers.py
@@ -45,10 +45,10 @@
 logger = logging.getLogger(__name__)
 
 _check_tracing_type: t.Callable[[str], bool] = lambda s: s in ("zipkin", "jaeger")
-_larger_than: t.Callable[[int], t.Callable[[int], bool]] = (
+_larger_than: t.Callable[[int | float], t.Callable[[int | float], bool]] = (
     lambda target: lambda val: val > target
 )
-_larger_than_zero: t.Callable[[int], bool] = _larger_than(0)
+_larger_than_zero: t.Callable[[int | float], bool] = _larger_than(0)
 
 
 def _is_ip_address(addr: str) -> bool:
@@ -100,7 +100,15 @@ def _is_ip_address(addr: str) -> bool:
                 Optional("ca_certs"): Or(str, None),
                 Optional("ciphers"): Or(str, None),
             },
-            "metrics": {"enabled": bool, "namespace": str},
+            "metrics": {
+                "enabled": bool,
+                "namespace": str,
+                Optional("duration"): {
+                    Optional("min"): And(float, _larger_than_zero),
+                    Optional("max"): And(float, _larger_than_zero),
+                    Optional("factor"): And(float, _larger_than(1.0)),
+                },
+            },
             "logging": {
                 # TODO add logging level configuration
                 "access": {
@@ -434,5 +442,32 @@ def tracing_excluded_urls(
     remote_runner_mapping = providers.Static[t.Dict[str, str]]({})
     plasma_db = providers.Static[t.Optional["ext.PlasmaClient"]](None)
 
+    @providers.SingletonFactory
+    @staticmethod
+    def duration_buckets(
+        metrics: dict[str, t.Any] = Provide[config.api_server.metrics],
+    ) -> tuple[float, ...]:
+        """
+        Returns a tuple of duration buckets in seconds. If not explicitly configured,
+        the Prometheus default is returned; otherwise, a set of exponential buckets
+        generated based on the configuration is returned.
+        """
+        from ..utils.metrics import DEFAULT_BUCKET
+        from ..utils.metrics import exponential_buckets
+
+        if "duration" not in metrics:
+            return DEFAULT_BUCKET
+        else:
+            duration: dict[str, float] = metrics["duration"]
+            if duration.keys() >= {"min", "max", "factor"}:
+                return exponential_buckets(
+                    duration["min"], duration["factor"], duration["max"]
+                )
+            else:
+                raise BentoMLConfigException(
+                    "Keys 'min', 'max', and 'factor' are required for "
+                    f"'duration' configuration, '{duration}'."
+                )
+
 
 BentoMLContainer = _BentoMLContainerClass()
diff --git a/bentoml/_internal/marshal/dispatcher.py b/bentoml/_internal/marshal/dispatcher.py
index 67a031adf2d..50b146c29ad 100644
--- a/bentoml/_internal/marshal/dispatcher.py
+++ b/bentoml/_internal/marshal/dispatcher.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import time
 import typing as t
 import asyncio
@@ -5,11 +7,20 @@
 import functools
 import traceback
 import collections
+from typing import TYPE_CHECKING
 
 import numpy as np
+from simple_di import inject
+from simple_di import Provide
 
 from ..utils import cached_property
 from ..utils.alg import TokenBucket
+from ..utils.metrics import metric_name
+from ..utils.metrics import exponential_buckets
+from ..configuration.containers import BentoMLContainer
+
+if TYPE_CHECKING:
+    from ..server.metrics.prometheus import PrometheusClient
 
 logger = logging.getLogger(__name__)
 
@@ -96,12 +107,17 @@ class CorkDispatcher:
     The wrapped function should be an async function.
     """
 
+    @inject
     def __init__(
         self,
+        runner_name: str,
+        worker_index: int,
+        method_name: str,
         max_latency_in_ms: int,
         max_batch_size: int,
         shared_sema: t.Optional[NonBlockSema] = None,
         fallback: t.Optional[t.Callable[[], t.Any]] = None,
+        metrics_client: "PrometheusClient" = Provide[BentoMLContainer.metrics_client],
     ):
         """
         params:
@@ -123,6 +139,15 @@ def __init__(
         self._queue = collections.deque()  # TODO(hrmthw): maxlen
         self._sema = shared_sema if shared_sema else NonBlockSema(1)
 
+        self.adaptive_batch_size_hist = metrics_client.Histogram(
+            name=metric_name(
+                runner_name, worker_index, method_name, "adaptive_batch_size"
+            ),
+            documentation=runner_name + " Runner adaptive batch size",
+            labelnames=[],  # TODO: add service version
+            buckets=exponential_buckets(1, 2, max_batch_size),
+        )
+
     def shutdown(self):
         if self._controller is not None:
             self._controller.cancel()
@@ -219,7 +244,9 @@ async def inbound_call(self, data):
     async def outbound_call(self, inputs_info):
         _time_start = time.time()
         _done = False
-        logger.debug("Dynamic batching cork released, batch size: %d", len(inputs_info))
+        batch_size = len(inputs_info)
+        logger.debug("Dynamic batching cork released, batch size: %d", batch_size)
+        self.adaptive_batch_size_hist.observe(batch_size)
         try:
             outputs = await self.callback(tuple(d for _, d, _ in inputs_info))
             assert len(outputs) == len(inputs_info)
diff --git a/bentoml/_internal/server/access.py b/bentoml/_internal/server/access.py
index ac87fef9ce9..972fa44c795 100644
--- a/bentoml/_internal/server/access.py
+++ b/bentoml/_internal/server/access.py
@@ -109,7 +109,7 @@ async def wrapped_send(message: "ext.ASGIMessage") -> None:
                 if self.has_response_content_length:
                     response.append(f"length={response_content_length.get().decode()}")
 
-                latency = max(default_timer() - start, 0)
+                latency = max(default_timer() - start, 0) * 1000
 
                 self.logger.info(
                     "%s (%s) (%s) %.3fms",
diff --git a/bentoml/_internal/server/instruments.py b/bentoml/_internal/server/instruments.py
index 1e5edbf736c..6b381a02c3e 100644
--- a/bentoml/_internal/server/instruments.py
+++ b/bentoml/_internal/server/instruments.py
@@ -1,3 +1,5 @@
+from __future__ import annotations
+
 import logging
 import contextvars
 from timeit import default_timer
@@ -6,7 +8,8 @@
 from simple_di import inject
 from simple_di import Provide
 
-from ..configuration.containers import BentoMLContainer
+from bentoml._internal.utils.metrics import metric_name
+from bentoml._internal.configuration.containers import BentoMLContainer
 
 if TYPE_CHECKING:
     from .. import external_typing as ext
@@ -33,25 +36,26 @@ def __init__(
     def _setup(
         self,
         metrics_client: "PrometheusClient" = Provide[BentoMLContainer.metrics_client],
+        duration_buckets: tuple[float, ...] = Provide[
+            BentoMLContainer.duration_buckets
+        ],
     ):
         self.metrics_client = metrics_client
         service_name = self.bento_service.name
-        # a valid tag name may includes invalid characters, so we need to escape them
-        # ref: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
-        service_name = service_name.replace("-", ":").replace(".", "::")
 
         self.metrics_request_duration = metrics_client.Histogram(
-            name=service_name + "_request_duration_seconds",
-            documentation=service_name + " API HTTP request duration in seconds",
+            name=metric_name(service_name, "request_duration_seconds"),
+            documentation="API HTTP request duration in seconds",
             labelnames=["endpoint", "service_version", "http_response_code"],
+            buckets=duration_buckets,
         )
         self.metrics_request_total = metrics_client.Counter(
-            name=service_name + "_request_total",
+            name=metric_name(service_name, "request_total"),
             documentation="Total number of HTTP requests",
             labelnames=["endpoint", "service_version", "http_response_code"],
         )
         self.metrics_request_in_progress = metrics_client.Gauge(
-            name=service_name + "_request_in_progress",
+            name=metric_name(service_name, "request_in_progress"),
             documentation="Total number of HTTP requests in progress now",
             labelnames=["endpoint", "service_version"],
             multiprocess_mode="livesum",
diff --git a/bentoml/_internal/server/runner_app.py b/bentoml/_internal/server/runner_app.py
index 153b8c2dba4..b1ecdaa1ae4 100644
--- a/bentoml/_internal/server/runner_app.py
+++ b/bentoml/_internal/server/runner_app.py
@@ -8,14 +8,14 @@
 from typing import TYPE_CHECKING
 from functools import partial
 
-from ..context import trace_context
-from ..runner.utils import Params
-from ..runner.utils import PAYLOAD_META_HEADER
-from ..runner.utils import payload_paramss_to_batch_params
-from ..server.base_app import BaseAppFactory
-from ..runner.container import AutoContainer
-from ..marshal.dispatcher import CorkDispatcher
-from ..configuration.containers import BentoMLContainer
+from bentoml._internal.context import trace_context
+from bentoml._internal.runner.utils import Params
+from bentoml._internal.runner.utils import PAYLOAD_META_HEADER
+from bentoml._internal.runner.utils import payload_paramss_to_batch_params
+from bentoml._internal.server.base_app import BaseAppFactory
+from bentoml._internal.runner.container import AutoContainer
+from bentoml._internal.marshal.dispatcher import CorkDispatcher
+from bentoml._internal.configuration.containers import BentoMLContainer
 
 feedback_logger = logging.getLogger("bentoml.feedback")
 logger = logging.getLogger(__name__)
@@ -49,6 +49,9 @@ def __init__(
             if not method.config.batchable:
                 continue
             self.dispatchers[method.name] = CorkDispatcher(
+                runner_name=runner.name,
+                worker_index=worker_index,
+                method_name=method.name,
                 max_latency_in_ms=method.max_latency_ms,
                 max_batch_size=method.max_batch_size,
                 fallback=TooManyRequests,
diff --git a/bentoml/_internal/utils/metrics.py b/bentoml/_internal/utils/metrics.py
new file mode 100644
index 00000000000..c0e95039d47
--- /dev/null
+++ b/bentoml/_internal/utils/metrics.py
@@ -0,0 +1,82 @@
+from __future__ import annotations
+
+import typing as t
+
+INF = float("inf")
+
+DEFAULT_BUCKET = (
+    0.005,
+    0.01,
+    0.025,
+    0.05,
+    0.075,
+    0.1,
+    0.25,
+    0.5,
+    0.75,
+    1.0,
+    2.5,
+    5.0,
+    7.5,
+    10.0,
+    INF,
+)
+
+MAX_BUCKET_COUNT = 100
+
+
+def metric_name(*args: t.Any) -> str:
+    """
+    Concatenates the given parts into a legal Prometheus metric name. For example,
+    a valid tag name may includes invalid characters, so we need to escape them
+    ref: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
+    """
+    return "_".join([str(arg).replace("-", ":").replace(".", "::") for arg in args])  # type: ignore
+
+
+def exponential_buckets(start: float, factor: float, end: float) -> tuple[float, ...]:
+    """
+    Creates buckets of a Prometheus histogram where the lowest bucket has an upper
+    bound of start and the upper bound of each following bucket is factor times the
+    previous buckets upper bound. The return tuple include the end as the second
+    last value and positive infinity as the last value.
+    """
+
+    assert start > 0.0
+    assert start < end
+    assert factor > 1.0
+
+    bound = start
+    buckets: list[float] = []
+    while bound < end:
+        buckets.append(bound)
+        bound *= factor
+
+    if len(buckets) > MAX_BUCKET_COUNT:
+        buckets = buckets[:MAX_BUCKET_COUNT]
+
+    return tuple(buckets) + (end, INF)
+
+
+def linear_buckets(start: float, step: float, end: float) -> tuple[float, ...]:
+    """
+    Creates buckets of a Prometheus histogram where the lowest bucket has an upper
+    bound of start and the upper bound of each following bucket is the previous
+    buckets upper bound plus step. The return tuple include the end as the second
+    last value and positive infinity as the last value.
+    """
+
+    assert start > 0.0
+    assert start < end
+    assert step > 0.0
+
+    bound = start
+    buckets: list[float] = []
+    while bound < end:
+        buckets.append(bound)
+        bound += step
+
+    if len(buckets) > MAX_BUCKET_COUNT:
+        buckets = buckets[:MAX_BUCKET_COUNT]
+
+    return tuple(buckets) + (end, INF)
diff --git a/tests/unit/_internal/utils/test_metrics.py b/tests/unit/_internal/utils/test_metrics.py
new file mode 100644
index 00000000000..ed480a7a15e
--- /dev/null
+++ b/tests/unit/_internal/utils/test_metrics.py
@@ -0,0 +1,200 @@
+from __future__ import annotations
+
+import math
+
+from bentoml._internal.utils.metrics import INF
+from bentoml._internal.utils.metrics import metric_name
+from bentoml._internal.utils.metrics import linear_buckets
+from bentoml._internal.utils.metrics import exponential_buckets
+
+
+def _assert_equal(actual: tuple[float, ...], expected: tuple[float, ...]):
+    assert len(actual) == len(expected)
+    for i in range(len(actual)):
+        assert math.isclose(actual[i], expected[i])
+
+
+def test_exponential_buckets():
+    _assert_equal(
+        exponential_buckets(0.005, 2.0, 10.0),
+        (0.005, 0.01, 0.02, 0.04, 0.08, 0.16, 0.32, 0.64, 1.28, 2.56, 5.12, 10.0, INF),
+    )
+    _assert_equal(
+        exponential_buckets(1, 1.5, 100.0),
+        (
+            1,
+            1.5,
+            2.25,
+            3.375,
+            5.0625,
+            7.59375,
+            11.390625,
+            17.0859375,
+            25.62890625,
+            38.443359375,
+            57.6650390625,
+            86.49755859375,
+            100.0,
+            INF,
+        ),
+    )
+    _assert_equal(
+        exponential_buckets(10, 2.5, 1000.0),
+        (10, 25.0, 62.5, 156.25, 390.625, 976.5625, 1000.0, INF),
+    )
+
+    _assert_equal(
+        exponential_buckets(1, 1.1, 100000.0),
+        (
+            1,
+            1.1,
+            1.2100000000000002,
+            1.3310000000000004,
+            1.4641000000000006,
+            1.6105100000000008,
+            1.771561000000001,
+            1.9487171000000014,
+            2.1435888100000016,
+            2.357947691000002,
+            2.5937424601000023,
+            2.853116706110003,
+            3.1384283767210035,
+            3.4522712143931042,
+            3.797498335832415,
+            4.177248169415656,
+            4.594972986357222,
+            5.054470284992944,
+            5.559917313492239,
+            6.115909044841463,
+            6.72749994932561,
+            7.400249944258172,
+            8.140274938683989,
+            8.954302432552389,
+            9.849732675807628,
+            10.834705943388391,
+            11.91817653772723,
+            13.109994191499954,
+            14.420993610649951,
+            15.863092971714948,
+            17.449402268886445,
+            19.19434249577509,
+            21.1137767453526,
+            23.22515441988786,
+            25.54766986187665,
+            28.102436848064315,
+            30.91268053287075,
+            34.003948586157826,
+            37.40434344477361,
+            41.14477778925097,
+            45.25925556817607,
+            49.785181124993684,
+            54.76369923749306,
+            60.24006916124237,
+            66.26407607736661,
+            72.89048368510328,
+            80.17953205361361,
+            88.19748525897498,
+            97.01723378487249,
+            106.71895716335975,
+            117.39085287969573,
+            129.1299381676653,
+            142.04293198443185,
+            156.24722518287504,
+            171.87194770116255,
+            189.05914247127882,
+            207.96505671840671,
+            228.7615623902474,
+            251.63771862927217,
+            276.80149049219943,
+            304.4816395414194,
+            334.9298034955614,
+            368.4227838451175,
+            405.2650622296293,
+            445.79156845259223,
+            490.3707252978515,
+            539.4077978276367,
+            593.3485776104004,
+            652.6834353714405,
+            717.9517789085846,
+            789.7469567994432,
+            868.7216524793876,
+            955.5938177273264,
+            1051.1531995000591,
+            1156.2685194500652,
+            1271.8953713950718,
+            1399.0849085345792,
+            1538.9933993880372,
+            1692.8927393268411,
+            1862.1820132595253,
+            2048.400214585478,
+            2253.240236044026,
+            2478.564259648429,
+            2726.4206856132723,
+            2999.0627541746,
+            3298.96902959206,
+            3628.8659325512663,
+            3991.7525258063934,
+            4390.927778387033,
+            4830.020556225737,
+            5313.0226118483115,
+            5844.324873033143,
+            6428.757360336458,
+            7071.633096370105,
+            7778.796406007116,
+            8556.676046607829,
+            9412.343651268613,
+            10353.578016395475,
+            11388.935818035023,
+            12527.829399838525,
+            100000.0,
+            INF,
+        ),
+    )
+
+
+def test_linear_buckets():
+    _assert_equal(
+        linear_buckets(1.0, 1.0, 10.0),
+        (1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0, 10.0, INF),
+    )
+
+    _assert_equal(
+        linear_buckets(0.1, 0.5, 10.0),
+        (
+            0.1,
+            0.6,
+            1.1,
+            1.6,
+            2.1,
+            2.6,
+            3.1,
+            3.6,
+            4.1,
+            4.6,
+            5.1,
+            5.6,
+            6.1,
+            6.6,
+            7.1,
+            7.6,
+            8.1,
+            8.6,
+            9.1,
+            9.6,
+            10.0,
+            INF,
+        ),
+    )
+
+
+def test_metric_name():
+    assert metric_name("runner_name", "metric_name") == "runner_name_metric_name"
+    assert metric_name("runner.name", "metric_name") == "runner::name_metric_name"
+    assert metric_name("runner-name", "metric_name") == "runner:name_metric_name"
+    assert metric_name("runner_name", "metric.name") == "runner_name_metric::name"
+    assert metric_name("runner_name", "metric-name") == "runner_name_metric:name"
+    assert metric_name("runner_name", 1, "metric_name") == "runner_name_1_metric_name"
+    assert (
+        metric_name("runner_name", 1, "method_name", "metric_name")
+        == "runner_name_1_method_name_metric_name"
+    )


Code review comments:
diff_hunk: @@ -96,12 +107,17 @@ class CorkDispatcher:
     The wrapped function should be an async function.
     """
 
+    @inject
     def __init__(
         self,
+        runner_name: str,
+        worker_index: int,
+        method_name: str,
         max_latency_in_ms: int,
         max_batch_size: int,
         shared_sema: t.Optional[NonBlockSema] = None,
         fallback: t.Optional[t.Callable[[], t.Any]] = None,
+        metrics_client: "PrometheusClient" = Provide[BentoMLContainer.metrics_client],
path: bentoml/_internal/marshal/dispatcher.py
text: @user1:
```suggestion
        metrics_client: PrometheusClient = Provide[BentoMLContainer.metrics_client],
```

diff_hunk: @@ -6,7 +8,8 @@
 from simple_di import inject
 from simple_di import Provide
 
-from ..configuration.containers import BentoMLContainer
+from bentoml._internal.utils.metrics import metric_name
+from bentoml._internal.configuration.containers import BentoMLContainer
path: bentoml/_internal/server/instruments.py
text: @user1:
imo for all `_internal` relative imports should be used.

diff_hunk: @@ -0,0 +1,82 @@
+from __future__ import annotations
+
+import typing as t
+
+INF = float("inf")
+
+DEFAULT_BUCKET = (
+    0.005,
+    0.01,
+    0.025,
+    0.05,
+    0.075,
+    0.1,
+    0.25,
+    0.5,
+    0.75,
+    1.0,
+    2.5,
+    5.0,
+    7.5,
+    10.0,
+    INF,
+)
+
+MAX_BUCKET_COUNT = 100
+
+
+def metric_name(*args: t.Any) -> str:
+    """
+    Concatenates the given parts into a legal Prometheus metric name. For example,
+    a valid tag name may includes invalid characters, so we need to escape them
+    ref: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
+    """
+    return "_".join([str(arg).replace("-", ":").replace(".", "::") for arg in args])  # type: ignore
path: bentoml/_internal/utils/metrics.py
text: @user1:
No need for `type: ignore` here

```suggestion
def metric_name(*args: str) -> str:
    """
    Concatenates the given parts into a legal Prometheus metric name. For example,
    a valid tag name may includes invalid characters, so we need to escape them
    ref: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
    """
    return "_".join([str(arg).replace("-", ":").replace(".", "::") for arg in args])
```

diff_hunk: @@ -434,5 +442,32 @@ def tracing_excluded_urls(
     remote_runner_mapping = providers.Static[t.Dict[str, str]]({})
     plasma_db = providers.Static[t.Optional["ext.PlasmaClient"]](None)
 
+    @providers.SingletonFactory
+    @staticmethod
+    def duration_buckets(
+        metrics: dict[str, t.Any] = Provide[config.api_server.metrics],
+    ) -> tuple[float, ...]:
+        """
+        Returns a tuple of duration buckets in seconds. If not explicitly configured,
+        the Prometheus default is returned; otherwise, a set of exponential buckets
+        generated based on the configuration is returned.
+        """
+        from ..utils.metrics import DEFAULT_BUCKET
+        from ..utils.metrics import exponential_buckets
+
+        if "duration" not in metrics:
+            return DEFAULT_BUCKET
+        else:
+            duration: dict[str, float] = metrics["duration"]
+            if duration.keys() >= {"min", "max", "factor"}:
+                return exponential_buckets(
+                    duration["min"], duration["factor"], duration["max"]
+                )
+            else:
+                raise BentoMLConfigException(
+                    "Keys 'min', 'max', and 'factor' are required for "
+                    f"'duration' configuration, '{duration}'."
+                )
path: bentoml/_internal/configuration/containers.py
text: @user1:
```suggestion
        if "duration" in metrics:
            duration: dict[str, float] = metrics["duration"]
            if duration.keys() >= {"min", "max", "factor"}:
                return exponential_buckets(
                    duration["min"], duration["factor"], duration["max"]
                )
            raise BentoMLConfigException(
                    "Keys 'min', 'max', and 'factor' are required for "
                    f"'duration' configuration, '{duration}'."
                )
        return DEFAULT_BUCKET
```