Repository: bentoml/BentoML

Issue:
bug: Adaptive batching exceeds `max_batch_size` configuration
### Describe the bug

The `max_batch_size` configuration is not being respected in the adaptive batching. If a user sends a request that is already a batch, e.g. `[input0, input1, input2]`, it is considered to be only one input instead of three, see [dispatcher.py](https://github.com/bentoml/BentoML/blob/main/src/bentoml/_internal/marshal/dispatcher.py). Note that the size of the input is simply determined by the queue size, as opposed to the actual input size.

```python
n = len(self._queue)
```

The fix to this issue does not need to further divide a single request if it already exists the `max_batch_size` because it is already loaded in memory.

### To reproduce

Run the quickstart example under [examples/quickstart](https://github.com/bentoml/BentoML/tree/main/examples/quickstart) which has adaptive batching turned on with a configuration to reduce the `max_batch_size` to small enough number, say 10.

```yaml
# configuration.yaml
runners:
  batching:
    max_batch_size: 10
```

Start the server with the configuration.

```sh
BENTOML_CONFIG=./configuration.yaml b serve --production
```

Swarm the server with the locust module included in the example.

I've added custom logging to `sklearn.py`.

```python
def add_runnable_method(method_name: str, options: ModelSignature):
        def _run(
            self: SklearnRunnable, input_data: ext.NpNDArray | ext.PdDataFrame
        ) -> ext.NpNDArray:
            print("Received input size, ", len(input_data))
            # TODO: set inner_max_num_threads and n_jobs param here base on strategy env vars
            with parallel_backend(backend="loky"):
                return getattr(self.model, method_name)(input_data)
```

When traffic is fully ramped, I can see batch size larger than the configured value of 10.

```
Received input size,  12
Received input size,  18
Received input size,  9
```

### Expected behavior

The batch size is kept below the configured `max_batch_size`.

### Environment

#### Environment variable

```bash
BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=True
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''
```

#### System information

`bentoml`: 1.0.16
`python`: 3.10.8
`platform`: macOS-13.1-arm64-arm-64bit
`uid_gid`: 502:20
`conda`: 4.12.0
`in_conda_env`: True
<details><summary><code>conda_packages</code></summary>

<br>

```yaml
name: bentoml
channels:
  - conda-forge
  - defaults
dependencies:
  - anyio=3.6.2=pyhd8ed1ab_0
  - appnope=0.1.3=pyhd8ed1ab_0
  - argon2-cffi=21.3.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py310h1a28f6b_0
  - asttokens=2.2.1=pyhd8ed1ab_0
  - backcall=0.2.0=pyh9f0ad1d_0
  - backports=1.0=pyhd8ed1ab_3
  - backports.functools_lru_cache=1.6.4=pyhd8ed1ab_0
  - bzip2=1.0.8=h620ffc9_4
  - ca-certificates=2023.01.10=hca03da5_0
  - certifi=2022.12.7=py310hca03da5_0
  - cffi=1.15.1=py310h80987f9_3
  - debugpy=1.5.1=py310hc377ac9_0
  - decorator=5.1.1=pyhd8ed1ab_0
  - defusedxml=0.7.1=pyhd8ed1ab_0
  - entrypoints=0.4=pyhd8ed1ab_0
  - executing=1.2.0=pyhd8ed1ab_0
  - flit-core=3.8.0=pyhd8ed1ab_0
  - gettext=0.21.1=h0186832_0
  - giflib=5.2.1=h27ca646_2
  - glib=2.69.1=h514c7bf_2
  - gst-plugins-base=1.14.1=hf0a386a_0
  - gstreamer=1.14.1=he09cfb7_0
  - icu=68.2=hbdafb3b_0
  - idna=3.4=pyhd8ed1ab_0
  - importlib_metadata=6.0.0=hd8ed1ab_0
  - importlib_resources=5.12.0=pyhd8ed1ab_0
  - ipykernel=6.15.0=pyh736e0ef_0
  - ipython_genutils=0.2.0=py_1
  - ipywidgets=8.0.4=pyhd8ed1ab_0
  - jedi=0.18.2=pyhd8ed1ab_0
  - jinja2=3.1.2=pyhd8ed1ab_1
  - jpeg=9e=h1a8c8d9_3
  - jsonschema=4.17.3=pyhd8ed1ab_0
  - jupyter=1.0.0=py_2
  - jupyter_client=8.0.3=pyhd8ed1ab_0
  - jupyter_console=6.5.1=pyhd8ed1ab_0
  - jupyter_core=5.2.0=py310hca03da5_0
  - jupyter_server=1.23.6=pyhd8ed1ab_0
  - jupyterlab_pygments=0.2.2=pyhd8ed1ab_0
  - jupyterlab_widgets=3.0.5=pyhd8ed1ab_0
  - krb5=1.19.3=hf9b2bbe_0
  - lerc=3.0=hc377ac9_0
  - libcxx=14.0.6=h2692d47_0
  - libdeflate=1.8=h1a28f6b_5
  - libedit=3.1.20191231=hc8eb9b7_2
  - libffi=3.4.2=hca03da5_6
  - libiconv=1.17=he4db4b2_0
  - libllvm12=12.0.1=h93073aa_2
  - libpng=1.6.37=hb8d0fd4_0
  - libpq=12.9=h65cfe13_3
  - libsodium=1.0.18=h27ca646_1
  - libtiff=4.4.0=h2fd578a_2
  - libwebp=1.2.4=h328b37c_0
  - libwebp-base=1.2.4=h57fd34a_0
  - libxml2=2.9.14=h8c5e841_0
  - libxslt=1.1.35=h9833966_0
  - lz4-c=1.9.4=hb7217d7_0
  - markupsafe=2.1.1=py310h1a28f6b_0
  - matplotlib-inline=0.1.6=pyhd8ed1ab_0
  - mistune=2.0.5=pyhd8ed1ab_0
  - nbclassic=0.5.2=pyhd8ed1ab_0
  - nbclient=0.7.2=pyhd8ed1ab_0
  - nbconvert=7.2.9=pyhd8ed1ab_0
  - nbconvert-core=7.2.9=pyhd8ed1ab_0
  - nbconvert-pandoc=7.2.9=pyhd8ed1ab_0
  - nbformat=5.7.3=pyhd8ed1ab_0
  - ncurses=6.3=h1a28f6b_3
  - nest-asyncio=1.5.6=pyhd8ed1ab_0
  - notebook=6.5.2=pyha770c72_1
  - notebook-shim=0.2.2=pyhd8ed1ab_0
  - nspr=4.35=hb7217d7_0
  - nss=3.74=h142855e_0
  - openssl=1.1.1t=h1a28f6b_0
  - packaging=23.0=pyhd8ed1ab_0
  - pandoc=2.19.2=hce30654_1
  - pandocfilters=1.5.0=pyhd8ed1ab_0
  - parso=0.8.3=pyhd8ed1ab_0
  - pcre=8.45=hbdafb3b_0
  - pexpect=4.8.0=pyh1a96a4e_2
  - pickleshare=0.7.5=py_1003
  - pip=22.2.2=py310hca03da5_0
  - pkgutil-resolve-name=1.3.10=pyhd8ed1ab_0
  - ply=3.11=py_1
  - prometheus_client=0.16.0=pyhd8ed1ab_0
  - prompt-toolkit=3.0.36=pyha770c72_0
  - prompt_toolkit=3.0.36=hd8ed1ab_0
  - ptyprocess=0.7.0=pyhd3deb0d_0
  - pure_eval=0.2.2=pyhd8ed1ab_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pyqt=5.15.7=py310hc377ac9_0
  - pyrsistent=0.18.0=py310h1a28f6b_0
  - python=3.10.8=hc0d8a6c_1
  - python-dateutil=2.8.2=pyhd8ed1ab_0
  - python-fastjsonschema=2.16.2=pyhd8ed1ab_0
  - qt-main=5.15.2=ha2d02b5_7
  - qt-webengine=5.15.9=h2903aaf_4
  - qtconsole=5.4.0=pyhd8ed1ab_0
  - qtconsole-base=5.4.0=pyha770c72_0
  - qtpy=2.3.0=pyhd8ed1ab_0
  - qtwebkit=5.212=h0f11f3c_4
  - readline=8.2=h1a28f6b_0
  - send2trash=1.8.0=pyhd8ed1ab_0
  - setuptools=65.5.0=py310hca03da5_0
  - sip=6.6.2=py310hc377ac9_0
  - six=1.16.0=pyh6c4a22f_0
  - sniffio=1.3.0=pyhd8ed1ab_0
  - soupsieve=2.3.2.post1=pyhd8ed1ab_0
  - sqlite=3.40.0=h7a7dc30_0
  - stack_data=0.6.2=pyhd8ed1ab_0
  - terminado=0.17.1=pyhd1c38e8_0
  - tinycss2=1.2.1=pyhd8ed1ab_0
  - tk=8.6.12=hb8d0fd4_0
  - toml=0.10.2=pyhd8ed1ab_0
  - tornado=6.2=py310h1a28f6b_0
  - typing-extensions=4.4.0=hd8ed1ab_0
  - typing_extensions=4.4.0=pyha770c72_0
  - tzdata=2022g=h04d1e81_0
  - wheel=0.37.1=pyhd3eb1b0_0
  - widgetsnbextension=4.0.5=pyhd8ed1ab_0
  - xz=5.2.8=h80987f9_0
  - zeromq=4.3.4=hbdafb3b_1
  - zlib=1.2.13=h5a0b063_0
  - zstd=1.5.2=h8574219_0
  - pip:
    - absl-py==1.4.0
    - aiohttp==3.8.3
    - aiosignal==1.3.1
    - alabaster==0.7.12
    - albumentations==1.3.0
    - alembic==1.9.1
    - appdirs==1.4.4
    - arize==5.3.2
    - asgiref==3.5.2
    - astroid==2.14.1
    - astunparse==1.6.3
    - async-timeout==4.0.2
    - attr==0.3.2
    - attrs==22.1.0
    - babel==2.11.0
    - backoff==2.2.1
    - beautifulsoup4==4.11.1
    - bentoml==1.0.16
    - bentoml-plugins-arize==0.1.0
    - black==22.12.0
    - bleach==5.0.1
    - boto3==1.26.24
    - botocore==1.29.24
    - brotli==1.0.9
    - build==0.10.0
    - cachetools==5.3.0
    - cattrs==22.2.0
    - charset-normalizer==2.1.1
    - circus==0.18.0
    - click==8.1.3
    - click-option-group==0.5.5
    - cloudpickle==2.2.0
    - codecov==2.1.12
    - colorama==0.4.6
    - commonmark==0.9.1
    - configargparse==1.5.3
    - contextlib2==21.6.0
    - contourpy==1.0.6
    - coverage==7.2.1
    - cycler==0.11.0
    - databricks-cli==0.17.4
    - deepmerge==1.1.0
    - deprecated==1.2.13
    - dill==0.3.6
    - distlib==0.3.6
    - docker==6.0.1
    - docutils==0.19
    - exceptiongroup==1.0.4
    - execnet==1.9.0
    - filelock==3.8.2
    - filetype==1.2.0
    - flask==2.2.2
    - flask-basicauth==0.2.0
    - flask-cors==3.0.10
    - flatbuffers==23.3.3
    - fonttools==4.38.0
    - frozenlist==1.3.3
    - fs==2.4.16
    - fs-s3fs==1.1.1
    - furo==2022.12.7
    - gast==0.4.0
    - gevent==22.10.2
    - geventhttpclient==2.0.2
    - gitdb==4.0.10
    - gitpython==3.1.30
    - google-auth==2.16.2
    - google-auth-oauthlib==0.4.6
    - google-pasta==0.2.0
    - googleapis-common-protos==1.56.2
    - greenlet==2.0.2
    - grpcio==1.48.1
    - grpcio-channelz==1.48.1
    - grpcio-health-checking==1.48.1
    - grpcio-reflection==1.48.1
    - gunicorn==20.1.0
    - h11==0.14.0
    - h5py==3.8.0
    - huggingface-hub==0.12.0
    - imageio==2.26.0
    - imagesize==1.4.1
    - importlib-metadata==5.1.0
    - iniconfig==1.1.1
    - ipython==8.7.0
    - isort==5.11.4
    - itsdangerous==2.1.2
    - jaraco-classes==3.2.3
    - jmespath==1.0.1
    - joblib==1.2.0
    - kaggle==1.5.12
    - keras==2.11.0
    - keyring==23.11.0
    - kiwisolver==1.4.4
    - lazy-loader==0.1
    - lazy-object-proxy==1.8.0
    - libclang==15.0.6.1
    - livereload==2.6.3
    - llvmlite==0.39.1
    - locust==2.15.0
    - mako==1.2.4
    - markdown==3.4.1
    - markdown-it-py==2.1.0
    - matplotlib==3.6.2
    - mccabe==0.7.0
    - mdit-py-plugins==0.3.3
    - mdurl==0.1.2
    - mlflow==2.1.1
    - more-itertools==9.0.0
    - msgpack==1.0.5
    - mss==7.0.1
    - multidict==6.0.3
    - mypy-extensions==0.4.3
    - myst-parser==0.18.1
    - networkx==3.0
    - numba==0.56.4
    - numpy==1.23.5
    - oauthlib==3.2.2
    - opencv-python==4.7.0.72
    - opencv-python-headless==4.7.0.72
    - opentelemetry-api==1.14.0
    - opentelemetry-exporter-jaeger==1.14.0
    - opentelemetry-exporter-jaeger-proto-grpc==1.14.0
    - opentelemetry-exporter-jaeger-thrift==1.14.0
    - opentelemetry-exporter-otlp==1.14.0
    - opentelemetry-exporter-otlp-proto-grpc==1.14.0
    - opentelemetry-exporter-otlp-proto-http==1.14.0
    - opentelemetry-exporter-zipkin==1.14.0
    - opentelemetry-exporter-zipkin-json==1.14.0
    - opentelemetry-exporter-zipkin-proto-http==1.14.0
    - opentelemetry-instrumentation==0.35b0
    - opentelemetry-instrumentation-aiohttp-client==0.35b0
    - opentelemetry-instrumentation-asgi==0.35b0
    - opentelemetry-instrumentation-grpc==0.35b0
    - opentelemetry-proto==1.14.0
    - opentelemetry-sdk==1.14.0
    - opentelemetry-semantic-conventions==0.35b0
    - opentelemetry-util-http==0.35b0
    - opt-einsum==3.3.0
    - pandas==1.5.2
    - pandas-stubs==1.5.2.230105
    - pathspec==0.10.2
    - pep517==0.13.0
    - pillow==9.3.0
    - pip-requirements-parser==31.2.0
    - pip-tools==6.11.0
    - pkginfo==1.9.2
    - platformdirs==2.5.4
    - pluggy==1.0.0
    - prometheus-client==0.15.0
    - protobuf==3.19.6
    - psutil==5.9.4
    - py4j==0.10.9.5
    - pyarrow==11.0.0
    - pyasn1==0.4.8
    - pyasn1-modules==0.2.8
    - pydantic==1.10.2
    - pyenchant==3.2.2
    - pygments==2.13.0
    - pyjwt==2.6.0
    - pylint==2.16.0
    - pynvml==11.4.1
    - pyparsing==3.0.9
    - pyproject-hooks==1.0.0
    - pyqt5-sip==12.11.0
    - pyspark==3.3.1
    - pytest==7.2.2
    - pytest-asyncio==0.20.3
    - pytest-cov==4.0.0
    - pytest-xdist==3.2.1
    - python-json-logger==2.0.4
    - python-multipart==0.0.5
    - python-rapidjson==1.9
    - python-slugify==8.0.0
    - pytz==2022.6
    - pywavelets==1.4.1
    - pyyaml==6.0
    - pyzmq==24.0.1
    - qudida==0.0.4
    - querystring-parser==1.2.4
    - readme-renderer==37.3
    - regex==2022.10.31
    - requests==2.28.1
    - requests-futures==1.0.0
    - requests-oauthlib==1.3.1
    - requests-toolbelt==0.10.1
    - rfc3986==2.0.0
    - rich==12.6.0
    - roundrobin==0.0.4
    - rsa==4.9
    - ruff==0.0.255
    - s3transfer==0.6.0
    - schema==0.7.5
    - scikit-image==0.20.0
    - scikit-learn==1.1.3
    - scipy==1.9.3
    - seaborn==0.12.2
    - setuptools-scm==7.1.0
    - shap==0.41.0
    - simple-di==0.1.5
    - sklearn==0.0.post1
    - slicer==0.0.7
    - smmap==5.0.0
    - snowballstemmer==2.2.0
    - sphinx==5.3.0
    - sphinx-autobuild==2021.3.14
    - sphinx-basic-ng==1.0.0b1
    - sphinx-click==4.4.0
    - sphinx-copybutton==0.5.1
    - sphinx-design==0.3.0
    - sphinx-hoverxref==1.3.0
    - sphinx-inline-tabs==2022.1.2b11
    - sphinx-issues==3.0.1
    - sphinxcontrib-applehelp==1.0.2
    - sphinxcontrib-devhelp==1.0.2
    - sphinxcontrib-fulltoc==1.2.0
    - sphinxcontrib-htmlhelp==2.0.0
    - sphinxcontrib-jquery==3.0.0
    - sphinxcontrib-jsmath==1.0.1
    - sphinxcontrib-qthelp==1.0.3
    - sphinxcontrib-serializinghtml==1.1.5
    - sphinxcontrib-spelling==7.7.0
    - sphinxext-opengraph==0.7.3
    - sqlalchemy==1.4.46
    - sqlparse==0.4.3
    - starlette==0.25.0
    - tabulate==0.9.0
    - tensorboard==2.11.2
    - tensorboard-data-server==0.6.1
    - tensorboard-plugin-wit==1.8.1
    - tensorflow-estimator==2.11.0
    - tensorflow-macos==2.11.0
    - termcolor==2.2.0
    - text-unidecode==1.3
    - thop==0.1.1-2209072238
    - threadpoolctl==3.1.0
    - thrift==0.16.0
    - tifffile==2023.2.28
    - tokenize-rt==5.0.0
    - tokenizers==0.13.2
    - tomli==2.0.1
    - tomlkit==0.11.6
    - torch==1.13.1
    - torchaudio==0.13.1
    - torchvision==0.14.1
    - tqdm==4.64.1
    - traitlets==5.6.0
    - transformers==4.26.1
    - tritonclient==2.31.0
    - twine==4.0.2
    - types-pytz==2022.7.1.0
    - urllib3==1.26.13
    - uvicorn==0.20.0
    - virtualenv==20.17.1
    - watchfiles==0.18.1
    - wcwidth==0.2.5
    - webencodings==0.5.1
    - websocket-client==1.4.2
    - werkzeug==2.2.2
    - wrapt==1.14.1
    - xgboost==1.7.4
    - yamllint==1.29.0
    - yarl==1.8.2
    - zipp==3.11.0
    - zope-event==4.6
    - zope-interface==5.5.2
prefix: /opt/miniconda3/envs/bentoml
```

</details>

<details><summary><code>pip_packages</code></summary>

<br>

```
absl-py==1.4.0
aiohttp==3.8.3
aiosignal==1.3.1
alabaster==0.7.12
albumentations==1.3.0
alembic==1.9.1
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1666191106763/work/dist
appdirs==1.4.4
appnope @ file:///home/conda/feedstock_root/build_artifacts/appnope_1649077682618/work
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1640817743617/work
argon2-cffi-bindings @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/croot-wbf5edig/argon2-cffi-bindings_1644845754377/work
arize==5.3.2
asgiref==3.5.2
astroid==2.14.1
asttokens @ file:///home/conda/feedstock_root/build_artifacts/asttokens_1670263926556/work
astunparse==1.6.3
async-timeout==4.0.2
attr==0.3.2
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1671632566681/work
Babel==2.11.0
backcall @ file:///home/conda/feedstock_root/build_artifacts/backcall_1592338393461/work
backoff==2.2.1
backports.functools-lru-cache @ file:///home/conda/feedstock_root/build_artifacts/backports.functools_lru_cache_1618230623929/work
beautifulsoup4 @ file:///home/conda/feedstock_root/build_artifacts/beautifulsoup4_1675252249248/work
bentoml==1.0.16
bentoml-plugins-arize==0.1.0
black==22.12.0
bleach @ file:///home/conda/feedstock_root/build_artifacts/bleach_1674535352125/work
boto3==1.26.24
botocore==1.29.24
Brotli==1.0.9
build==0.10.0
cachetools==5.3.0
cattrs==22.2.0
certifi @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_5d968ni_yn/croot/certifi_1671487774636/work/certifi
cffi @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_ab19r4bji3/croot/cffi_1670423206034/work
charset-normalizer==2.1.1
circus==0.18.0
click==8.1.3
click-option-group==0.5.5
cloudpickle==2.2.0
codecov==2.1.12
colorama==0.4.6
commonmark==0.9.1
ConfigArgParse==1.5.3
contextlib2==21.6.0
contourpy==1.0.6
coverage==7.2.1
cycler==0.11.0
databricks-cli==0.17.4
debugpy @ file:///Users/ktietz/ci_310/debugpy_1643965577625/work
decorator @ file:///home/conda/feedstock_root/build_artifacts/decorator_1641555617451/work
deepmerge==1.1.0
defusedxml @ file:///home/conda/feedstock_root/build_artifacts/defusedxml_1615232257335/work
Deprecated==1.2.13
dill==0.3.6
distlib==0.3.6
docker==6.0.1
docutils==0.19
entrypoints @ file:///home/conda/feedstock_root/build_artifacts/entrypoints_1643888246732/work
exceptiongroup==1.0.4
execnet==1.9.0
executing @ file:///home/conda/feedstock_root/build_artifacts/executing_1667317341051/work
fastjsonschema @ file:///home/conda/feedstock_root/build_artifacts/python-fastjsonschema_1663619548554/work/dist
filelock==3.8.2
filetype==1.2.0
Flask==2.2.2
Flask-BasicAuth==0.2.0
Flask-Cors==3.0.10
flatbuffers==23.3.3
flit_core @ file:///home/conda/feedstock_root/build_artifacts/flit-core_1667734568827/work/source/flit_core
fonttools==4.38.0
frozenlist==1.3.3
fs==2.4.16
fs-s3fs==1.1.1
furo==2022.12.7
gast==0.4.0
gevent==22.10.2
geventhttpclient==2.0.2
gitdb==4.0.10
GitPython==3.1.30
google-auth==2.16.2
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
googleapis-common-protos==1.56.2
greenlet==2.0.2
grpcio==1.48.1
grpcio-channelz==1.48.1
grpcio-health-checking==1.48.1
grpcio-reflection==1.48.1
gunicorn==20.1.0
h11==0.14.0
h5py==3.8.0
huggingface-hub==0.12.0
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1663625384323/work
imageio==2.26.0
imagesize==1.4.1
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1672612343532/work
importlib-resources @ file:///home/conda/feedstock_root/build_artifacts/importlib_resources_1676919000169/work
iniconfig==1.1.1
ipykernel @ file:///Users/runner/miniforge3/conda-bld/ipykernel_1655369698656/work
ipython @ file:///Users/runner/miniforge3/conda-bld/ipython_1676047761112/work
ipython-genutils==0.2.0
ipywidgets @ file:///home/conda/feedstock_root/build_artifacts/ipywidgets_1671720089366/work
isort==5.11.4
itsdangerous==2.1.2
jaraco.classes==3.2.3
jedi @ file:///home/conda/feedstock_root/build_artifacts/jedi_1669134318875/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1654302431367/work
jmespath==1.0.1
joblib==1.2.0
jsonschema @ file:///home/conda/feedstock_root/build_artifacts/jsonschema-meta_1669810440410/work
jupyter-console @ file:///home/conda/feedstock_root/build_artifacts/jupyter_console_1676328545892/work
jupyter-server @ file:///home/conda/feedstock_root/build_artifacts/jupyter_server_1676473377907/work
jupyter_client @ file:///home/conda/feedstock_root/build_artifacts/jupyter_client_1676579893731/work
jupyter_core @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_41npretn_h/croot/jupyter_core_1676538590424/work
jupyterlab-pygments @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_pygments_1649936611996/work
jupyterlab-widgets @ file:///home/conda/feedstock_root/build_artifacts/jupyterlab_widgets_1671722028097/work
kaggle==1.5.12
keras==2.11.0
keyring==23.11.0
kiwisolver==1.4.4
lazy-object-proxy==1.8.0
lazy_loader==0.1
libclang==15.0.6.1
livereload==2.6.3
llvmlite==0.39.1
locust==2.15.0
Mako==1.2.4
Markdown==3.4.1
markdown-it-py==2.1.0
MarkupSafe @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_12c133f5-0720-4727-9c18-599a3af825723lzwham3/croots/recipe/markupsafe_1654597866058/work
matplotlib==3.6.2
matplotlib-inline @ file:///home/conda/feedstock_root/build_artifacts/matplotlib-inline_1660814786464/work
mccabe==0.7.0
mdit-py-plugins==0.3.3
mdurl==0.1.2
mistune @ file:///home/conda/feedstock_root/build_artifacts/mistune_1675771498296/work
mlflow==2.1.1
more-itertools==9.0.0
msgpack==1.0.5
mss==7.0.1
multidict==6.0.3
mypy-extensions==0.4.3
myst-parser==0.18.1
nbclassic @ file:///home/conda/feedstock_root/build_artifacts/nbclassic_1676729186918/work
nbclient @ file:///home/conda/feedstock_root/build_artifacts/nbclient_1669795076334/work
nbconvert @ file:///home/conda/feedstock_root/build_artifacts/nbconvert-meta_1674590374792/work
nbformat @ file:///home/conda/feedstock_root/build_artifacts/nbformat_1673560067442/work
nest-asyncio @ file:///home/conda/feedstock_root/build_artifacts/nest-asyncio_1664684991461/work
networkx==3.0
notebook @ file:///home/conda/feedstock_root/build_artifacts/notebook_1667565639349/work
notebook_shim @ file:///home/conda/feedstock_root/build_artifacts/notebook-shim_1667478401171/work
numba==0.56.4
numpy==1.23.5
oauthlib==3.2.2
opencv-python==4.7.0.72
opencv-python-headless==4.7.0.72
opentelemetry-api==1.14.0
opentelemetry-exporter-jaeger==1.14.0
opentelemetry-exporter-jaeger-proto-grpc==1.14.0
opentelemetry-exporter-jaeger-thrift==1.14.0
opentelemetry-exporter-otlp==1.14.0
opentelemetry-exporter-otlp-proto-grpc==1.14.0
opentelemetry-exporter-otlp-proto-http==1.14.0
opentelemetry-exporter-zipkin==1.14.0
opentelemetry-exporter-zipkin-json==1.14.0
opentelemetry-exporter-zipkin-proto-http==1.14.0
opentelemetry-instrumentation==0.35b0
opentelemetry-instrumentation-aiohttp-client==0.35b0
opentelemetry-instrumentation-asgi==0.35b0
opentelemetry-instrumentation-grpc==0.35b0
opentelemetry-proto==1.14.0
opentelemetry-sdk==1.14.0
opentelemetry-semantic-conventions==0.35b0
opentelemetry-util-http==0.35b0
opt-einsum==3.3.0
packaging==23.0
pandas==1.5.2
pandas-stubs==1.5.2.230105
pandocfilters @ file:///home/conda/feedstock_root/build_artifacts/pandocfilters_1631603243851/work
parso @ file:///home/conda/feedstock_root/build_artifacts/parso_1638334955874/work
pathspec==0.10.2
pep517==0.13.0
pexpect @ file:///home/conda/feedstock_root/build_artifacts/pexpect_1667297516076/work
pickleshare @ file:///home/conda/feedstock_root/build_artifacts/pickleshare_1602536217715/work
Pillow==9.3.0
pip-requirements-parser==31.2.0
pip-tools==6.11.0
pkginfo==1.9.2
pkgutil_resolve_name @ file:///home/conda/feedstock_root/build_artifacts/pkgutil-resolve-name_1633981968097/work
platformdirs==2.6.2
pluggy==1.0.0
ply==3.11
prometheus-client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1674535637125/work
prompt-toolkit @ file:///home/conda/feedstock_root/build_artifacts/prompt-toolkit_1670414775770/work
protobuf==3.19.6
psutil==5.9.4
ptyprocess @ file:///home/conda/feedstock_root/build_artifacts/ptyprocess_1609419310487/work/dist/ptyprocess-0.7.0-py2.py3-none-any.whl
pure-eval @ file:///home/conda/feedstock_root/build_artifacts/pure_eval_1642875951954/work
py4j==0.10.9.5
pyarrow==11.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pydantic==1.10.2
pyenchant==3.2.2
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1672682006896/work
PyJWT==2.6.0
pylint==2.16.0
pynvml==11.4.1
pyparsing==3.0.9
pyproject_hooks==1.0.0
PyQt5-sip==12.11.0
pyrsistent @ file:///Users/ktietz/ci_310/pyrsistent_1643962172005/work
pyspark==3.3.1
pytest==7.2.2
pytest-asyncio==0.20.3
pytest-cov==4.0.0
pytest-xdist==3.2.1
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1626286286081/work
python-json-logger==2.0.4
python-multipart==0.0.5
python-rapidjson==1.9
python-slugify==8.0.0
pytz==2022.6
PyWavelets==1.4.1
PyYAML==6.0
pyzmq==24.0.1
qtconsole @ file:///home/conda/feedstock_root/build_artifacts/qtconsole-base_1667404144336/work
QtPy @ file:///home/conda/feedstock_root/build_artifacts/qtpy_1667873092748/work
qudida==0.0.4
querystring-parser==1.2.4
readme-renderer==37.3
regex==2022.10.31
requests==2.28.1
requests-futures==1.0.0
requests-oauthlib==1.3.1
requests-toolbelt==0.10.1
rfc3986==2.0.0
rich==12.6.0
roundrobin==0.0.4
rsa==4.9
ruff==0.0.255
s3transfer==0.6.0
schema==0.7.5
scikit-image==0.20.0
scikit-learn==1.1.3
scipy==1.9.3
seaborn==0.12.2
Send2Trash @ file:///home/conda/feedstock_root/build_artifacts/send2trash_1628511208346/work
setuptools-scm==7.1.0
shap==0.41.0
simple-di==0.1.5
sip @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_fbqiv4bzwo/croots/recipe/sip_1659012372184/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
sklearn==0.0.post1
slicer==0.0.7
smmap==5.0.0
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1662051266223/work
snowballstemmer==2.2.0
soupsieve @ file:///home/conda/feedstock_root/build_artifacts/soupsieve_1658207591808/work
Sphinx==5.3.0
sphinx-autobuild==2021.3.14
sphinx-basic-ng==1.0.0b1
sphinx-click==4.4.0
sphinx-copybutton==0.5.1
sphinx-hoverxref==1.3.0
sphinx-issues==3.0.1
sphinx_design==0.3.0
sphinx_inline_tabs==2022.1.2b11
sphinxcontrib-applehelp==1.0.2
sphinxcontrib-devhelp==1.0.2
sphinxcontrib-fulltoc==1.2.0
sphinxcontrib-htmlhelp==2.0.0
sphinxcontrib-jsmath==1.0.1
sphinxcontrib-qthelp==1.0.3
sphinxcontrib-serializinghtml==1.1.5
sphinxcontrib-spelling==7.7.0
sphinxcontrib.jquery==3.0.0
sphinxext-opengraph==0.7.3
SQLAlchemy==1.4.46
sqlparse==0.4.3
stack-data @ file:///home/conda/feedstock_root/build_artifacts/stack_data_1669632077133/work
starlette==0.25.0
tabulate==0.9.0
tensorboard==2.11.2
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow-estimator==2.11.0
tensorflow-macos==2.11.0
termcolor==2.2.0
terminado @ file:///Users/runner/miniforge3/conda-bld/terminado_1670254106711/work
text-unidecode==1.3
thop==0.1.1.post2209072238
threadpoolctl==3.1.0
thrift==0.16.0
tifffile==2023.2.28
tinycss2 @ file:///home/conda/feedstock_root/build_artifacts/tinycss2_1666100256010/work
tokenize-rt==5.0.0
tokenizers==0.13.2
toml @ file:///home/conda/feedstock_root/build_artifacts/toml_1604308577558/work
tomli==2.0.1
tomlkit==0.11.6
torch==1.13.1
torchaudio==0.13.1
torchvision==0.14.1
tornado @ file:///private/var/folders/nz/j6p8yfhx1mv_0grj5xl4650h0000gp/T/abs_a61b4xoie9/croots/recipe/tornado_1662061692951/work
tqdm==4.64.1
traitlets @ file:///home/conda/feedstock_root/build_artifacts/traitlets_1675110562325/work
transformers==4.26.1
tritonclient==2.31.0
twine==4.0.2
types-pytz==2022.7.1.0
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1665144421445/work
urllib3==1.26.13
uvicorn==0.20.0
virtualenv==20.17.1
watchfiles==0.18.1
wcwidth @ file:///home/conda/feedstock_root/build_artifacts/wcwidth_1673864653149/work
webencodings==0.5.1
websocket-client @ file:///home/conda/feedstock_root/build_artifacts/websocket-client_1675567828044/work
Werkzeug==2.2.2
widgetsnbextension @ file:///home/conda/feedstock_root/build_artifacts/widgetsnbextension_1672066693230/work
wrapt==1.14.1
xgboost==1.7.4
yamllint==1.29.0
yarl==1.8.2
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1676708471276/work
zope.event==4.6
zope.interface==5.5.2
```

</details>


Discussion:

Resolving patch:
diff --git a/src/bentoml/_internal/marshal/dispatcher.py b/src/bentoml/_internal/marshal/dispatcher.py
--- a/src/bentoml/_internal/marshal/dispatcher.py
+++ b/src/bentoml/_internal/marshal/dispatcher.py
@@ -17,6 +17,11 @@
 logger = logging.getLogger(__name__)
 
 
+if t.TYPE_CHECKING:
+    from ..runner.utils import Params
+    from ..runner.container import Payload
+
+
 class NonBlockSema:
     def __init__(self, count: int):
         self.sema = count
@@ -33,13 +38,15 @@ def is_locked(self):
     def release(self):
         self.sema += 1
 
+
 @attr.define
 class Job:
     enqueue_time: float
-    data: t.Any
+    data: Params[Payload]
     future: asyncio.Future[t.Any]
     dispatch_time: float = 0
 
+
 class Optimizer:
     """
     Analyse historical data to optimize CorkDispatcher.
@@ -130,7 +137,9 @@ def __init__(
         self.tick_interval = 0.001
 
         self._controller = None
-        self._queue: collections.deque[Job] = collections.deque()  # TODO(bojiang): maxlen
+        self._queue: collections.deque[
+            Job
+        ] = collections.deque()  # TODO(bojiang): maxlen
         self._sema = shared_sema if shared_sema else NonBlockSema(1)
 
     def shutdown(self):
@@ -138,7 +147,7 @@ def shutdown(self):
             self._controller.cancel()
         try:
             while True:
-                _, _, fut = self._queue.pop()
+                fut = self._queue.pop().future
                 fut.cancel()
         except IndexError:
             pass
@@ -186,7 +195,7 @@ async def controller(self):
 
                 n = len(self._queue)
                 now = time.time()
-                w0 = now - self._queue[0][0]
+                w0 = now - self._queue[0].enqueue_time
 
                 # only cancel requests if there are more than enough for training
                 if (
@@ -197,7 +206,7 @@ async def controller(self):
                     and w0 >= self.max_latency_in_ms
                 ):
                     # we're being very conservative and only canceling requests if they have already timed out
-                    self._queue.popleft()[2].cancel()
+                    self._queue.popleft().future.cancel()
                     continue
                 # don't try to be smart here, just serve the first few requests
                 if self._sema.is_locked():
@@ -224,12 +233,12 @@ async def controller(self):
 
                 n = len(self._queue)
                 now = time.time()
-                w0 = now - self._queue[0][0]
+                w0 = now - self._queue[0].enqueue_time
 
                 # only cancel requests if there are more than enough for training
                 if n > 6 and w0 >= self.max_latency_in_ms:
                     # we're being very conservative and only canceling requests if they have already timed out
-                    self._queue.popleft()[2].cancel()
+                    self._queue.popleft().future.cancel()
                     continue
                 if self._sema.is_locked():
                     await asyncio.sleep(self.tick_interval)
@@ -266,14 +275,14 @@ async def controller(self):
                     n = len(self._queue)
                     dt = self.tick_interval
                     now = time.time()
-                    w0 = now - self._queue[0][0]
+                    w0 = now - self._queue[0].enqueue_time
                     a = self.optimizer.o_a
                     b = self.optimizer.o_b
 
                     # only cancel requests if there are more than enough for training
                     if n > 5 and w0 >= self.max_latency_in_ms:
                         # we're being very conservative and only canceling requests if they have already timed out
-                        self._queue.popleft()[2].cancel()
+                        self._queue.popleft().future.cancel()
                         continue
                     if n < 2 and (2 * a + b) + w0 <= step_2_wait:
                         await asyncio.sleep(self.tick_interval)
@@ -313,14 +322,14 @@ async def controller(self):
                     n = len(self._queue)
                     dt = self.tick_interval
                     now = time.time()
-                    w0 = now - self._queue[0][0]
+                    w0 = now - self._queue[0].enqueue_time
                     a = self.optimizer.o_a
                     b = self.optimizer.o_b
 
                     # only cancel requests if there are more than enough for training
                     if n > 3 and w0 >= self.max_latency_in_ms:
                         # we're being very conservative and only canceling requests if they have already timed out
-                        self._queue.popleft()[2].cancel()
+                        self._queue.popleft().future.cancel()
                         continue
                     if n < 3 and (3 * a + b) + w0 <= step_3_wait:
                         await asyncio.sleep(self.tick_interval)
@@ -384,7 +393,25 @@ async def controller(self):
                     await asyncio.sleep(self.tick_interval)
                     continue
 
-                n_call_out = min(self.max_batch_size, n)
+                n_call_out = 0
+                batch_size = 0
+                try:
+                    for input_info in self._queue:
+                        if (
+                            batch_size + input_info.data.sample.batch_size
+                            < self.max_batch_size
+                        ):
+                            n_call_out += 1
+                            batch_size += input_info.data.sample.batch_size
+                        else:
+                            break
+                except Exception as e:
+                    n_call_out = min(n, self.max_batch_size)
+                    logger.error(
+                        "error in batch-size aware batching, falling back to regular batching method",
+                        exc_info=e,
+                    )
+
                 # call
                 self._sema.acquire()
                 inputs_info = tuple(self._queue.pop() for _ in range(n_call_out))
@@ -394,26 +421,32 @@ async def controller(self):
             except Exception as e:  # pylint: disable=broad-except
                 logger.error(traceback.format_exc(), exc_info=e)
 
-    async def inbound_call(self, data: t.Any):
+    async def inbound_call(self, data: Params[Payload]):
+        if data.sample.batch_size > self.max_batch_size:
+            raise RuntimeError(
+                f"batch of size {data.sample.batch_size} exceeds configured max batch size of {self.max_batch_size}."
+            )
+
         now = time.time()
         future = self._loop.create_future()
-        input_info = (now, data, future)
+        input_info = Job(now, data, future)
         self._queue.append(input_info)
         async with self._wake_event:
             self._wake_event.notify_all()
         return await future
 
-    async def outbound_call(
-        self, inputs_info: tuple[Job, ...]
-    ):
+    async def outbound_call(self, inputs_info: tuple[Job, ...]):
         _time_start = time.time()
         _done = False
         batch_size = len(inputs_info)
         logger.debug("Dynamic batching cork released, batch size: %d", batch_size)
         try:
-            outputs = await self.callback(tuple(d for _, d, _ in inputs_info))
+            outputs = await self.callback(
+                tuple(t.cast(t.Any, input_info.data) for input_info in inputs_info)
+            )
             assert len(outputs) == len(inputs_info)
-            for (_, _, fut), out in zip(inputs_info, outputs):
+            for input_info, out in zip(inputs_info, outputs):
+                fut = input_info.future
                 if not fut.done():
                     fut.set_result(out)
             _done = True
@@ -425,13 +458,15 @@ async def outbound_call(
         except asyncio.CancelledError:
             pass
         except Exception as e:  # pylint: disable=broad-except
-            for _, _, fut in inputs_info:
+            for input_info in inputs_info:
+                fut = input_info.future
                 if not fut.done():
                     fut.set_result(e)
             _done = True
         finally:
             if not _done:
-                for _, _, fut in inputs_info:
+                for input_info in inputs_info:
+                    fut = input_info.future
                     if not fut.done():
                         fut.cancel()
             self._sema.release()


Test patch: