Repository: bentoml/BentoML

Issue:
bug: non batched request executed in non FIFO manier
### Describe the bug

In a configuration with a not embedded runner, without batching, without parallelization of workers, requests must be executed one after another according to the FIFO principle. At the moment requests are executed in LIFO mode. In some cases, this causes earlier requests to time out because they do not get a chance to execute in the required time.

I made a workaround for myself by replacing __queue.pop()_ with __queue.popleft()_ in the code below:
https://github.com/bentoml/BentoML/blob/eb0ad1a673f0e32f513ca85c1759fc03287709a3/src/bentoml/_internal/marshal/dispatcher.py#L263
and
https://github.com/bentoml/BentoML/blob/eb0ad1a673f0e32f513ca85c1759fc03287709a3/src/bentoml/_internal/marshal/dispatcher.py#L361

I'm not sure that this workaround doesn't have any side effects. But for me it works.


### To reproduce

1. Make a simple service with a custom runner.
  Set the runner parameters:
  SUPPORTED RESOURCES = "nvidia.com/gpu"
  SUPPORTS CPU MULTITHREADING = True
  And BentoML configuration:

```
api_server:
  traffic:
    timeout: 30
  workers: 1

runners:
  traffic:
    max_concurrency: 3
```
2. Make sure the runner method is performed for a sufficient amount of time, such as 10 seconds.
3. Start bentoml server. Execute several requests to your runner's API one after another, but without waiting for the response from the previous launch. I used several regular browser tabs.
4. The server response is returned out of order. For example, you made 3 requests. But the answer will come first for the first request, then for the third, and only then for the second request.

### Expected behavior

The BentoML server must respond in the order in which requests are received.

### Environment

#### Environment variable

```bash
BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''
```

#### System information

`bentoml`: 1.2.2
`python`: 3.10.13
`platform`: Linux-5.15.0-100-generic-x86_64-with-glibc2.35
`uid_gid`: 1000:1000
`conda`: 23.11.0
`in_conda_env`: True
<details><summary><code>conda_packages</code></summary>

<br>

```yaml
name: NDA
channels:
  - pytorch
  - nvidia
  - conda-forge
  - defaults
dependencies:
  - _libgcc_mutex=0.1=conda_forge
  - _openmp_mutex=4.5=2_gnu
  - accelerate=0.27.0=pyhd8ed1ab_0
  - aiohttp=3.9.3=py310h2372a71_1
  - aiosignal=1.3.1=pyhd8ed1ab_0
  - annotated-types=0.6.0=pyhd8ed1ab_0
  - anyio=4.3.0=pyhd8ed1ab_0
  - appdirs=1.4.4=pyh9f0ad1d_0
  - argon2-cffi=23.1.0=pyhd8ed1ab_0
  - argon2-cffi-bindings=21.2.0=py310h2372a71_4
  - asgiref=3.7.2=pyhd8ed1ab_0
  - async-timeout=4.0.3=pyhd8ed1ab_0
  - attrs=23.2.0=pyh71513ae_0
  - aws-c-auth=0.7.16=haed3651_8
  - aws-c-cal=0.6.10=ha9bf9b1_2
  - aws-c-common=0.9.14=hd590300_0
  - aws-c-compression=0.2.18=h4466546_2
  - aws-c-event-stream=0.4.2=he635cd5_6
  - aws-c-http=0.8.1=hbfc29b2_7
  - aws-c-io=0.14.6=h6b388c4_1
  - aws-c-mqtt=0.10.3=hffff1cc_2
  - aws-c-s3=0.5.2=h4893938_2
  - aws-c-sdkutils=0.1.15=h4466546_2
  - aws-checksums=0.1.18=h4466546_2
  - aws-crt-cpp=0.26.3=h137ae52_2
  - aws-sdk-cpp=1.11.267=he0cb598_3
  - bentoml=1.2.2=pyhd8ed1ab_0
  - blas=1.0=mkl
  - brotli-python=1.1.0=py310hc6cd4ac_1
  - bzip2=1.0.8=hd590300_5
  - c-ares=1.27.0=hd590300_0
  - ca-certificates=2024.2.2=hbcca054_0
  - cattrs=23.1.2=pyhd8ed1ab_0
  - certifi=2024.2.2=pyhd8ed1ab_0
  - cffi=1.16.0=py310h2fee648_0
  - charset-normalizer=3.3.2=pyhd8ed1ab_0
  - circus=0.18.0=pyhd8ed1ab_0
  - click=8.1.7=unix_pyh707e725_0
  - click-option-group=0.5.6=pyhd8ed1ab_0
  - cloudpickle=3.0.0=pyhd8ed1ab_0
  - colorama=0.4.6=pyhd8ed1ab_0
  - contextlib2=21.6.0=pyhd8ed1ab_0
  - cuda-cudart=12.1.105=0
  - cuda-cupti=12.1.105=0
  - cuda-libraries=12.1.0=0
  - cuda-nvrtc=12.1.105=0
  - cuda-nvtx=12.1.105=0
  - cuda-opencl=12.4.99=0
  - cuda-runtime=12.1.0=0
  - datasets=2.18.0=pyhd8ed1ab_0
  - deepmerge=1.1.1=pyhd8ed1ab_0
  - deprecated=1.2.14=pyh1a96a4e_0
  - diffusers=0.26.3=pyhd8ed1ab_0
  - dill=0.3.8=pyhd8ed1ab_0
  - exceptiongroup=1.2.0=pyhd8ed1ab_2
  - ffmpeg=4.3=hf484d3e_0
  - filelock=3.13.1=pyhd8ed1ab_0
  - freetype=2.12.1=h267a509_2
  - frozenlist=1.4.1=py310h2372a71_0
  - fs=2.4.16=pyhd8ed1ab_0
  - fsspec=2024.2.0=pyhca7485f_0
  - gflags=2.2.2=he1b5a44_1004
  - glog=0.7.0=hed5481d_0
  - gmp=6.3.0=h59595ed_1
  - gmpy2=2.1.2=py310h3ec546c_1
  - gnutls=3.6.13=h85f3911_1
  - h11=0.14.0=pyhd8ed1ab_0
  - h2=4.1.0=pyhd8ed1ab_0
  - hpack=4.0.0=pyh9f0ad1d_0
  - httpcore=1.0.4=pyhd8ed1ab_0
  - httpx=0.27.0=pyhd8ed1ab_0
  - huggingface_hub=0.21.4=pyhd8ed1ab_0
  - hyperframe=6.0.1=pyhd8ed1ab_0
  - icu=73.2=h59595ed_0
  - idna=3.6=pyhd8ed1ab_0
  - importlib-metadata=6.0.0=pyha770c72_0
  - inflection=0.5.1=pyh9f0ad1d_0
  - intel-openmp=2022.1.0=h9e868ea_3769
  - jinja2=3.1.3=pyhd8ed1ab_0
  - keyutils=1.6.1=h166bdaf_0
  - krb5=1.21.2=h659d440_0
  - lame=3.100=h166bdaf_1003
  - lcms2=2.16=hb7c19ff_0
  - ld_impl_linux-64=2.40=h41732ed_0
  - lerc=4.0.0=h27087fc_0
  - libabseil=20240116.1=cxx17_h59595ed_2
  - libarrow=15.0.1=h6bfc85a_2_cpu
  - libarrow-acero=15.0.1=h59595ed_2_cpu
  - libarrow-dataset=15.0.1=h59595ed_2_cpu
  - libarrow-flight=15.0.1=hc6145d9_2_cpu
  - libarrow-flight-sql=15.0.1=h757c851_2_cpu
  - libarrow-gandiva=15.0.1=hb016d2e_2_cpu
  - libarrow-substrait=15.0.1=h757c851_2_cpu
  - libblas=3.9.0=16_linux64_mkl
  - libbrotlicommon=1.1.0=hd590300_1
  - libbrotlidec=1.1.0=hd590300_1
  - libbrotlienc=1.1.0=hd590300_1
  - libcblas=3.9.0=16_linux64_mkl
  - libcrc32c=1.1.2=h9c3ff4c_0
  - libcublas=12.1.0.26=0
  - libcufft=11.0.2.4=0
  - libcufile=1.9.0.20=0
  - libcurand=10.3.5.119=0
  - libcurl=8.6.0=hca28451_0
  - libcusolver=11.4.4.55=0
  - libcusparse=12.0.2.55=0
  - libdeflate=1.19=hd590300_0
  - libedit=3.1.20191231=he28a2e2_2
  - libev=4.33=hd590300_2
  - libevent=2.1.12=hf998b51_1
  - libffi=3.4.2=h7f98852_5
  - libgcc-ng=13.2.0=h807b86a_5
  - libgomp=13.2.0=h807b86a_5
  - libgoogle-cloud=2.22.0=h9be4e54_1
  - libgoogle-cloud-storage=2.22.0=hc7a4891_1
  - libgrpc=1.62.1=h15f2491_0
  - libiconv=1.17=hd590300_2
  - libjpeg-turbo=3.0.0=hd590300_1
  - liblapack=3.9.0=16_linux64_mkl
  - libllvm16=16.0.6=hb3ce162_3
  - libnghttp2=1.58.0=h47da74e_1
  - libnl=3.9.0=hd590300_0
  - libnpp=12.0.2.50=0
  - libnsl=2.0.1=hd590300_0
  - libnvjitlink=12.1.105=0
  - libnvjpeg=12.1.1.14=0
  - libparquet=15.0.1=h352af49_2_cpu
  - libpng=1.6.43=h2797004_0
  - libprotobuf=4.25.3=h08a7969_0
  - libre2-11=2023.09.01=h5a48ba9_2
  - libsodium=1.0.18=h36c2ea0_1
  - libsqlite=3.45.2=h2797004_0
  - libssh2=1.11.0=h0841786_0
  - libstdcxx-ng=13.2.0=h7e041cc_5
  - libthrift=0.19.0=hb90f79a_1
  - libtiff=4.6.0=ha9c0a0a_2
  - libutf8proc=2.8.0=h166bdaf_0
  - libuuid=2.38.1=h0b41bf4_0
  - libwebp-base=1.3.2=hd590300_0
  - libxcb=1.15=h0b41bf4_0
  - libxcrypt=4.4.36=hd590300_1
  - libxml2=2.12.5=h232c23b_0
  - libzlib=1.2.13=hd590300_5
  - llvm-openmp=15.0.7=h0cdce71_0
  - lz4-c=1.9.4=hcb278e6_0
  - markdown-it-py=3.0.0=pyhd8ed1ab_0
  - markupsafe=2.1.5=py310h2372a71_0
  - mdurl=0.1.2=pyhd8ed1ab_0
  - minio=7.2.4=pyhd8ed1ab_0
  - mkl=2022.1.0=hc2b9512_224
  - mpc=1.3.1=hfe3b2da_0
  - mpfr=4.2.1=h9458935_0
  - mpmath=1.3.0=pyhd8ed1ab_0
  - multidict=6.0.5=py310h2372a71_0
  - multiprocess=0.70.16=py310h2372a71_0
  - ncurses=6.4=h59595ed_2
  - nettle=3.6=he412f7d_0
  - networkx=3.2.1=pyhd8ed1ab_0
  - numpy=1.26.4=py310hb13e2d6_0
  - nvidia-ml-py=11.525.131=pyhd8ed1ab_0
  - openh264=2.1.1=h780b84a_0
  - openjpeg=2.5.2=h488ebb8_0
  - openssl=3.2.1=hd590300_0
  - opentelemetry-api=1.20.0=pyhd8ed1ab_0
  - opentelemetry-instrumentation=0.41b0=pyhd8ed1ab_0
  - opentelemetry-instrumentation-aiohttp-client=0.41b0=pyhd8ed1ab_0
  - opentelemetry-instrumentation-asgi=0.41b0=pyhd8ed1ab_0
  - opentelemetry-sdk=1.20.0=pyhd8ed1ab_0
  - opentelemetry-semantic-conventions=0.41b0=pyhd8ed1ab_0
  - opentelemetry-util-http=0.41b0=pyhd8ed1ab_0
  - orc=2.0.0=h1e5e2c1_0
  - packaging=24.0=pyhd8ed1ab_0
  - pandas=2.2.1=py310hcc13569_0
  - pathspec=0.12.1=pyhd8ed1ab_0
  - pillow=10.2.0=py310h01dd4db_0
  - pip=24.0=pyhd8ed1ab_0
  - pip-requirements-parser=32.0.1=pyhd8ed1ab_0
  - pip-tools=7.4.1=pyhd8ed1ab_0
  - prometheus_client=0.20.0=pyhd8ed1ab_0
  - psutil=5.9.8=py310h2372a71_0
  - pthread-stubs=0.4=h36c2ea0_1001
  - pyarrow=15.0.1=py310hf9e7431_2_cpu
  - pyarrow-hotfix=0.6=pyhd8ed1ab_0
  - pycparser=2.21=pyhd8ed1ab_0
  - pycryptodome=3.20.0=py310hb0f0acc_0
  - pydantic=2.6.3=pyhd8ed1ab_0
  - pydantic-core=2.16.3=py310hcb5633a_0
  - pygments=2.17.2=pyhd8ed1ab_0
  - pyparsing=3.1.2=pyhd8ed1ab_0
  - pyproject_hooks=1.0.0=pyhd8ed1ab_0
  - pysocks=1.7.1=pyha2e5f31_6
  - python=3.10.13=hd12c33a_1_cpython
  - python-build=1.1.1=pyhd8ed1ab_0
  - python-dateutil=2.9.0=pyhd8ed1ab_0
  - python-json-logger=2.0.7=pyhd8ed1ab_0
  - python-multipart=0.0.9=pyhd8ed1ab_0
  - python-tzdata=2024.1=pyhd8ed1ab_0
  - python-xxhash=3.4.1=py310h2372a71_0
  - python_abi=3.10=4_cp310
  - pytorch=2.2.0=py3.10_cuda12.1_cudnn8.9.2_0
  - pytorch-cuda=12.1=ha16c6d3_5
  - pytorch-mutex=1.0=cuda
  - pytz=2024.1=pyhd8ed1ab_0
  - pyyaml=6.0.1=py310h2372a71_1
  - pyzmq=25.1.2=py310h795f18f_0
  - rdma-core=50.0=hd3aeb46_1
  - re2=2023.09.01=h7f4b329_2
  - readline=8.2=h8228510_1
  - regex=2023.12.25=py310h2372a71_0
  - requests=2.31.0=pyhd8ed1ab_0
  - rich=13.7.1=pyhd8ed1ab_0
  - s2n=1.4.7=h06160fa_0
  - safetensors=0.4.2=py310hcb5633a_0
  - schema=0.7.5=pyhd8ed1ab_1
  - setuptools=69.2.0=pyhd8ed1ab_0
  - simple-di=0.1.5=pyhd8ed1ab_0
  - six=1.16.0=pyh6c4a22f_0
  - snappy=1.1.10=h9fff704_0
  - sniffio=1.3.1=pyhd8ed1ab_0
  - starlette=0.37.2=pyhd8ed1ab_0
  - sympy=1.12=pypyh9d50eac_103
  - tk=8.6.13=noxft_h4845f30_101
  - tokenizers=0.15.2=py310h320607d_0
  - tomli=2.0.1=pyhd8ed1ab_0
  - torchaudio=2.2.0=py310_cu121
  - torchtriton=2.2.0=py310
  - torchvision=0.17.0=py310_cu121
  - tornado=6.4=py310h2372a71_0
  - tqdm=4.66.2=pyhd8ed1ab_0
  - transformers=4.37.2=pyhd8ed1ab_0
  - typing-extensions=4.10.0=hd8ed1ab_0
  - typing_extensions=4.10.0=pyha770c72_0
  - tzdata=2024a=h0c530f3_0
  - ucx=1.15.0=h11edf95_7
  - urllib3=2.2.1=pyhd8ed1ab_0
  - uvicorn=0.28.0=py310hff52083_0
  - watchfiles=0.21.0=py310hcb5633a_0
  - wheel=0.42.0=pyhd8ed1ab_0
  - wrapt=1.16.0=py310h2372a71_0
  - xorg-libxau=1.0.11=hd590300_0
  - xorg-libxdmcp=1.1.3=h7f98852_0
  - xxhash=0.8.2=hd590300_0
  - xz=5.2.6=h166bdaf_0
  - yaml=0.2.5=h7f98852_2
  - yarl=1.9.4=py310h2372a71_0
  - zeromq=4.3.5=h59595ed_1
  - zipp=3.17.0=pyhd8ed1ab_0
  - zlib=1.2.13=hd590300_5
  - zstd=1.5.5=hfc55251_0
prefix: NDA
```

</details>

<details><summary><code>pip_packages</code></summary>

<br>

```
accelerate @ file:///home/conda/feedstock_root/build_artifacts/accelerate_1707501497624/work
aiohttp @ file:///home/conda/feedstock_root/build_artifacts/aiohttp_1710511536979/work
aiosignal @ file:///home/conda/feedstock_root/build_artifacts/aiosignal_1667935791922/work
annotated-types @ file:///home/conda/feedstock_root/build_artifacts/annotated-types_1696634205638/work
anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1708355285029/work
appdirs @ file:///home/conda/feedstock_root/build_artifacts/appdirs_1603108395799/work
argon2-cffi @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi_1692818318753/work
argon2-cffi-bindings @ file:///home/conda/feedstock_root/build_artifacts/argon2-cffi-bindings_1695386546427/work
asgiref @ file:///home/conda/feedstock_root/build_artifacts/asgiref_1685243452975/work
async-timeout @ file:///home/conda/feedstock_root/build_artifacts/async-timeout_1691763562544/work
attrs @ file:///home/conda/feedstock_root/build_artifacts/attrs_1704011227531/work
bentoml @ file:///home/conda/feedstock_root/build_artifacts/bentoml_1707123960469/work
Brotli @ file:///home/conda/feedstock_root/build_artifacts/brotli-split_1695989787169/work
build @ file:///home/conda/feedstock_root/build_artifacts/python-build_1709251527949/work
cattrs @ file:///home/conda/feedstock_root/build_artifacts/cattrs_1685682319569/work
certifi @ file:///home/conda/feedstock_root/build_artifacts/certifi_1707022139797/work/certifi
cffi @ file:///home/conda/feedstock_root/build_artifacts/cffi_1696001684923/work
charset-normalizer @ file:///home/conda/feedstock_root/build_artifacts/charset-normalizer_1698833585322/work
circus @ file:///home/conda/feedstock_root/build_artifacts/circus_1668846026855/work
click @ file:///home/conda/feedstock_root/build_artifacts/click_1692311806742/work
click-option-group @ file:///home/conda/feedstock_root/build_artifacts/click-option-group_1686394190925/work
cloudpickle @ file:///home/conda/feedstock_root/build_artifacts/cloudpickle_1697464713350/work
colorama @ file:///home/conda/feedstock_root/build_artifacts/colorama_1666700638685/work
contextlib2 @ file:///home/conda/feedstock_root/build_artifacts/contextlib2_1624848568296/work
datasets @ file:///home/conda/feedstock_root/build_artifacts/datasets_1709395865330/work
deepmerge @ file:///home/conda/feedstock_root/build_artifacts/deepmerge_1702941685750/work
Deprecated @ file:///home/conda/feedstock_root/build_artifacts/deprecated_1685233314779/work
diffusers @ file:///home/conda/feedstock_root/build_artifacts/diffusers_1707822915454/work
dill @ file:///home/conda/feedstock_root/build_artifacts/dill_1706434688412/work
exceptiongroup @ file:///home/conda/feedstock_root/build_artifacts/exceptiongroup_1704921103267/work
filelock @ file:///home/conda/feedstock_root/build_artifacts/filelock_1698714947081/work
frozenlist @ file:///home/conda/feedstock_root/build_artifacts/frozenlist_1702645481127/work
fs @ file:///home/conda/feedstock_root/build_artifacts/fs_1683650158618/work
fsspec @ file:///home/conda/feedstock_root/build_artifacts/fsspec_1707102468451/work
gmpy2 @ file:///home/conda/feedstock_root/build_artifacts/gmpy2_1666808654411/work
h11 @ file:///home/conda/feedstock_root/build_artifacts/h11_1664132893548/work
h2 @ file:///home/conda/feedstock_root/build_artifacts/h2_1634280454336/work
hpack==4.0.0
httpcore @ file:///home/conda/feedstock_root/build_artifacts/httpcore_1708529260247/work
httpx @ file:///home/conda/feedstock_root/build_artifacts/httpx_1708530890843/work
huggingface_hub @ file:///home/conda/feedstock_root/build_artifacts/huggingface_hub_1709810635566/work
hyperframe @ file:///home/conda/feedstock_root/build_artifacts/hyperframe_1619110129307/work
idna @ file:///home/conda/feedstock_root/build_artifacts/idna_1701026962277/work
importlib-metadata @ file:///home/conda/feedstock_root/build_artifacts/importlib-metadata_1672612343532/work
inflection @ file:///home/conda/feedstock_root/build_artifacts/inflection_1598089801258/work
Jinja2 @ file:///home/conda/feedstock_root/build_artifacts/jinja2_1704966972576/work
markdown-it-py @ file:///home/conda/feedstock_root/build_artifacts/markdown-it-py_1686175045316/work
MarkupSafe @ file:///home/conda/feedstock_root/build_artifacts/markupsafe_1706899921127/work
mdurl @ file:///home/conda/feedstock_root/build_artifacts/mdurl_1704317613764/work
minio @ file:///home/conda/feedstock_root/build_artifacts/minio_1707620722786/work
mpmath @ file:///home/conda/feedstock_root/build_artifacts/mpmath_1678228039184/work
multidict @ file:///home/conda/feedstock_root/build_artifacts/multidict_1707040698785/work
multiprocess @ file:///home/conda/feedstock_root/build_artifacts/multiprocess_1706514640841/work
networkx @ file:///home/conda/feedstock_root/build_artifacts/networkx_1698504735452/work
numpy @ file:///home/conda/feedstock_root/build_artifacts/numpy_1707225380409/work/dist/numpy-1.26.4-cp310-cp310-linux_x86_64.whl#sha256=51131fd8fc130cd168aecaf1bc0ea85f92e8ffebf211772ceb16ac2e7f10d7ca
nvidia-ml-py @ file:///home/conda/feedstock_root/build_artifacts/nvidia-ml-py_1688171990873/work
opentelemetry-api @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-api_1693876773665/work
opentelemetry-instrumentation @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-instrumentation_1694479097639/work
opentelemetry-instrumentation-aiohttp-client @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-instrumentation-aiohttp-client_1694590957674/work
opentelemetry-instrumentation-asgi @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-instrumentation-asgi_1694598742186/work
opentelemetry-sdk @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-sdk_1694936834333/work
opentelemetry-semantic-conventions @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-semantic-conventions_1694312669793/work
opentelemetry-util-http @ file:///home/conda/feedstock_root/build_artifacts/opentelemetry-util-http_1694494133477/work
packaging @ file:///home/conda/feedstock_root/build_artifacts/packaging_1710075952259/work
pandas @ file:///home/conda/feedstock_root/build_artifacts/pandas_1708708607448/work
pathspec @ file:///home/conda/feedstock_root/build_artifacts/pathspec_1702249949303/work
pillow @ file:///home/conda/feedstock_root/build_artifacts/pillow_1704252020178/work
pip-requirements-parser @ file:///home/conda/feedstock_root/build_artifacts/pip-requirements-parser_1672265598496/work
pip-tools @ file:///home/conda/feedstock_root/build_artifacts/pip-tools_1709736059034/work
prometheus_client @ file:///home/conda/feedstock_root/build_artifacts/prometheus_client_1707932675456/work
psutil @ file:///home/conda/feedstock_root/build_artifacts/psutil_1705722392846/work
pyarrow==15.0.1
pyarrow-hotfix @ file:///home/conda/feedstock_root/build_artifacts/pyarrow-hotfix_1700596371886/work
pycparser @ file:///home/conda/feedstock_root/build_artifacts/pycparser_1636257122734/work
pycryptodome @ file:///home/conda/feedstock_root/build_artifacts/pycryptodome_1704930718567/work
pydantic @ file:///home/conda/feedstock_root/build_artifacts/pydantic_1709075187841/work
pydantic_core @ file:///home/conda/feedstock_root/build_artifacts/pydantic-core_1708700740324/work
Pygments @ file:///home/conda/feedstock_root/build_artifacts/pygments_1700607939962/work
pyparsing @ file:///home/conda/feedstock_root/build_artifacts/pyparsing_1709721012883/work
pyproject_hooks @ file:///home/conda/feedstock_root/build_artifacts/pyproject_hooks_1670268672584/work
PySocks @ file:///home/conda/feedstock_root/build_artifacts/pysocks_1661604839144/work
python-dateutil @ file:///home/conda/feedstock_root/build_artifacts/python-dateutil_1709299778482/work
python-json-logger @ file:///home/conda/feedstock_root/build_artifacts/python-json-logger_1677079630776/work
python-multipart @ file:///home/conda/feedstock_root/build_artifacts/python-multipart_1707760088566/work
pytz @ file:///home/conda/feedstock_root/build_artifacts/pytz_1706886791323/work
PyYAML @ file:///home/conda/feedstock_root/build_artifacts/pyyaml_1695373428874/work
pyzmq @ file:///home/conda/feedstock_root/build_artifacts/pyzmq_1701783169502/work
regex @ file:///home/conda/feedstock_root/build_artifacts/regex_1703393490683/work
requests @ file:///home/conda/feedstock_root/build_artifacts/requests_1684774241324/work
rich @ file:///home/conda/feedstock_root/build_artifacts/rich-split_1709150387247/work/dist
safetensors @ file:///home/conda/feedstock_root/build_artifacts/safetensors_1707377218239/work
schema @ file:///home/conda/feedstock_root/build_artifacts/schema_1684241031536/work
simple-di @ file:///home/conda/feedstock_root/build_artifacts/simple-di_1656418562483/work
six @ file:///home/conda/feedstock_root/build_artifacts/six_1620240208055/work
sniffio @ file:///home/conda/feedstock_root/build_artifacts/sniffio_1708952932303/work
starlette @ file:///home/conda/feedstock_root/build_artifacts/starlette-recipe_1709667058396/work
sympy @ file:///home/conda/feedstock_root/build_artifacts/sympy_1684180540116/work
tokenizers @ file:///home/conda/feedstock_root/build_artifacts/tokenizers_1707733850559/work/bindings/python
tomli @ file:///home/conda/feedstock_root/build_artifacts/tomli_1644342247877/work
torch==2.2.0
torchaudio==2.2.0
torchvision==0.17.0
tornado @ file:///home/conda/feedstock_root/build_artifacts/tornado_1708363098266/work
tqdm @ file:///home/conda/feedstock_root/build_artifacts/tqdm_1707598593068/work
transformers @ file:///home/conda/feedstock_root/build_artifacts/transformers_1707370358661/work
triton==2.2.0
typing_extensions @ file:///home/conda/feedstock_root/build_artifacts/typing_extensions_1708904622550/work
tzdata @ file:///home/conda/feedstock_root/build_artifacts/python-tzdata_1707747584337/work
urllib3 @ file:///home/conda/feedstock_root/build_artifacts/urllib3_1708239446578/work
uvicorn @ file:///home/conda/feedstock_root/build_artifacts/uvicorn-split_1710145967899/work
watchfiles @ file:///home/conda/feedstock_root/build_artifacts/watchfiles_1701077795738/work
wrapt @ file:///home/conda/feedstock_root/build_artifacts/wrapt_1699532811524/work
xxhash @ file:///home/conda/feedstock_root/build_artifacts/python-xxhash_1696486308932/work
yarl @ file:///home/conda/feedstock_root/build_artifacts/yarl_1705508292061/work
zipp @ file:///home/conda/feedstock_root/build_artifacts/zipp_1695255097490/work
```

</details>



Discussion:
This looks like a careless mistake, can you submit a PR to fix these two places?

Resolving patch:
diff --git a/src/bentoml/_internal/marshal/dispatcher.py b/src/bentoml/_internal/marshal/dispatcher.py
--- a/src/bentoml/_internal/marshal/dispatcher.py
+++ b/src/bentoml/_internal/marshal/dispatcher.py
@@ -260,7 +260,7 @@ async def train_optimizer(
                 req_count += 1
                 # call
                 self._sema.acquire()
-                inputs_info = tuple(self._queue.pop() for _ in range(n_call_out))
+                inputs_info = tuple(self._queue.popleft() for _ in range(n_call_out))
                 for info in inputs_info:
                     # fake wait as 0 for training requests
                     info.enqueue_time = now
@@ -358,7 +358,7 @@ async def controller(self) -> None:
                         )
                 # call
                 self._sema.acquire()
-                inputs_info = tuple(self._queue.pop() for _ in range(n_call_out))
+                inputs_info = tuple(self._queue.popleft() for _ in range(n_call_out))
                 self._loop.create_task(self.outbound_call(inputs_info))
             except Exception as e:  # pylint: disable=broad-except
                 logger.error(traceback.format_exc(), exc_info=e)


Test patch: