Repository: bentoml/BentoML

Issue:
bug: Bentos performance issues
### Describe the bug

I've been conducting load testing on Bento using the MobileNetV2 model. I've observed that the CPU is fully utilized and upon profiling the app, I discovered that the workers are consuming all of the CPU, despite the relatively low RPS (around 30). To further investigate this issue, I decided to remove the model inference and test the load with an empty runner that receives an image and returns it without any computation. Despite this, the CPU is still fully utilized at a very low RPS (around 30). Additionally, the latency is quite high - 1900ms on the 50th percentile.
I've also compared Bento with FastAPI and exposed the same model on an endpoint, and FastAPI outperforms Bentos significantly. I also couldn't see any improvement when running the adaptive batching inference.


### To reproduce

Here's the code to reproduce the performance issue i faced (no inference in this case, just HTTP endpoint with logic-less runner. (it also happens when inferencing ofc)):

```
class CustomRunnable(bentoml.Runnable):
    SUPPORTED_RESOURCES = ("nvidia.com/gpu",)
    @bentoml.Runnable.method(batchable=False)
    def classify_batch(self, input_batch):
       return input_batch


runner = bentoml.Runner(CustomRunnable, name="gore_classifier_runnable")

svc = bentoml.Service("gore-binary-classifier", runners=[runner])

@svc.api(input=Image(),
         output=JSON(pydantic_model=ImageClassifierResponse),
         route="/classify/image",
         )
async def classify_image(req: Image) -> ImageClassifierResponse:
    """running the dummy runner"""
    outputs = await runner.async_run(req)
    
    """return hard coded value"""
    category, conf = "ooc", 0.1231


    return ImageClassifierResponse(request_id=1,
                                   predicted_category=category,
                                   confidence=conf)
```

The maximum RPS I've managed to get here is around ~30 then i observed that the CPU is fully utilized. (to make a comparison, i ran the same test on fastAPI, i've used a way smaller VM (4 CPUs compared to Bentos 16cpus VM) and got ~400RPS for the exact same operation with way lower latencies)



### Expected behavior

i'd expect Bento to provide a reasonable performance (at least the same as FastAPI which and i understand is based on)

### Environment

VM type - n1-standard-16(16 CPU + 60G Memory) + Tesla T4 GPU (1 GPU)
Image created using Bento’s image Containerize command and deployed on GKE Cluster using 1 pod, which gets all node resources.

here's the ```bentoml env``` output - 

```
#### Environment variable

```bash
BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''
```

#### System information

`bentoml`: 1.0.12
`python`: 3.9.16
`platform`: Linux-5.4.188+-x86_64-with-glibc2.31
`uid_gid`: 0:0
<details><summary><code>pip_packages</code></summary>

<br>

```
aiohttp==3.8.3
aiosignal==1.3.1
anyio==3.6.2
appdirs==1.4.4
asgiref==3.6.0
async-timeout==4.0.2
attrs==22.2.0
backoff==2.2.1
bentoml==1.0.12
build==0.10.0
cattrs==22.2.0
certifi==2019.11.28
chardet==3.0.4
charset-normalizer==2.1.1
circus==0.18.0
click==8.1.3
click-option-group==0.5.5
cloudpickle==2.2.0
commonmark==0.9.1
contextlib2==21.6.0
dbus-python==1.2.16
deepmerge==1.1.0
Deprecated==1.2.13
exceptiongroup==1.1.0
filetype==1.2.0
frozenlist==1.3.3
fs==2.4.16
googleapis-common-protos==1.58.0
h11==0.14.0
idna==2.8
Jinja2==3.1.2
MarkupSafe==2.1.1
multidict==6.0.4
numpy==1.24.1
opentelemetry-api==1.14.0
opentelemetry-exporter-otlp-proto-http==1.14.0
opentelemetry-instrumentation==0.35b0
opentelemetry-instrumentation-aiohttp-client==0.35b0
opentelemetry-instrumentation-asgi==0.35b0
opentelemetry-proto==1.14.0
opentelemetry-sdk==1.14.0
opentelemetry-semantic-conventions==0.35b0
opentelemetry-util-http==0.35b0
packaging==21.3
pathspec==0.10.3
Pillow==9.2.0
pip-requirements-parser==32.0.1
pip-tools==6.12.1
prometheus-client==0.15.0
protobuf==3.20.3
psutil==5.9.4
pydantic==1.8.2
Pygments==2.14.0
PyGObject==3.36.0
pynvml==11.4.1
pyparsing==3.0.9
pyproject_hooks==1.0.0
python-apt==2.0.0+ubuntu0.20.4.8
python-dateutil==2.8.2
python-json-logger==2.0.4
python-multipart==0.0.5
PyYAML==6.0
pyzmq==25.0.0
requests==2.22.0
requests-unixsocket==0.2.0
rich==13.1.0
schema==0.7.5
simple-di==0.1.5
six==1.14.0
sniffio==1.3.0
starlette==0.23.1
tomli==2.0.1
torch==1.11.0
torchvision==0.12.0
tornado==6.2
typing_extensions==4.4.0
urllib3==1.25.8
uvicorn==0.20.0
watchfiles==0.18.1
wrapt==1.14.1
yarl==1.8.2
```

</details>

```




Discussion:
Hi there, can you send us the benchmark code that you have and the framework you are using for the testing?
Hey, I've posted the code above to reproduce the issue. (it's basically running a logic-less runner, but ofc happens also when running model inference).
regarding the framework for testing - I used locust.

Oh I'm asking for the code that you are running against the server. Are you using something like locust to get the RPS or something else.
yeah i'm using locust and there's an RPS indicator there which tells you how much RPS the server handles. (i also checked it with bento built in metrics and it seem to be consistent).

here's the locust code - 


```
from locust import HttpUser, between, task

class WebsiteUser(HttpUser):
    wait_time = between(1, 1.2)


    @task
    def classify(self):

        with open('/locust-tasks/test_img.jpg', "rb") as f:
            URL = "/classify/image"
            self.client.post(URL, files={"upload_file": ("foo.jpeg", f, "image/jpeg")})
```

so basically i'm just sending the same image to be inferenced over and over again..
I have noticed a similar behaviour. Using bentoml without runners seems to be a lot faster and less resource intensive than using runners. Is there any performance comparison benchmark that we can look into for Bentoml and FastAPI?

Resolving patch:
diff --git a/src/bentoml/_internal/external_typing/__init__.py b/src/bentoml/_internal/external_typing/__init__.py
--- a/src/bentoml/_internal/external_typing/__init__.py
+++ b/src/bentoml/_internal/external_typing/__init__.py
@@ -34,6 +34,8 @@
     from .starlette import ASGIReceive
     from .starlette import AsgiMiddleware
 
+    from PIL.Image import Image as PILImage
+
     WSGIApp = t.Callable[[F, t.Mapping[str, t.Any]], t.Iterable[bytes]]
 
     __all__ = [
@@ -41,6 +43,7 @@
         "PdDataFrame",
         "PdDType",
         "PdDTypeArg",
+        "PILImage",
         "DataFrameOrient",
         "SeriesOrient",
         "ObjectID",
diff --git a/src/bentoml/_internal/runner/container.py b/src/bentoml/_internal/runner/container.py
--- a/src/bentoml/_internal/runner/container.py
+++ b/src/bentoml/_internal/runner/container.py
@@ -2,6 +2,7 @@
 
 import abc
 import base64
+import io
 import itertools
 import pickle
 import typing as t
@@ -148,9 +149,7 @@ def batch_to_payloads(
     @classmethod
     @abc.abstractmethod
     def from_batch_payloads(
-        cls,
-        payloads: t.Sequence[Payload],
-        batch_dim: int,
+        cls, payloads: t.Sequence[Payload], batch_dim: int
     ) -> tuple[BatchType, list[int]]:
         ...
 
@@ -450,6 +449,49 @@ def from_batch_payloads(  # pylint: disable=arguments-differ
         return cls.batches_to_batch(batches, batch_dim)
 
 
+class PILImageContainer(DataContainer["ext.PILImage", "ext.PILImage"]):
+    _error = (
+        "PIL.Image doesn't support batch inference."
+        "You can convert it to numpy.ndarray before passing to the runner."
+    )
+
+    @classmethod
+    def to_payload(cls, batch: ext.PILImage, batch_dim: int) -> Payload:
+        buffer = io.BytesIO()
+        batch.save(buffer, format=batch.format)
+        return cls.create_payload(buffer.getvalue(), batch_size=1)
+
+    @classmethod
+    def from_payload(cls, payload: Payload) -> ext.PILImage:
+        from ..io_descriptors.image import PIL
+
+        return PIL.Image.open(io.BytesIO(payload.data))
+
+    @classmethod
+    def batch_to_payloads(
+        cls, batch: ext.PILImage, indices: t.Sequence[int], batch_dim: int
+    ) -> list[Payload]:
+        raise NotImplementedError(cls._error)
+
+    @classmethod
+    def batches_to_batch(
+        cls, batches: t.Sequence[ext.PILImage], batch_dim: int
+    ) -> tuple[ext.PILImage, list[int]]:
+        raise NotImplementedError(cls._error)
+
+    @classmethod
+    def batch_to_batches(
+        cls, batch: ext.PILImage, indices: t.Sequence[int], batch_dim: int
+    ) -> list[ext.PILImage]:
+        raise NotImplementedError(cls._error)
+
+    @classmethod
+    def from_batch_payloads(
+        cls, payloads: t.Sequence[Payload], batch_dim: int = 0
+    ) -> tuple[ext.PILImage, list[int]]:
+        raise NotImplementedError(cls._error)
+
+
 class DefaultContainer(DataContainer[t.Any, t.List[t.Any]]):
     @classmethod
     def batches_to_batch(
@@ -540,20 +582,24 @@ def find_by_single_type(
         cls, type_: t.Type[SingleType] | LazyType[SingleType]
     ) -> t.Type[DataContainer[SingleType, t.Any]]:
         typeref = LazyType.from_type(type_)
-        return cls.CONTAINER_SINGLE_TYPE_MAP.get(
-            typeref,
-            DefaultContainer,
-        )
+        if typeref in cls.CONTAINER_SINGLE_TYPE_MAP:
+            return cls.CONTAINER_SINGLE_TYPE_MAP[typeref]
+        for klass in cls.CONTAINER_SINGLE_TYPE_MAP:
+            if klass.issubclass(type_):
+                return cls.CONTAINER_SINGLE_TYPE_MAP[klass]
+        return DefaultContainer
 
     @classmethod
     def find_by_batch_type(
         cls, type_: t.Type[BatchType] | LazyType[BatchType]
     ) -> t.Type[DataContainer[t.Any, BatchType]]:
         typeref = LazyType.from_type(type_)
-        return cls.CONTAINER_BATCH_TYPE_MAP.get(
-            typeref,
-            DefaultContainer,
-        )
+        if typeref in cls.CONTAINER_BATCH_TYPE_MAP:
+            return cls.CONTAINER_BATCH_TYPE_MAP[typeref]
+        for klass in cls.CONTAINER_BATCH_TYPE_MAP:
+            if klass.issubclass(type_):
+                return cls.CONTAINER_BATCH_TYPE_MAP[klass]
+        return DefaultContainer
 
     @classmethod
     def find_by_name(cls, name: str) -> t.Type[DataContainer[t.Any, t.Any]]:
@@ -600,6 +646,11 @@ def register_builtin_containers():
         LazyType("tritonclient.http.aio", "InferInput"),
         TritonInferInputDataContainer,
     )
+    DataContainerRegistry.register_container(
+        LazyType("PIL.Image", "Image"),
+        LazyType("PIL.Image", "Image"),
+        PILImageContainer,
+    )
 
 
 register_builtin_containers()
diff --git a/src/bentoml/_internal/types.py b/src/bentoml/_internal/types.py
--- a/src/bentoml/_internal/types.py
+++ b/src/bentoml/_internal/types.py
@@ -255,6 +255,12 @@ def isinstance(self, obj: t.Any) -> t.TypeGuard[T]:
         except ValueError:
             return False
 
+    def issubclass(self, klass: type) -> bool:
+        try:
+            return issubclass(klass, self.get_class(import_module=False))
+        except ValueError:
+            return False
+
 
 if TYPE_CHECKING:
     from types import UnionType


Test patch: