Repository: bentoml/BentoML
Pull request: feat: PIL image encoder to improve the performance
Signed-off-by: Frost Ming <me@frostming.com>

## What does this PR address?

This adds a data container for PIL image types. Instead of pickling the whold image object, we only extract the bytes data from it.

After change, the performance is 3-4x better.

**Before:** 125 RPS
**After:** 430 RPS
**Development mode(no serialization):** 440 RPS

![image](https://github.com/bentoml/BentoML/assets/16336606/4d6f333e-e006-4573-aadc-92f734e9c71e)


<!-- Remove if not applicable -->

Fixes #3431

## Before submitting:

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
<!--- If you plan to update documentation or tests in follow-up, please note -->

- [ ] Does the Pull Request follow [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/#summary) naming? Here are [GitHub's
      guide](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) on how to create a pull request.
- [ ] Does the code follow BentoML's code style, `pre-commit run -a` script has passed ([instructions](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md#style-check-auto-formatting-type-checking))?
- [ ] Did you read through [contribution guidelines](https://github.com/bentoml/BentoML/blob/main/CONTRIBUTING.md#ways-to-contribute) and follow [development guidelines](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md#start-developing)?
- [ ] Did your changes require updates to the documentation? Have you updated
      those accordingly? Here are [documentation guidelines](https://github.com/bentoml/BentoML/tree/main/docs) and [tips on writting docs](https://github.com/bentoml/BentoML/tree/main/docs#writing-documentation).
- [ ] Did you write tests to cover your changes?


Problem statement:
bug: Bentos performance issues
### Describe the bug

I've been conducting load testing on Bento using the MobileNetV2 model. I've observed that the CPU is fully utilized and upon profiling the app, I discovered that the workers are consuming all of the CPU, despite the relatively low RPS (around 30). To further investigate this issue, I decided to remove the model inference and test the load with an empty runner that receives an image and returns it without any computation. Despite this, the CPU is still fully utilized at a very low RPS (around 30). Additionally, the latency is quite high - 1900ms on the 50th percentile.
I've also compared Bento with FastAPI and exposed the same model on an endpoint, and FastAPI outperforms Bentos significantly. I also couldn't see any improvement when running the adaptive batching inference.


### To reproduce

Here's the code to reproduce the performance issue i faced (no inference in this case, just HTTP endpoint with logic-less runner. (it also happens when inferencing ofc)):

```
class CustomRunnable(bentoml.Runnable):
    SUPPORTED_RESOURCES = ("nvidia.com/gpu",)
    @bentoml.Runnable.method(batchable=False)
    def classify_batch(self, input_batch):
       return input_batch


runner = bentoml.Runner(CustomRunnable, name="gore_classifier_runnable")

svc = bentoml.Service("gore-binary-classifier", runners=[runner])

@svc.api(input=Image(),
         output=JSON(pydantic_model=ImageClassifierResponse),
         route="/classify/image",
         )
async def classify_image(req: Image) -> ImageClassifierResponse:
    """running the dummy runner"""
    outputs = await runner.async_run(req)
    
    """return hard coded value"""
    category, conf = "ooc", 0.1231


    return ImageClassifierResponse(request_id=1,
                                   predicted_category=category,
                                   confidence=conf)
```

The maximum RPS I've managed to get here is around ~30 then i observed that the CPU is fully utilized. (to make a comparison, i ran the same test on fastAPI, i've used a way smaller VM (4 CPUs compared to Bentos 16cpus VM) and got ~400RPS for the exact same operation with way lower latencies)



### Expected behavior

i'd expect Bento to provide a reasonable performance (at least the same as FastAPI which and i understand is based on)

### Environment

VM type - n1-standard-16(16 CPU + 60G Memory) + Tesla T4 GPU (1 GPU)
Image created using Bento’s image Containerize command and deployed on GKE Cluster using 1 pod, which gets all node resources.

here's the ```bentoml env``` output - 

```
#### Environment variable

```bash
BENTOML_DEBUG=''
BENTOML_QUIET=''
BENTOML_BUNDLE_LOCAL_BUILD=''
BENTOML_DO_NOT_TRACK=''
BENTOML_CONFIG=''
BENTOML_CONFIG_OPTIONS=''
BENTOML_PORT=''
BENTOML_HOST=''
BENTOML_API_WORKERS=''
```

#### System information

`bentoml`: 1.0.12
`python`: 3.9.16
`platform`: Linux-5.4.188+-x86_64-with-glibc2.31
`uid_gid`: 0:0
<details><summary><code>pip_packages</code></summary>

<br>

```
aiohttp==3.8.3
aiosignal==1.3.1
anyio==3.6.2
appdirs==1.4.4
asgiref==3.6.0
async-timeout==4.0.2
attrs==22.2.0
backoff==2.2.1
bentoml==1.0.12
build==0.10.0
cattrs==22.2.0
certifi==2019.11.28
chardet==3.0.4
charset-normalizer==2.1.1
circus==0.18.0
click==8.1.3
click-option-group==0.5.5
cloudpickle==2.2.0
commonmark==0.9.1
contextlib2==21.6.0
dbus-python==1.2.16
deepmerge==1.1.0
Deprecated==1.2.13
exceptiongroup==1.1.0
filetype==1.2.0
frozenlist==1.3.3
fs==2.4.16
googleapis-common-protos==1.58.0
h11==0.14.0
idna==2.8
Jinja2==3.1.2
MarkupSafe==2.1.1
multidict==6.0.4
numpy==1.24.1
opentelemetry-api==1.14.0
opentelemetry-exporter-otlp-proto-http==1.14.0
opentelemetry-instrumentation==0.35b0
opentelemetry-instrumentation-aiohttp-client==0.35b0
opentelemetry-instrumentation-asgi==0.35b0
opentelemetry-proto==1.14.0
opentelemetry-sdk==1.14.0
opentelemetry-semantic-conventions==0.35b0
opentelemetry-util-http==0.35b0
packaging==21.3
pathspec==0.10.3
Pillow==9.2.0
pip-requirements-parser==32.0.1
pip-tools==6.12.1
prometheus-client==0.15.0
protobuf==3.20.3
psutil==5.9.4
pydantic==1.8.2
Pygments==2.14.0
PyGObject==3.36.0
pynvml==11.4.1
pyparsing==3.0.9
pyproject_hooks==1.0.0
python-apt==2.0.0+ubuntu0.20.4.8
python-dateutil==2.8.2
python-json-logger==2.0.4
python-multipart==0.0.5
PyYAML==6.0
pyzmq==25.0.0
requests==2.22.0
requests-unixsocket==0.2.0
rich==13.1.0
schema==0.7.5
simple-di==0.1.5
six==1.14.0
sniffio==1.3.0
starlette==0.23.1
tomli==2.0.1
torch==1.11.0
torchvision==0.12.0
tornado==6.2
typing_extensions==4.4.0
urllib3==1.25.8
uvicorn==0.20.0
watchfiles==0.18.1
wrapt==1.14.1
yarl==1.8.2
```

</details>

```

Hints from issue discussion:
Hi there, can you send us the benchmark code that you have and the framework you are using for the testing?
Hey, I've posted the code above to reproduce the issue. (it's basically running a logic-less runner, but ofc happens also when running model inference).
regarding the framework for testing - I used locust.

Oh I'm asking for the code that you are running against the server. Are you using something like locust to get the RPS or something else.
yeah i'm using locust and there's an RPS indicator there which tells you how much RPS the server handles. (i also checked it with bento built in metrics and it seem to be consistent).

here's the locust code - 


```
from locust import HttpUser, between, task

class WebsiteUser(HttpUser):
    wait_time = between(1, 1.2)


    @task
    def classify(self):

        with open('/locust-tasks/test_img.jpg', "rb") as f:
            URL = "/classify/image"
            self.client.post(URL, files={"upload_file": ("foo.jpeg", f, "image/jpeg")})
```

so basically i'm just sending the same image to be inferenced over and over again..
I have noticed a similar behaviour. Using bentoml without runners seems to be a lot faster and less resource intensive than using runners. Is there any performance comparison benchmark that we can look into for Bentoml and FastAPI?

Commit message: fix: change errors

Signed-off-by: Frost Ming <me@frostming.com>

Patch to review:
diff --git a/src/bentoml/_internal/external_typing/__init__.py b/src/bentoml/_internal/external_typing/__init__.py
index 8fea4d2d365..9cddd093ca3 100644
--- a/src/bentoml/_internal/external_typing/__init__.py
+++ b/src/bentoml/_internal/external_typing/__init__.py
@@ -34,6 +34,8 @@
     from .starlette import ASGIReceive
     from .starlette import AsgiMiddleware
 
+    from PIL.Image import Image as PILImage
+
     WSGIApp = t.Callable[[F, t.Mapping[str, t.Any]], t.Iterable[bytes]]
 
     __all__ = [
@@ -41,6 +43,7 @@
         "PdDataFrame",
         "PdDType",
         "PdDTypeArg",
+        "PILImage",
         "DataFrameOrient",
         "SeriesOrient",
         "ObjectID",
diff --git a/src/bentoml/_internal/runner/container.py b/src/bentoml/_internal/runner/container.py
index e330dc06ab1..bc65b357844 100644
--- a/src/bentoml/_internal/runner/container.py
+++ b/src/bentoml/_internal/runner/container.py
@@ -2,6 +2,7 @@
 
 import abc
 import base64
+import io
 import itertools
 import pickle
 import typing as t
@@ -148,9 +149,7 @@ def batch_to_payloads(
     @classmethod
     @abc.abstractmethod
     def from_batch_payloads(
-        cls,
-        payloads: t.Sequence[Payload],
-        batch_dim: int,
+        cls, payloads: t.Sequence[Payload], batch_dim: int
     ) -> tuple[BatchType, list[int]]:
         ...
 
@@ -450,6 +449,51 @@ def from_batch_payloads(  # pylint: disable=arguments-differ
         return cls.batches_to_batch(batches, batch_dim)
 
 
+class PILImageContainer(DataContainer["ext.PILImage", "ext.PILImage"]):
+    _error = (
+        "PIL.Image doesn't support batch inference."
+        "You can convert it to numpy.ndarray before passing to the runner."
+    )
+
+    @classmethod
+    def to_payload(cls, batch: ext.PILImage, batch_dim: int) -> Payload:
+        buffer = io.BytesIO()
+        batch.save(buffer, format=batch.format)
+        return cls.create_payload(buffer.getvalue(), batch_size=1)
+
+    @classmethod
+    def from_payload(cls, payload: Payload) -> ext.PILImage:
+        from ..io_descriptors.image import PIL
+
+        PIL.Image.Image.size
+
+        return PIL.Image.open(io.BytesIO(payload.data))
+
+    @classmethod
+    def batch_to_payloads(
+        cls, batch: ext.PILImage, indices: t.Sequence[int], batch_dim: int
+    ) -> list[Payload]:
+        raise NotImplementedError(cls._error)
+
+    @classmethod
+    def batches_to_batch(
+        cls, batches: t.Sequence[ext.PILImage], batch_dim: int
+    ) -> tuple[ext.PILImage, list[int]]:
+        raise NotImplementedError(cls._error)
+
+    @classmethod
+    def batch_to_batches(
+        cls, batch: ext.PILImage, indices: t.Sequence[int], batch_dim: int
+    ) -> list[ext.PILImage]:
+        raise NotImplementedError(cls._error)
+
+    @classmethod
+    def from_batch_payloads(
+        cls, payloads: t.Sequence[Payload], batch_dim: int = 0
+    ) -> tuple[ext.PILImage, list[int]]:
+        raise NotImplementedError(cls._error)
+
+
 class DefaultContainer(DataContainer[t.Any, t.List[t.Any]]):
     @classmethod
     def batches_to_batch(
@@ -540,20 +584,24 @@ def find_by_single_type(
         cls, type_: t.Type[SingleType] | LazyType[SingleType]
     ) -> t.Type[DataContainer[SingleType, t.Any]]:
         typeref = LazyType.from_type(type_)
-        return cls.CONTAINER_SINGLE_TYPE_MAP.get(
-            typeref,
-            DefaultContainer,
-        )
+        if typeref in cls.CONTAINER_SINGLE_TYPE_MAP:
+            return cls.CONTAINER_SINGLE_TYPE_MAP[typeref]
+        for klass in cls.CONTAINER_SINGLE_TYPE_MAP:
+            if klass.issubclass(type_):
+                return cls.CONTAINER_SINGLE_TYPE_MAP[klass]
+        return DefaultContainer
 
     @classmethod
     def find_by_batch_type(
         cls, type_: t.Type[BatchType] | LazyType[BatchType]
     ) -> t.Type[DataContainer[t.Any, BatchType]]:
         typeref = LazyType.from_type(type_)
-        return cls.CONTAINER_BATCH_TYPE_MAP.get(
-            typeref,
-            DefaultContainer,
-        )
+        if typeref in cls.CONTAINER_BATCH_TYPE_MAP:
+            return cls.CONTAINER_BATCH_TYPE_MAP[typeref]
+        for klass in cls.CONTAINER_BATCH_TYPE_MAP:
+            if klass.issubclass(type_):
+                return cls.CONTAINER_BATCH_TYPE_MAP[klass]
+        return DefaultContainer
 
     @classmethod
     def find_by_name(cls, name: str) -> t.Type[DataContainer[t.Any, t.Any]]:
@@ -600,6 +648,11 @@ def register_builtin_containers():
         LazyType("tritonclient.http.aio", "InferInput"),
         TritonInferInputDataContainer,
     )
+    DataContainerRegistry.register_container(
+        LazyType("PIL.Image", "Image"),
+        LazyType("PIL.Image", "Image"),
+        PILImageContainer,
+    )
 
 
 register_builtin_containers()
diff --git a/src/bentoml/_internal/types.py b/src/bentoml/_internal/types.py
index 06c02f722c6..81b14d2f161 100644
--- a/src/bentoml/_internal/types.py
+++ b/src/bentoml/_internal/types.py
@@ -255,6 +255,12 @@ def isinstance(self, obj: t.Any) -> t.TypeGuard[T]:
         except ValueError:
             return False
 
+    def issubclass(self, klass: type) -> bool:
+        try:
+            return issubclass(klass, self.get_class(import_module=False))
+        except ValueError:
+            return False
+
 
 if TYPE_CHECKING:
     from types import UnionType


Code review comments:
diff_hunk: @@ -450,6 +449,51 @@ def from_batch_payloads(  # pylint: disable=arguments-differ
         return cls.batches_to_batch(batches, batch_dim)
 
 
+class PILImageContainer(DataContainer["ext.PILImage", "ext.PILImage"]):
+    _error = (
+        "PIL.Image doesn't support batch inference."
+        "You can convert it to numpy.ndarray before passing to the runner."
+    )
+
+    @classmethod
+    def to_payload(cls, batch: ext.PILImage, batch_dim: int) -> Payload:
+        buffer = io.BytesIO()
+        batch.save(buffer, format=batch.format)
+        return cls.create_payload(buffer.getvalue(), batch_size=1)
+
+    @classmethod
+    def from_payload(cls, payload: Payload) -> ext.PILImage:
+        from ..io_descriptors.image import PIL
+
+        PIL.Image.Image.size
path: src/bentoml/_internal/runner/container.py
text: @user1:
Why are we doing this? Is there any side-effect for here?

@author:
oh, it was a mistake