Repository: bentoml/BentoML
Pull request #3776 (2023-04-20 04:01:30)

feat: Allow spawning more than 1 worker on each resource
## What does this PR address?

Add a new runner configuration `workers_per_resource` to spawn multiple workers on each allocated resource. The default value is 1.

The effect of this configuration varies depending on whether the CPU or GPU is used.

- CPU and multi-threading=false
  * Spawns `workers_per_resources * cpu` workers
- CPU and multi-threading=true
  * Spawns `workers_per_resources` workers with `cpu` threads on each worker.
- GPU
   * Spawns `workers_per_resources` workers on each GPU.

## Before submitting:

<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
<!--- If you plan to update documentation or tests in follow-up, please note -->

- [x] Does the Pull Request follow [Conventional Commits specification](https://www.conventionalcommits.org/en/v1.0.0/#summary) naming? Here are [GitHub's
      guide](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request) on how to create a pull request.
- [x] Does the code follow BentoML's code style, both `make format` and `make lint` script have passed ([instructions](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md#style-check-auto-formatting-type-checking))?
- [x] Did you read through [contribution guidelines](https://github.com/bentoml/BentoML/blob/main/CONTRIBUTING.md#ways-to-contribute) and follow [development guidelines](https://github.com/bentoml/BentoML/blob/main/DEVELOPMENT.md#start-developing)?
- [x] Did your changes require updates to the documentation? Have you updated
      those accordingly? Here are [documentation guidelines](https://github.com/bentoml/BentoML/tree/main/docs) and [tips on writting docs](https://github.com/bentoml/BentoML/tree/main/docs#writing-documentation).
- [x] Did you write tests to cover your changes?


codecov[bot]: ## [Codecov](https://codecov.io/gh/bentoml/BentoML/pull/3776?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml) Report
> Merging [#3776](https://codecov.io/gh/bentoml/BentoML/pull/3776?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml) (ff7f660) into [main](https://codecov.io/gh/bentoml/BentoML/commit/f11b16665a46f0fd37612764add6089602bd2da3?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml) (f11b166) will **not change** coverage.
> The diff coverage is `0.00%`.

[![Impacted file tree graph](https://codecov.io/gh/bentoml/BentoML/pull/3776/graphs/tree.svg?width=650&height=150&src=pr&token=GMzgXdpE5b&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml)](https://codecov.io/gh/bentoml/BentoML/pull/3776?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml)

```diff
@@          Coverage Diff           @@
##            main   #3776    +/-   ##
======================================
  Coverage   0.00%   0.00%            
======================================
  Files        149     154     +5     
  Lines      12447   12614   +167     
======================================
- Misses     12447   12614   +167     
```

| [Impacted Files](https://codecov.io/gh/bentoml/BentoML/pull/3776?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml) | Coverage Δ | |
|---|---|---|
| [src/bentoml/\_internal/runner/runner.py](https://codecov.io/gh/bentoml/BentoML/pull/3776?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml#diff-c3JjL2JlbnRvbWwvX2ludGVybmFsL3J1bm5lci9ydW5uZXIucHk=) | `0.00% <0.00%> (ø)` | |
| [src/bentoml/\_internal/runner/strategy.py](https://codecov.io/gh/bentoml/BentoML/pull/3776?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml#diff-c3JjL2JlbnRvbWwvX2ludGVybmFsL3J1bm5lci9zdHJhdGVneS5weQ==) | `0.00% <0.00%> (ø)` | |

... and [7 files with indirect coverage changes](https://codecov.io/gh/bentoml/BentoML/pull/3776/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=bentoml)


Patch:
diff --git a/docs/source/concepts/runner.rst b/docs/source/concepts/runner.rst
index d5ebda25..ce54560c 100644
--- a/docs/source/concepts/runner.rst
+++ b/docs/source/concepts/runner.rst
@@ -228,7 +228,7 @@ Sequential Runs
         ocr_text = ocr_runner.run(input)
         return transformers_runner.run(ocr_text)
 
-It’s as simple as creating two runners and invoking them synchronously in your prediction endpoint. Note that an async endpoint is often preferred in these use cases as the primary event loop is yielded while waiting for other IO-expensive tasks. 
+It’s as simple as creating two runners and invoking them synchronously in your prediction endpoint. Note that an async endpoint is often preferred in these use cases as the primary event loop is yielded while waiting for other IO-expensive tasks.
 
 For example, the same API above can be achieved as an ``async`` endpoint:
 
@@ -416,6 +416,8 @@ can be specified for the ``nvidia.com/gpu`` key. For example, the following conf
               resources:
                 nvidia.com/gpu: [2, 4]
 
+For the detailed information on the meaning of each resource allocation configuration, see :doc:`/guides/scheduling`.
+
 Timeout
 ^^^^^^^
 
diff --git a/docs/source/guides/index.rst b/docs/source/guides/index.rst
index d535878c..b162f33e 100644
--- a/docs/source/guides/index.rst
+++ b/docs/source/guides/index.rst
@@ -36,6 +36,10 @@ into this part of the documentation.
         :link: /guides/configuration
         :link-type: doc
 
+    .. grid-item-card:: :doc:`/guides/scheduling`
+        :link: /guides/scheduling
+        :link-type: doc
+
     .. grid-item-card:: :doc:`/guides/graph`
         :link: /guides/graph
         :link-type: doc
@@ -84,6 +88,7 @@ into this part of the documentation.
     client
     server
     configuration
+    scheduling
     envmanager
     graph
     monitoring
diff --git a/docs/source/guides/scheduling.rst b/docs/source/guides/scheduling.rst
new file mode 100644
index 00000000..16c907c9
--- /dev/null
+++ b/docs/source/guides/scheduling.rst
@@ -0,0 +1,128 @@
+============================
+Resource Scheduling Strategy
+============================
+
+To make the best use of machine resources, BentoML provides a few configuration options to control how many runner workers
+will be spawned and how they distribute on the CPUs and GPUs. It can be configured for all runners as well as individual runner.
+The following configuration options are available:
+
+.. tab-set::
+
+    .. tab-item:: Global configuration (Applies to all runners)
+       :sync: all_runners
+
+       .. code-block:: yaml
+          :caption: ⚙️ `configuration.yml`
+
+          runners:
+            resources:
+              cpu: 4
+              nvidia.com/gpu: 2
+            workers_per_resource: 2
+
+    .. tab-item:: Individual Runner
+        :sync: individual_runner
+
+        .. code-block:: yaml
+           :caption: ⚙️ `configuration.yml`
+
+           runners:
+             iris_clf:
+               resources:
+                 cpu: 4
+                 nvidia.com/gpu: 2
+               workers_per_resource: 2
+
+Configuration Options
+---------------------
+
+``runners.resources.cpu``
+    A float value indicating the number of processes(rounded up) to spawn. If the runner supports multi-threading, it will be the number of threads per process.
+``runners.resources.nvidia.com/gpu``
+    An integer indicating the number of GPUs to use. If the runner supports running on GPU, it will create a number of instances defined by ``runners.workers_per_resource`` on each allocated GPU.
+    Alternatively, you can specify what GPUs to use by setting it to a list of GPU IDs, e.g. ``[2, 4]``, the IDs are 1-indexed.
+``runners.workers_per_resource``
+    A resource multiplier to control how many workers will be spawned on each allocated resource. The default value is 1.
+
+Examples
+--------
+
+The following configuration spawns 4 workers on CPU for runners not supporting multi-threading. If the runner supports multi-threading, it will be 4 threads with only one worker.
+
+.. tab-set::
+
+    .. tab-item:: Global configuration (Applies to all runners)
+       :sync: all_runners
+
+       .. code-block:: yaml
+          :caption: ⚙️ `configuration.yml`
+
+          runners:
+            resources:
+              cpu: 4
+
+    .. tab-item:: Individual Runner
+        :sync: individual_runner
+
+        .. code-block:: yaml
+           :caption: ⚙️ `configuration.yml`
+
+           runners:
+             iris_clf:
+               resources:
+                 cpu: 4
+
+When running in multi-threading mode, the following configuration will create 2 workers with 4 threads each.
+However, if multi-threading is not supported, it will instead create 8 workers by multiplying 2 and 4.
+
+.. tab-set::
+
+    .. tab-item:: Global configuration (Applies to all runners)
+       :sync: all_runners
+
+       .. code-block:: yaml
+          :caption: ⚙️ `configuration.yml`
+
+          runners:
+            resources:
+              cpu: 4
+            workers_per_resource: 2
+
+    .. tab-item:: Individual Runner
+        :sync: individual_runner
+
+        .. code-block:: yaml
+           :caption: ⚙️ `configuration.yml`
+
+           runners:
+             iris_clf:
+               resources:
+                 cpu: 4
+               workers_per_resource: 2
+
+If the runner supports running on GPU, the following configuration will spawn 2 workers on each GPU, hence 4 workers will be spawned for 2 GPUs in total:
+
+.. tab-set::
+
+    .. tab-item:: Global configuration (Applies to all runners)
+       :sync: all_runners
+
+       .. code-block:: yaml
+          :caption: ⚙️ `configuration.yml`
+
+          runners:
+            resources:
+              nvidia.com/gpu: 2
+            workers_per_resource: 2
+
+    .. tab-item:: Individual Runner
+        :sync: individual_runner
+
+        .. code-block:: yaml
+           :caption: ⚙️ `configuration.yml`
+
+           runners:
+             iris_clf:
+               resources:
+                 nvidia.com/gpu: 2
+               workers_per_resource: 2
diff --git a/pyproject.toml b/pyproject.toml
index 9d12e833..ca2909c9 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -268,6 +268,7 @@ convention = "google"
 
 [tool.ruff.isort]
 lines-after-imports = 2
+force-single-line = true
 
 [tool.isort]
 profile = "black"
diff --git a/src/bentoml/_internal/configuration/containers.py b/src/bentoml/_internal/configuration/containers.py
index 2c93e57b..96f30bbe 100644
--- a/src/bentoml/_internal/configuration/containers.py
+++ b/src/bentoml/_internal/configuration/containers.py
@@ -139,7 +139,14 @@ class BentoMLConfiguration:
                 ) from None
 
     def _finalize(self):
-        RUNNER_CFG_KEYS = ["batching", "resources", "logging", "metrics", "timeout"]
+        RUNNER_CFG_KEYS = [
+            "batching",
+            "resources",
+            "logging",
+            "metrics",
+            "timeout",
+            "workers_per_resource",
+        ]
         global_runner_cfg = {k: self.config["runners"][k] for k in RUNNER_CFG_KEYS}
         custom_runners_cfg = dict(
             filter(
diff --git a/src/bentoml/_internal/configuration/v1/__init__.py b/src/bentoml/_internal/configuration/v1/__init__.py
index 1509bf4e..9d24dff2 100644
--- a/src/bentoml/_internal/configuration/v1/__init__.py
+++ b/src/bentoml/_internal/configuration/v1/__init__.py
@@ -146,6 +146,7 @@ _RUNNER_CONFIG = {
     # NOTE: there is a distinction between being unset and None here; if set to 'None'
     # in configuration for a specific runner, it will override the global configuration.
     s.Optional("resources"): s.Or({s.Optional(str): object}, lambda s: s == "system", None),  # type: ignore (incomplete schema typing)
+    s.Optional("workers_per_resource"): s.And(int, ensure_larger_than_zero),
     s.Optional("logging"): {
         s.Optional("access"): {
             s.Optional("enabled"): bool,
diff --git a/src/bentoml/_internal/configuration/v1/default_configuration.yaml b/src/bentoml/_internal/configuration/v1/default_configuration.yaml
index 831d3de7..335f21b0 100644
--- a/src/bentoml/_internal/configuration/v1/default_configuration.yaml
+++ b/src/bentoml/_internal/configuration/v1/default_configuration.yaml
@@ -80,6 +80,7 @@ api_server:
     period: 10
 runners:
   resources: ~
+  workers_per_resource: 1
   timeout: 300
   batching:
     enabled: true
diff --git a/src/bentoml/_internal/runner/runner.py b/src/bentoml/_internal/runner/runner.py
index 7441cc20..d73c2b14 100644
--- a/src/bentoml/_internal/runner/runner.py
+++ b/src/bentoml/_internal/runner/runner.py
@@ -104,7 +104,6 @@ class AbstractRunner(ABC):
 
 @attr.define(slots=False, frozen=True, eq=False)
 class Runner(AbstractRunner):
-
     if t.TYPE_CHECKING:
         # This will be set by __init__. This is for type checking only.
         run: t.Callable[..., t.Any]
@@ -125,6 +124,7 @@ class Runner(AbstractRunner):
 
     runner_methods: list[RunnerMethod[t.Any, t.Any, t.Any]]
     scheduling_strategy: type[Strategy]
+    workers_per_resource: int = 1
     runnable_init_params: dict[str, t.Any] = attr.field(
         default=None, converter=attr.converters.default_if_none(factory=dict)
     )
@@ -245,6 +245,7 @@ class Runner(AbstractRunner):
             runnable_class=runnable_class,
             runnable_init_params=runnable_init_params,
             resource_config=config["resources"],
+            workers_per_resource=config.get("workers_per_resource", 1),
             runner_methods=list(runner_method_map.values()),
             scheduling_strategy=scheduling_strategy,
         )
@@ -327,6 +328,7 @@ class Runner(AbstractRunner):
         return self.scheduling_strategy.get_worker_count(
             self.runnable_class,
             self.resource_config,
+            self.workers_per_resource,
         )
 
     @property
@@ -335,6 +337,7 @@ class Runner(AbstractRunner):
             worker_id: self.scheduling_strategy.get_worker_env(
                 self.runnable_class,
                 self.resource_config,
+                self.workers_per_resource,
                 worker_id,
             )
             for worker_id in range(self.scheduled_worker_count)
diff --git a/src/bentoml/_internal/runner/strategy.py b/src/bentoml/_internal/runner/strategy.py
index 4cdd7668..702e53d1 100644
--- a/src/bentoml/_internal/runner/strategy.py
+++ b/src/bentoml/_internal/runner/strategy.py
@@ -19,6 +19,7 @@ class Strategy(abc.ABC):
         cls,
         runnable_class: t.Type[Runnable],
         resource_request: dict[str, t.Any],
+        workers_per_resource: int,
     ) -> int:
         ...
 
@@ -28,6 +29,7 @@ class Strategy(abc.ABC):
         cls,
         runnable_class: t.Type[Runnable],
         resource_request: dict[str, t.Any],
+        workers_per_resource: int,
         worker_index: int,
     ) -> dict[str, t.Any]:
         """
@@ -64,6 +66,7 @@ class DefaultStrategy(Strategy):
         cls,
         runnable_class: t.Type[Runnable],
         resource_request: dict[str, t.Any] | None,
+        workers_per_resource: int,
     ) -> int:
         if resource_request is None:
             resource_request = system_resources()
@@ -75,7 +78,7 @@ class DefaultStrategy(Strategy):
             and len(nvidia_gpus) > 0
             and "nvidia.com/gpu" in runnable_class.SUPPORTED_RESOURCES
         ):
-            return len(nvidia_gpus)
+            return len(nvidia_gpus) * workers_per_resource
 
         # use CPU
         cpus = get_resource(resource_request, "cpu")
@@ -87,9 +90,9 @@ class DefaultStrategy(Strategy):
                 )
 
             if runnable_class.SUPPORTS_CPU_MULTI_THREADING:
-                return 1
+                return workers_per_resource
 
-            return math.ceil(cpus)
+            return math.ceil(cpus) * workers_per_resource
 
         # this should not be reached by user since we always read system resource as default
         raise ValueError(
@@ -102,6 +105,7 @@ class DefaultStrategy(Strategy):
         cls,
         runnable_class: t.Type[Runnable],
         resource_request: dict[str, t.Any] | None,
+        workers_per_resource: int,
         worker_index: int,
     ) -> dict[str, t.Any]:
         """
@@ -125,7 +129,7 @@ class DefaultStrategy(Strategy):
             and len(nvidia_gpus) > 0
             and "nvidia.com/gpu" in runnable_class.SUPPORTED_RESOURCES
         ):
-            dev = str(nvidia_gpus[worker_index])
+            dev = str(nvidia_gpus[worker_index // workers_per_resource])
             environ["CUDA_VISIBLE_DEVICES"] = dev
             logger.info(
                 "Environ for worker %s: set CUDA_VISIBLE_DEVICES to %s",


Test patch:
diff --git a/tests/integration/frameworks/test_frameworks.py b/tests/integration/frameworks/test_frameworks.py
index ec6d3922..61b15f50 100644
--- a/tests/integration/frameworks/test_frameworks.py
+++ b/tests/integration/frameworks/test_frameworks.py
@@ -298,7 +298,7 @@ def test_runner_cpu_multi_threading(
         for meth, inputs in config.test_inputs.items():
             strategy = DefaultStrategy()
 
-            os.environ.update(strategy.get_worker_env(runnable, resource_cfg, 0))
+            os.environ.update(strategy.get_worker_env(runnable, resource_cfg, 1, 0))
 
             runner.init_local()
 
@@ -344,7 +344,7 @@ def test_runner_cpu(
         for meth, inputs in config.test_inputs.items():
             strategy = DefaultStrategy()
 
-            os.environ.update(strategy.get_worker_env(runnable, resource_cfg, 0))
+            os.environ.update(strategy.get_worker_env(runnable, resource_cfg, 1, 0))
 
             runner.init_local()
 
@@ -392,7 +392,7 @@ def test_runner_nvidia_gpu(
         for meth, inputs in config.test_inputs.items():
             strategy = DefaultStrategy()
 
-            os.environ.update(strategy.get_worker_env(runnable, resource_cfg, 0))
+            os.environ.update(strategy.get_worker_env(runnable, resource_cfg, 1, 0))
 
             runner.init_local()
 
diff --git a/tests/unit/_internal/configuration/test_containers.py b/tests/unit/_internal/configuration/test_containers.py
index 253857f8..22c73f07 100644
--- a/tests/unit/_internal/configuration/test_containers.py
+++ b/tests/unit/_internal/configuration/test_containers.py
@@ -57,14 +57,17 @@ runners:
         max_batch_size: 10
     resources:
         cpu: 4
+    workers_per_resource: 1
     logging:
         access:
             enabled: False
     test_runner_1:
         resources: system
+        workers_per_resource: 2
     test_runner_2:
         resources:
             cpu: 2
+        workers_per_resource: 4
     test_runner_gpu:
         resources:
             nvidia.com/gpu: 1
@@ -84,6 +87,7 @@ runners:
     assert test_runner_1["batching"]["enabled"] is False
     assert test_runner_1["batching"]["max_batch_size"] == 10
     assert test_runner_1["logging"]["access"]["enabled"] is False
+    assert test_runner_1["workers_per_resource"] == 2
     # assert test_runner_1["resources"]["cpu"] == 4
 
     # test_runner_2
@@ -92,6 +96,7 @@ runners:
     assert test_runner_2["batching"]["max_batch_size"] == 10
     assert test_runner_2["logging"]["access"]["enabled"] is False
     assert test_runner_2["resources"]["cpu"] == 2
+    assert test_runner_2["workers_per_resource"] == 4
 
     # test_runner_gpu
     test_runner_gpu = runner_cfg["test_runner_gpu"]
@@ -100,6 +105,7 @@ runners:
     assert test_runner_gpu["logging"]["access"]["enabled"] is False
     assert test_runner_gpu["resources"]["cpu"] == 4  # should use global
     assert test_runner_gpu["resources"]["nvidia.com/gpu"] == 1
+    assert test_runner_gpu["workers_per_resource"] == 1
 
     # test_runner_batching
     test_runner_batching = runner_cfg["test_runner_batching"]
@@ -117,18 +123,22 @@ def test_runner_gpu_configuration(
 runners:
     resources:
         nvidia.com/gpu: [1, 2, 4]
+    workers_per_resource: 1
 """
     bentoml_cfg = container_from_file(GPU_INDEX)
     assert bentoml_cfg["runners"]["resources"] == {"nvidia.com/gpu": [1, 2, 4]}
+    assert bentoml_cfg["runners"]["workers_per_resource"] == 1
 
     GPU_INDEX_WITH_STRING = """\
 runners:
     resources:
         nvidia.com/gpu: "[1, 2, 4]"
+    workers_per_resource: 2
 """
     bentoml_cfg = container_from_file(GPU_INDEX_WITH_STRING)
     # this behaviour can be confusing
     assert bentoml_cfg["runners"]["resources"] == {"nvidia.com/gpu": "[1, 2, 4]"}
+    assert bentoml_cfg["runners"]["workers_per_resource"] == 2
 
 
 @pytest.mark.usefixtures("container_from_file")
diff --git a/tests/unit/_internal/runner/test_strategy.py b/tests/unit/_internal/runner/test_strategy.py
index efdca84d..3462a9e2 100644
--- a/tests/unit/_internal/runner/test_strategy.py
+++ b/tests/unit/_internal/runner/test_strategy.py
@@ -17,23 +17,75 @@ class GPURunnable(bentoml.Runnable):
     SUPPORTED_RESOURCES = ("nvidia.com/gpu",)
 
 
+class SingleThreadRunnable(bentoml.Runnable):
+    SUPPORTED_RESOURCES = ("cpu",)
+    SUPPORTS_CPU_MULTI_THREADING = False
+
+
+class MultiThreadRunnable(bentoml.Runnable):
+    SUPPORTED_RESOURCES = ("cpu",)
+    SUPPORTS_CPU_MULTI_THREADING = True
+
+
 def unvalidated_get_resource(x: t.Dict[str, t.Any], y: str):
     return get_resource(x, y, validate=False)
 
 
 def test_default_gpu_strategy(monkeypatch: MonkeyPatch):
     monkeypatch.setattr(strategy, "get_resource", unvalidated_get_resource)
-    assert DefaultStrategy.get_worker_count(GPURunnable, {"nvidia.com/gpu": 2}) == 2
+    assert DefaultStrategy.get_worker_count(GPURunnable, {"nvidia.com/gpu": 2}, 1) == 2
+    assert DefaultStrategy.get_worker_count(GPURunnable, {"nvidia.com/gpu": 2}, 2) == 4
     assert pytest.raises(
-        ValueError, DefaultStrategy.get_worker_count, GPURunnable, {"nvidia.com/gpu": 0}
+        ValueError,
+        DefaultStrategy.get_worker_count,
+        GPURunnable,
+        {"nvidia.com/gpu": 0},
+        1,
     )
     assert (
-        DefaultStrategy.get_worker_count(GPURunnable, {"nvidia.com/gpu": [2, 7]}) == 2
+        DefaultStrategy.get_worker_count(GPURunnable, {"nvidia.com/gpu": [2, 7]}, 1)
+        == 2
+    )
+    assert (
+        DefaultStrategy.get_worker_count(GPURunnable, {"nvidia.com/gpu": [2, 7]}, 2)
+        == 4
     )
 
-    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 0)
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 1, 0)
     assert envs.get("CUDA_VISIBLE_DEVICES") == "0"
-    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 1)
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 1, 1)
     assert envs.get("CUDA_VISIBLE_DEVICES") == "1"
-    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": [2, 7]}, 1)
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": [2, 7]}, 1, 1)
     assert envs.get("CUDA_VISIBLE_DEVICES") == "7"
+
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 2, 0)
+    assert envs.get("CUDA_VISIBLE_DEVICES") == "0"
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 2, 1)
+    assert envs.get("CUDA_VISIBLE_DEVICES") == "0"
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": 2}, 2, 2)
+    assert envs.get("CUDA_VISIBLE_DEVICES") == "1"
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": [2, 7]}, 2, 1)
+    assert envs.get("CUDA_VISIBLE_DEVICES") == "2"
+    envs = DefaultStrategy.get_worker_env(GPURunnable, {"nvidia.com/gpu": [2, 7]}, 2, 2)
+    assert envs.get("CUDA_VISIBLE_DEVICES") == "7"
+
+
+def test_default_cpu_strategy(monkeypatch: MonkeyPatch):
+    monkeypatch.setattr(strategy, "get_resource", unvalidated_get_resource)
+    assert DefaultStrategy.get_worker_count(SingleThreadRunnable, {"cpu": 2}, 1) == 2
+    assert DefaultStrategy.get_worker_count(SingleThreadRunnable, {"cpu": 0.5}, 1) == 1
+    assert DefaultStrategy.get_worker_count(SingleThreadRunnable, {"cpu": 2}, 2) == 4
+    assert DefaultStrategy.get_worker_count(MultiThreadRunnable, {"cpu": 4}, 1) == 1
+    assert DefaultStrategy.get_worker_count(MultiThreadRunnable, {"cpu": 4}, 2) == 2
+
+    envs = DefaultStrategy.get_worker_env(SingleThreadRunnable, {"cpu": 4}, 1, 0)
+    assert envs["CUDA_VISIBLE_DEVICES"] == "-1"
+    assert envs["BENTOML_NUM_THREAD"] == "1"
+
+    envs = DefaultStrategy.get_worker_env(SingleThreadRunnable, {"cpu": 4}, 1, 1)
+    assert envs["BENTOML_NUM_THREAD"] == "1"
+
+    envs = DefaultStrategy.get_worker_env(MultiThreadRunnable, {"cpu": 4}, 1, 0)
+    assert envs["BENTOML_NUM_THREAD"] == "4"
+    envs = DefaultStrategy.get_worker_env(MultiThreadRunnable, {"cpu": 3.5}, 1, 1)
+    assert envs["BENTOML_NUM_THREAD"] == "4"