Repository: Project-MONAI/MONAI

Issue:
Auto3DSeg tests improvement on mGPU ensemble
**Describe the bug**

It runs on single GPU, but fails on 4-GPU.

```
2023-04-28 13:02:27,188 - INFO - Ensembling using 4 GPU!
WARNING:torch.distributed.run:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
2023-04-28 13:02:33,112 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,126 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,159 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
2023-04-28 13:02:33,272 - INFO - The output_dir is not specified. /opt/toolkit/tutorials/monai/auto3dseg/notebooks/work_dir/ensemble_output will be used to save ensemble predictions.
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/__main__.py", line 24, in <module>
    fire.Fire(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.8/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 544, in ensemble
    self.ensembler(pred_param=self.kwargs)
  File "/usr/local/lib/python3.8/dist-packages/monai/apps/auto3dseg/ensemble_builder.py", line 155, in __call__
    files = files[slices]
TypeError: list indices must be integers or slices, not str
```

The cause may be that there are only 2 testing images, and the number of testing images is less than number of GPUs


Discussion:

Resolving patch:
diff --git a/monai/apps/auto3dseg/ensemble_builder.py b/monai/apps/auto3dseg/ensemble_builder.py
--- a/monai/apps/auto3dseg/ensemble_builder.py
+++ b/monai/apps/auto3dseg/ensemble_builder.py
@@ -136,14 +136,22 @@ def __call__(self, pred_param: dict | None = None) -> list:
                 in this function, and the second group will be passed to the `InferClass` to override the
                 parameters of the class functions.
                 The first group contains:
-                'files_slices': a value type of `slice`. The files_slices will slice the infer_files and only
-                    make prediction on the infer_files[file_slices].
-                'mode': ensemble mode. Currently "mean" and "vote" (majority voting) schemes are supported.
-                'sigmoid': use the sigmoid function (e.g. x>0.5) to convert the prediction probability map to
-                    the label class prediction, otherwise argmax(x) is used.
+
+                    - ``"infer_files"``: file paths to the images to read in a list.
+                    - ``"files_slices"``: a value type of `slice`. The files_slices will slice the ``"infer_files"`` and
+                      only make prediction on the infer_files[file_slices].
+                    - ``"mode"``: ensemble mode. Currently "mean" and "vote" (majority voting) schemes are supported.
+                    - ``"image_save_func"``: a dictionary used to instantiate the ``SaveImage`` transform. When specified,
+                      the ensemble prediction will save the prediciton files, instead of keeping the files in the memory.
+                      Example: `{"_target_": "SaveImage", "output_dir": "./"}`
+                    - ``"sigmoid"``: use the sigmoid function (e.g. x > 0.5) to convert the prediction probability map
+                      to the label class prediction, otherwise argmax(x) is used.
+
+                The parameters in the second group is defined in the ``config`` of each Algo templates. Please check:
+                https://github.com/Project-MONAI/research-contributions/tree/main/auto3dseg/algorithm_templates
 
         Returns:
-            A list of tensors.
+            A list of tensors or file paths, depending on whether ``"image_save_func"`` is set.
         """
         param = {} if pred_param is None else deepcopy(pred_param)
         files = self.infer_files


Test patch:
diff --git a/tests/test_auto3dseg_ensemble.py b/tests/test_auto3dseg_ensemble.py
--- a/tests/test_auto3dseg_ensemble.py
+++ b/tests/test_auto3dseg_ensemble.py
@@ -46,7 +46,7 @@
 num_images_per_batch = 2
 
 fake_datalist: dict[str, list[dict]] = {
-    "testing": [{"image": "val_001.fake.nii.gz"}, {"image": "val_002.fake.nii.gz"}],
+    "testing": [{"image": f"ts_image_{idx:03d}.nii.gz"} for idx in range(num_images_perfold)],
     "training": [
         {
             "fold": f,
diff --git a/tests/test_integration_autorunner.py b/tests/test_integration_autorunner.py
--- a/tests/test_integration_autorunner.py
+++ b/tests/test_integration_autorunner.py
@@ -38,7 +38,7 @@
 num_images_per_batch = 2
 
 sim_datalist: dict[str, list[dict]] = {
-    "testing": [{"image": "val_001.fake.nii.gz"}, {"image": "val_002.fake.nii.gz"}],
+    "testing": [{"image": f"ts_image__{idx:03d}.nii.gz"} for idx in range(num_images_perfold)],
     "training": [
         {
             "fold": f,