Repository: lightly-ai/lightly

Issue:
loader.num_workers=12 doesn't actually spawn multiple workers
I've noticed that when trying to create an embedding with the param `loader.num_workers` there seems to be no difference in CPU consumption;
`lightly-embed input_dir='./75000' trainer.max_epochs=1 loader.num_workers=12`

This could be that my images are ~8kb and there is some other bottleneck preventing the CPUs to max-out? in My ActivityMonitor (macOSX) I see that I have 80% idle CPU

System: MacBook Pro (16-inch, 2019), 2.6 GHz 6-Core Intel Core i7
OS: MacOSX 11.2.3





Discussion:
also, using `lightly-upload token='asdad' dataset_id='asdasd' input_dir=./75000 loader.batch_size=64 loader.num_workers=12` doesnt actually spawn 12, but it reports that it spawn 8 workers.
## lightly-embed not using CPU fully:
This is quite normal, something is the bottleneck which does not represent 100% utilisation (e.g. memory speed, cache, ...)

## lightly-upload using 8 workers:

@japrescott You are right, the function
https://github.com/lightly-ai/lightly/blob/fc50052f9b2dc62dad9e4823783a471c17c58831/lightly/api/api_workflow_upload_dataset.py#L17-L18 is called without specifying the workers when calling it from
https://github.com/lightly-ai/lightly/blob/fc50052f9b2dc62dad9e4823783a471c17c58831/lightly/cli/upload_cli.py#L51-L56

Resolving patch:
diff --git a/lightly/api/api_workflow_upload_dataset.py b/lightly/api/api_workflow_upload_dataset.py
--- a/lightly/api/api_workflow_upload_dataset.py
+++ b/lightly/api/api_workflow_upload_dataset.py
@@ -61,6 +61,7 @@ def upload_dataset(self, input: Union[str, LightlyDataset], max_workers: int = 8
 
         # handle the case where len(dataset) < max_workers
         max_workers = min(len(dataset), max_workers)
+        max_workers = max(max_workers, 1)
 
         # upload the samples
         if verbose:
diff --git a/lightly/cli/upload_cli.py b/lightly/cli/upload_cli.py
--- a/lightly/cli/upload_cli.py
+++ b/lightly/cli/upload_cli.py
@@ -52,7 +52,7 @@ def _upload_cli(cfg, is_cli_call=True):
         mode = cfg['upload']
         dataset = LightlyDataset(input_dir=input_dir, transform=transform)
         api_workflow_client.upload_dataset(
-            input=dataset, mode=mode
+            input=dataset, mode=mode, max_workers=cfg['loader']['num_workers']
         )
 
     if path_to_embeddings:


Test patch:
diff --git a/tests/UNMOCKED_end2end_tests/test_api.py b/tests/UNMOCKED_end2end_tests/test_api.py
--- a/tests/UNMOCKED_end2end_tests/test_api.py
+++ b/tests/UNMOCKED_end2end_tests/test_api.py
@@ -4,6 +4,9 @@
 from typing import List, Tuple
 
 import numpy as np
+from hydra.experimental import initialize, compose
+
+from lightly.cli import upload_cli
 from lightly.data.dataset import LightlyDataset
 
 from lightly.active_learning.scorers.classification import ScorerClassification
@@ -69,7 +72,14 @@ def create_new_dataset_with_embeddings(path_to_dataset: str,
     api_workflow_client.create_new_dataset_with_unique_name(dataset_basename=dataset_name)
 
     # upload to the dataset
-    api_workflow_client.upload_dataset(input=path_to_dataset)
+    initialize(config_path="../../lightly/cli/config", job_name="test_app")
+    cfg = compose(config_name="config", overrides=[
+        f"input_dir='{path_to_dataset}'",
+        f"token='{token}'",
+        f"dataset_id={api_workflow_client.dataset_id}",
+        f"loader.num_workers=9"
+        ])
+    upload_cli(cfg)
 
     # calculate and save the embeddings
     path_to_embeddings_csv = f"{path_to_dataset}/embeddings.csv"