Repository: lightly-ai/lightly

Issue:
Error: number of embeddings does not match number of samples
I got an error while using `lightly-magic` on a larger dataset.
My gut feeling tells me that this could be related to the number of workers or the batch size. Let's debug and fix the issue :)

I used the following command:
```
lightly-magic token='TOKEN' dataset_id='DATASET_ID' input_dir=/datasets/data trainer.max_epochs=40 loader.batch_size=256  loader.num_workers=12
```

The dataset consists of `74202` images.

```
Epoch 39: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 289/289 [02:24<00:00,  2.00it/s, loss=4.66, v_num=0]
Best model is stored at: /datasets/lightly_outputs/2021-06-07/07-44-35/lightly_epoch_9.ckpt                                                                            
########## Starting to embed your dataset.
Compute efficiency: 0.03: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 290/290 [00:31<00:00,  9.25it/s]
Embeddings are stored at /datasets/lightly_outputs/2021-06-07/07-44-35/embeddings.csv                                                                                  
########## Starting to upload your dataset to the Lightly platform.
Uploading images (with 12 workers).
  0%|▏                                                                                                                                     | 99/74202 [00:09<1:31:21, 13.52imgs/s]/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly_utils/image_processing/metadata.py:53: RuntimeWarning: divide by zero encountered in double_scalars                  
  return float(np.where(std == 0., 0, mean / std))
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 74202/74202 [3:47:52<00:00,  4.35imgs/s]Finished the upload of the dataset.
Starting upload of embeddings.
Traceback (most recent call last):
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/lightly_cli.py", line 80, in lightly_cli                                                                 
    return _lightly_cli(cfg)
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/lightly_cli.py", line 38, in _lightly_cli                                                                
    _upload_cli(cfg)
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/cli/upload_cli.py", line 76, in _upload_cli                                                                  
    path_to_embeddings_csv=path_to_embeddings, name=name
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/api/api_workflow_upload_embeddings.py", line 71, in upload_embeddings                                        
    path_to_embeddings_csv=path_to_embeddings_csv)
  File "/opt/conda/envs/lightly/lib/python3.7/site-packages/lightly/api/api_workflow_upload_embeddings.py", line 104, in _order_csv_by_filenames                                 
    raise ValueError(f'There are {len(filenames)} rows in the embedding file, but '
ValueError: There are 74202 rows in the embedding file, but 74208 filenames/samples on the server.                                                                               

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 74202/74202 [3:48:50<00:00,  5.40imgs/s]
```


Discussion:
I assume the problem lies in 
https://github.com/lightly-ai/lightly/blob/ffabd63bd340c1cc69def46b3c772dcfc783efa4/lightly/api/api_workflow_upload_dataset.py#L153-L157

If this function is retried, multiple samples are created on the backend side for the same sample in the lightly package.

One general solution to this problem would be the following change:
If an sample_id is requested for a filename, which already exists on the server, not a new sample_id, but the existing sample_id is returned. Then it would not even cause errors if you start an upload, stop it, and start the new upload from the beginning.
A solution only on the pip side would be the following:
- before the upload: get the filenames of all samples already uploaded with getSamplesByDatasetId
- skip all samples having a filename already uploaded

Advantages:
- Solves the problem of stopping and continuing the upload, this is handled
- very little overhead, as only one additional API call
- as usually the dataset is empty, this call should return fast

Disadvantages:
- requires unique filenames
- does not handle errors after creating the sample, but not uploading the thumbname/image

Resolving patch:
diff --git a/lightly/api/api_workflow_upload_dataset.py b/lightly/api/api_workflow_upload_dataset.py
--- a/lightly/api/api_workflow_upload_dataset.py
+++ b/lightly/api/api_workflow_upload_dataset.py
@@ -1,12 +1,12 @@
 import os
 import warnings
 from concurrent.futures.thread import ThreadPoolExecutor
-from typing import Union
+from typing import Union, List
 
 import lightly_utils.image_processing
 import tqdm
 
-from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls
+from lightly.openapi_generated.swagger_client import TagCreator, SamplesApi, SampleWriteUrls, SampleData
 from lightly.openapi_generated.swagger_client.models.sample_create_request import SampleCreateRequest
 from lightly.api.utils import check_filename, PIL_to_bytes
 from lightly.openapi_generated.swagger_client.models.initial_tag_create_request import InitialTagCreateRequest
@@ -73,16 +73,29 @@ def upload_dataset(self, input: Union[str, LightlyDataset], max_workers: int = 8
         if verbose:
             print(f'Uploading images (with {max_workers} workers).', flush=True)
 
-        pbar = tqdm.tqdm(unit='imgs', total=len(dataset))
-        tqdm_lock = tqdm.tqdm.get_lock()
 
         # calculate the files size more efficiently
         lightly_utils.image_processing.metadata._size_in_bytes = lambda img: 0
 
+        # get the filenames of the samples already on the server
+        self.samples_api: SamplesApi
+        samples: List[SampleData] = self.samples_api.get_samples_by_dataset_id(dataset_id=self.dataset_id)
+        filenames = [sample.file_name for sample in samples]
+        if len(filenames) > 0:
+            print(f"Found {len(filenames)} images already on the server, they are skipped during the upload.")
+        filenames_set = set(filenames)
+
+        pbar = tqdm.tqdm(unit='imgs', total=len(dataset)-len(filenames))
+        tqdm_lock = tqdm.tqdm.get_lock()
+
         # define lambda function for concurrent upload
         def lambda_(i):
             # load image
             image, label, filename = dataset[i]
+            if filename in filenames_set:
+                # the sample was already uploaded
+                return True
+
             filepath = dataset.get_filepath_from_filename(filename, image)
             # try to upload image
             try:


Test patch:
diff --git a/tests/api_workflow/mocked_api_workflow_client.py b/tests/api_workflow/mocked_api_workflow_client.py
--- a/tests/api_workflow/mocked_api_workflow_client.py
+++ b/tests/api_workflow/mocked_api_workflow_client.py
@@ -17,7 +17,7 @@
 
 from lightly.openapi_generated.swagger_client import ScoresApi, CreateEntityResponse, SamplesApi, SampleCreateRequest, \
     InitialTagCreateRequest, ApiClient, VersioningApi, QuotaApi, TagArithmeticsRequest, TagBitMaskResponse, \
-    SampleWriteUrls
+    SampleWriteUrls, SampleData
 from lightly.openapi_generated.swagger_client.api.embeddings_api import EmbeddingsApi
 from lightly.openapi_generated.swagger_client.api.jobs_api import JobsApi
 from lightly.openapi_generated.swagger_client.api.mappings_api import MappingsApi
@@ -135,9 +135,21 @@ def get_sample_mappings_by_dataset_id(self, dataset_id, field, **kwargs):
 
 
 class MockedSamplesApi(SamplesApi):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.sample_create_requests: List[SampleCreateRequest] = []
+
+    def get_samples_by_dataset_id(self, dataset_id, **kwargs) -> List[SampleData]:
+        samples = []
+        for i, body in enumerate(self.sample_create_requests):
+            sample = SampleData(id=f'{i}_xyz', dataset_id='dataset_id_xyz', file_name=body.file_name)
+            samples.append(sample)
+        return samples
+
     def create_sample_by_dataset_id(self, body, dataset_id, **kwargs):
         assert isinstance(body, SampleCreateRequest)
         response_ = CreateEntityResponse(id="xyz")
+        self.sample_create_requests.append(body)
         return response_
 
     def get_sample_image_write_url_by_id(self, dataset_id, sample_id, is_thumbnail, **kwargs):
diff --git a/tests/api_workflow/test_api_workflow_upload_dataset.py b/tests/api_workflow/test_api_workflow_upload_dataset.py
--- a/tests/api_workflow/test_api_workflow_upload_dataset.py
+++ b/tests/api_workflow/test_api_workflow_upload_dataset.py
@@ -1,4 +1,6 @@
+import copy
 import os
+import random
 import tempfile
 import pathlib
 
@@ -80,4 +82,31 @@ def create_fake_video_dataset(self, n_videos=5, n_frames_per_video=10, w=32, h=3
 
     def test_upload_video_dataset_from_folder(self):
         self.create_fake_video_dataset()
-        self.api_workflow_client.upload_dataset(input=self.video_input_dir, max_workers=0)
+        self.api_workflow_client.upload_dataset(input=self.folder_path)
+
+    def test_upload_dataset_twice(self):
+        rng = np.random.default_rng(2021)
+
+        base_upload_single_image = self.api_workflow_client._upload_single_image
+
+        # Upload with some uploads failing
+        def failing_upload_sample(*args, **kwargs):
+            if rng.random() < 0.9:
+                return base_upload_single_image(*args, **kwargs)
+            else:
+                raise ValueError()
+        self.api_workflow_client._upload_single_image = failing_upload_sample
+        self.api_workflow_client.upload_dataset(input=self.folder_path)
+
+        # Ensure that not all samples were uploaded
+        samples = self.api_workflow_client.samples_api.get_samples_by_dataset_id(dataset_id="does not matter")
+        self.assertLess(len(samples), len(self.sample_names))
+
+        # Upload without failing uploads
+        self.api_workflow_client._upload_single_image = base_upload_single_image
+        self.api_workflow_client.upload_dataset(input=self.folder_path)
+
+        # Ensure that now all samples were uploaded exactly once
+        samples = self.api_workflow_client.samples_api.get_samples_by_dataset_id(dataset_id="does not matter")
+        self.assertEqual(len(self.sample_names), len(samples))
+