Repository: lightly-ai/lightly

Issue:
447 register dataset before upload pw
# 447 register dataset before upload

Closes #447.

- Adds the generated api code to register dataset uploads and uses them.


Discussion:
# [Codecov](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai) Report
> Merging [#451](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai) (f8a6d29) into [master](https://codecov.io/gh/lightly-ai/lightly/commit/d23b639d33f68ce2e986000516770e8f2ee29453?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai) (d23b639) will **decrease** coverage by `0.03%`.
> The diff coverage is `71.42%`.

[![Impacted file tree graph](https://codecov.io/gh/lightly-ai/lightly/pull/451/graphs/tree.svg?width=650&height=150&src=pr&token=1NEAVROK3W&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai)](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai)

```diff
@@            Coverage Diff             @@
##           master     #451      +/-   ##
==========================================
- Coverage   85.46%   85.42%   -0.04%     
==========================================
  Files          78       78              
  Lines        2738     2745       +7     
==========================================
+ Hits         2340     2345       +5     
- Misses        398      400       +2     
```


| [Impacted Files](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai) | Coverage Δ | |
|---|---|---|
| [lightly/api/api\_workflow\_upload\_dataset.py](https://codecov.io/gh/lightly-ai/lightly/pull/451/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai#diff-bGlnaHRseS9hcGkvYXBpX3dvcmtmbG93X3VwbG9hZF9kYXRhc2V0LnB5) | `94.06% <71.42%> (-1.43%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai). Last update [d23b639...f8a6d29](https://codecov.io/gh/lightly-ai/lightly/pull/451?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=lightly-ai).


Resolving patch:
diff --git a/lightly/cli/upload_cli.py b/lightly/cli/upload_cli.py
--- a/lightly/cli/upload_cli.py
+++ b/lightly/cli/upload_cli.py
@@ -92,6 +92,10 @@ def _upload_cli(cfg, is_cli_call=True):
         )
         print("Finished upload of embeddings.")
 
+    if custom_metadata is not None and not input_dir:
+        # upload custom metadata separately
+        api_workflow_client.upload_custom_metadata(custom_metadata, verbose=True)
+
     if new_dataset_name_ok:
         print(f'The dataset_id of the newly created dataset is '
               f'{bcolors.OKBLUE}{api_workflow_client.dataset_id}{bcolors.ENDC}')


Test patch:
diff --git a/tests/cli/test_cli_upload.py b/tests/cli/test_cli_upload.py
--- a/tests/cli/test_cli_upload.py
+++ b/tests/cli/test_cli_upload.py
@@ -1,6 +1,7 @@
 import os
 import re
 import sys
+import json
 import tempfile
 
 import torchvision
@@ -32,6 +33,19 @@ def create_fake_dataset(self, n_data: int=5):
             data = self.dataset[sample_idx]
             path = os.path.join(self.folder_path, sample_names[sample_idx])
             data[0].save(path)
+        
+        coco_json = {}
+        coco_json['images'] = [
+            {'id': i, 'file_name': fname} for i, fname in enumerate(self.sample_names)
+        ]
+        coco_json['metadata'] = [
+            {'id': i, 'image_id': i, 'custom_metadata': 0 } for i, _ in enumerate(self.sample_names)
+        ]
+        
+        self.tfile = tempfile.NamedTemporaryFile(mode="w+")
+        json.dump(coco_json, self.tfile)
+        self.tfile.flush()
+
 
     def parse_cli_string(self, cli_words: str):
         cli_words = cli_words.replace("lightly-upload ", "")
@@ -77,5 +91,15 @@ def test_upload_both_dataset(self):
         with self.assertWarns(UserWarning):
             lightly.cli.upload_cli(self.cfg)
 
+    def test_upload_custom_metadata(self):
+        cli_string = f"lightly-upload token='123' dataset_id='xyz' custom_metadata='{self.tfile.name}'"
+        self.parse_cli_string(cli_string)
+        lightly.cli.upload_cli(self.cfg)
 
-
+    def test_upload_custom_metadata_only(self):
+        cli_string = f"lightly-upload token='123' dataset_id='xyz' custom_metadata='{self.tfile.name}'"
+        self.parse_cli_string(cli_string)
+        input_dir = self.cfg['input_dir']
+        self.cfg['input_dir'] = ''
+        lightly.cli.upload_cli(self.cfg)
+        self.cfg['input_dir'] = input_dir