Repository: voxel51/fiftyone

Issue:
[BUG] Cannot upload large CVAT annotation tasks: Request Entity Too Large
Hi, 

When uploading to CVAT I am receiving a OSError: [Errno 24] Too many open files. 

The scripts works fine with a couple of test images, but as soon as I move to a large dataset ~16,000 images it will crash with the above error. 

Things I have tried: 
- Tested with small dataset and confirmed working. 
- Tested with 2 x different CVAT servers (inlcluding CVAT.org) 
- Increased the ulimit -a on the ubuntu 18 machine to 1048576
- Attempted to use segments to upload. 

Appreciate any suggestions or help. 

Full trace: 

```
100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16683/16683 [21.9s elapsed, 0s remaining, 972.2 samples/s]      
Name:        Dataset-10
Media type:  image
Num samples: 16683
Persistent:  True
Tags:        []
Sample fields:
    id:           fiftyone.core.fields.ObjectIdField
    filepath:     fiftyone.core.fields.StringField
    tags:         fiftyone.core.fields.ListField(fiftyone.core.fields.StringField)
    metadata:     fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.metadata.Metadata)
    ground_truth: fiftyone.core.fields.EmbeddedDocumentField(fiftyone.core.labels.Detections)
Uploading samples to CVAT...
Computing image metadata...
 100% |████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16683/16683 [20.7s elapsed, 0s remaining, 963.0 samples/s]      
Uncaught exception
Traceback (most recent call last):
  File "/home/inviol/dev/AI_tools/data_management/fiftyone/fiftyone_yolo_dataset_to_CVAT.py", line 53, in <module>
    password="g_nekgrHj33JPCa"
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/core/collections.py", line 6188, in annotate
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/annotations.py", line 212, in annotate
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 2657, in upload_annotations
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3516, in upload_samples
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3942, in _create_task_upload_data
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3374, in upload_data
OSError: [Errno 24] Too many open files: '/home/inviol/inviol/new_model/new_data/Dataset-10-Vulcan-Device/19.10.2021 3969.jpg'
```

Script: 

```py
import fiftyone as fo

name = "Dataset-10"

fo.delete_dataset(name) #If you have a persistent DB you want to delete

dataset_dir = "/home/inviol/inviol/new_model/new_data"

dataset_type = fo.types.YOLOv4Dataset  

dataset = fo.Dataset.from_dir(
    dataset_dir=dataset_dir,
    dataset_type=dataset_type,
    name=name,
)

dataset = dataset=fo.load_dataset(name)
dataset.persistent = True
print(dataset)

view = dataset.view()

anno_key = "dataset_10_vulcan_12"

view.annotate(
    anno_key,
    label_field="ground_truth",
    attributes=False,
    launch_editor=False,
    #url="",
    #username="",
    #segment_size=100,
    #password="",
    #image_quality=100,
    username="",
    password=""
)
print(dataset.get_annotation_info(anno_key))
```


Discussion:
Hi @pullmyleg, the current implementation uploads all media for the run in one batch:

https://github.com/voxel51/fiftyone/blob/935cad90646364c97834fdd935f9be6141d504d1/fiftyone/utils/cvat.py#L3368-L3374

so you will need to make sure your environment is configured to allow enough open files to cover the size of your collection.

I see that you tried to allow that with `ulimit`, but that only applies to the current shell. You'll also need to bump up the limit at the system level or else your process still won't be allowed to open enough files.

Can you double-check that system-level limits are high enough?

Linux users: https://tecadmin.net/increase-open-files-limit-ubuntu/
Mac users: https://superuser.com/a/443168

@brimoor thanks for that! You we're correct, I had set my ulimit incorrectly. Now this is 200,000 and am no longer getting this error. 

Sadly I have run into another error. Request Entity Too Large for url. This may be a CVAT issue, but I couldnt find any help in there docs. Would appreciate any ideas. 

Trace: 

```
`9) 4.jpg'>), 'client_files[16678]': ('016678_out(9) 5.jpg', <_io.BufferedReader name='/home/inviol/inviol/new_model/new_data/Dataset-10-Vulcan-Device/out(9) 5.jpg'>), 'client_files[16679]': ('016679_out(9) 6.jpg', <_io.BufferedReader name='/home/inviol/inviol/new_model/new_data/Dataset-10-Vulcan-Device/out(9) 6.jpg'>), 'client_files[16680]': ('016680_out(9) 7.jpg', <_io.BufferedReader name='/home/inviol/inviol/new_model/new_data/Dataset-10-Vulcan-Device/out(9) 7.jpg'>), 'client_files[16681]': ('016681_out(9) 8.jpg', <_io.BufferedReader name='/home/inviol/inviol/new_model/new_data/Dataset-10-Vulcan-Device/out(9) 8.jpg'>), 'client_files[16682]': ('016682_out(9) 9.jpg', <_io.BufferedReader name='/home/inviol/inviol/new_model/new_data/Dataset-10-Vulcan-Device/out(9) 9.jpg'>)}}
Uncaught exception
Traceback (most recent call last):
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 5399, in _validate
    response.raise_for_status()
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 413 Client Error: Request Entity Too Large for url: http://xx:8080/api/v1/tasks/39/data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/inviol/dev/AI_tools/data_management/fiftyone/fiftyone_yolo_dataset_to_CVAT.py", line 50, in <module>
    password="qgcNT9P9Ksk.G_V",
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/core/collections.py", line 6188, in annotate
    **kwargs,
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/annotations.py", line 212, in annotate
    samples, launch_editor=launch_editor
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 2657, in upload_annotations
    results = api.upload_samples(samples, self)
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3516, in upload_samples
    frame_id_map,
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3942, in _create_task_upload_data
    job_reviewers=_job_reviewers,
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3376, in upload_data
    self.post(self.task_data_url(task_id), data=data, files=files)
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 3046, in post
    self._validate(response, kwargs)
  File "/home/inviol/dev/AI_tools/venv/lib/python3.6/site-packages/fiftyone/utils/cvat.py", line 5412, in _validate
    d["_content"],
Exception: 413 error for request <PreparedRequest [POST]> to url http://xx:8080/api/v1/tasks/39/data with the reason Request Entity Too Large. Error content: b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>413 Request Entity Too Large</title>\n</head><body>\n<h1>Request Entity Too Large</h1>\nThe requested resource does not allow request data with POST requests, or the amount of data provided in\nthe request exceeds the capacity limit.\n</body></html>\n'
(venv) inviol@inviol-waterpistol:~/dev/AI_tools$ `
```
Yeah looks like the implementation of `upload_data()` needs to be updated to upload media for large tasks in batches to satisfy any size limits on the CVAT API side.

@ehofesmann can you help with this?

https://github.com/voxel51/fiftyone/blob/935cad90646364c97834fdd935f9be6141d504d1/fiftyone/utils/cvat.py#L3327

In the meantime, a simple workaround is to split the tasks into multiple smaller runs yourself.

Adding the runs [to a project](https://voxel51.com/docs/fiftyone/integrations/cvat.html#creating-projects) would be a good way to organize the tasks under one roof:

```py
# The dataset or view you want to annotate
dataset = ...

dataset[:1000].annotate("run1", ..., project_name="your-project")
dataset[1000:2000].annotate("run2", ..., project_name="your-project")
...

dataset.load_annotations("run1")
dataset.load_annotations("run2")
...
```

Awesome thanks @brimoor. I will workaround it as suggested in the meantime. Appreciate the help and quick responses. 

Resolving patch:
diff --git a/fiftyone/utils/cvat.py b/fiftyone/utils/cvat.py
--- a/fiftyone/utils/cvat.py
+++ b/fiftyone/utils/cvat.py
@@ -2498,6 +2498,8 @@ class CVATBackendConfig(foua.AnnotationBackendConfig):
         password (None): the CVAT password
         headers (None): an optional dict of headers to add to all CVAT API
             requests
+        task_size (None): an optional maximum number of images to upload per
+            task. Videos are always uploaded one per task
         segment_size (None): maximum number of images per job. Not applicable
             to videos
         image_quality (75): an int in `[0, 100]` determining the image quality
@@ -2536,6 +2538,7 @@ def __init__(
         username=None,
         password=None,
         headers=None,
+        task_size=None,
         segment_size=None,
         image_quality=75,
         use_cache=True,
@@ -2551,6 +2554,7 @@ def __init__(
     ):
         super().__init__(name, label_schema, media_field=media_field, **kwargs)
         self.url = url
+        self.task_size = task_size
         self.segment_size = segment_size
         self.image_quality = image_quality
         self.use_cache = use_cache
@@ -3489,6 +3493,7 @@ def upload_samples(self, samples, backend):
         config = backend.config
         label_schema = config.label_schema
         occluded_attr = config.occluded_attr
+        task_size = config.task_size
         project_name, project_id = self._parse_project_details(
             config.project_name, config.project_id
         )
@@ -3507,7 +3512,7 @@ def upload_samples(self, samples, backend):
         labels_task_map = {}
 
         num_samples = len(samples)
-        batch_size = self._get_batch_size(samples)
+        batch_size = self._get_batch_size(samples, task_size)
 
         (
             cvat_schema,
@@ -3988,7 +3993,9 @@ def _ensure_one_field_per_type(self, label_schema, verbose=True):
                     label_field,
                 )
 
-    def _get_batch_size(self, samples):
+    def _get_batch_size(self, samples, task_size):
+        samples.compute_metadata()
+
         if samples.media_type == fom.VIDEO:
             # The current implementation (both upload and download) requires
             # frame IDs for all frames that might get labels
@@ -3997,10 +4004,13 @@ def _get_batch_size(self, samples):
             # CVAT only allows for one video per task
             return 1
 
-        samples.compute_metadata()
+        num_samples = len(samples)
+
+        if task_size is None:
+            # Put all image samples in one task
+            return num_samples
 
-        # Put all image samples in one task
-        return len(samples)
+        return min(task_size, num_samples)
 
     def _create_task_upload_data(
         self,


Test patch: