Repository: cvat-ai/cvat
Pull request: [GSoC2024] feature: allow downloading video frames in custom extension
<!-- Raise an issue to propose your change (https://github.com/opencv/cvat/issues).
It helps to avoid duplication of efforts from multiple independent contributors.
Discuss your ideas with maintainers to be sure that changes will be approved and merged.
Read the [Contribution guide](https://opencv.github.io/cvat/docs/contributing/). -->

<!-- Provide a general summary of your changes in the Title above -->

### Motivation and context
<!-- Why is this change required? What problem does it solve? If it fixes an open
issue, please link to the issue here. Describe your changes in detail, add
screenshots. -->

Allows to download annotated video frames with custom extensions. Previously, it was only available in .png format. Resolves #3867 

### Checklist
<!-- Go over all the following points, and put an `x` in all the boxes that apply.
If an item isn't applicable for some reason, then ~~explicitly strikethrough~~ the whole
line. If you don't do that, GitHub will show incorrect progress for the pull request.
If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [x] I submit my changes into the `develop` branch
- [ ] I have created a changelog fragment <!-- see top comment in CHANGELOG.md -->
- [ ] I have updated the documentation accordingly
- [ ] I have added tests to cover my changes
- [x] I have linked related issues (see [GitHub docs](
  https://help.github.com/en/github/managing-your-work-on-github/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword))
- [x] I have increased versions of npm packages if it is necessary
  ([cvat-canvas](https://github.com/opencv/cvat/tree/develop/cvat-canvas#versioning),
  [cvat-core](https://github.com/opencv/cvat/tree/develop/cvat-core#versioning),
  [cvat-data](https://github.com/opencv/cvat/tree/develop/cvat-data#versioning) and
  [cvat-ui](https://github.com/opencv/cvat/tree/develop/cvat-ui#versioning))

### License

- [x] I submit _my code changes_ under the same [MIT License](
  https://github.com/opencv/cvat/blob/develop/LICENSE) that covers the project.
  Feel free to contact the maintainers if that's a concern.


Problem statement:
Feature Request - Allow to download annotated video frames as JPG images (instead of PNG)
Hello everyone, I have a question regarding the "export dataset" feature in CVAT. When I use it and select the "save images" option, then all video frames will be downloaded as PNG images. Is there a way to tell CVAT that I would rather download all those images as JPG files?

If that is not an option currently, is there any chance you could add such an option in a future release?
I understand that you might want to provide lossless images for video frames, but that might not be necessary in all cases.

Thanks for your consideration.

Hints from issue discussion:
@haimat , thanks for the report. I agree that it is an important feature which should be implemented.
You can export in any desired dataset format and convert images using [Datumaro](https://openvinotoolkit.github.io/datumaro/docs/user-manual/command-reference/convert/):

```
pip install datumaro[default]
datum convert -i dataset/path/ -if <input_format> -o output/path/ -f <input_format> -- --save-images --image-ext=jpg
```

Maybe, we could extend the CLI script with an option to save images in some other format?
> You can export in any desired dataset format and convert images using [Datumaro](https://openvinotoolkit.github.io/datumaro/docs/user-manual/command-reference/convert/):

I know that I can convert the images after downloading them. But the thing is their size - downloading all frames from a video as PNGs often leads to a 10gb+ zip archive. However, after converting them to JPGs they have about 1gb or even less as zip archive. And in my case I don't need PNGs, as we work with JPGs anyway ...
@nmanovic Any update on this? If we would like to implement that on our own in CVAT, where would we want to look for that in the CVAT sources?
Hi, no, no updates yet. Please check this code fragment: https://github.com/opencv/cvat/blob/develop/cvat-sdk/cvat_sdk/core/proxies/tasks.py#L138-L184
Hi any update on this? My team too would also love if this were to be implemented as it would save us a lot of time and resource in post-processing
Hi, I think it could be supported fully when https://github.com/opencv/cvat/issues/1804 is addressed. However, at the moment we can easily support frame downloading with custom extension in the [CLI/SDK "download frames"](https://github.com/opencv/cvat/blob/acb68a54d75456338802e2774ef0ffd388bad761/cvat-sdk/cvat_sdk/core/proxies/tasks.py#L219) functionality.
@nmanovic I am interested in solving this issue. Please assign it to me.
Plan to fix: 

- pass an additonal argument to the [CLI/SDK "download frames"](https://github.com/opencv/cvat/blob/acb68a54d75456338802e2774ef0ffd388bad761/cvat-sdk/cvat_sdk/core/proxies/tasks.py#L219) function in the form of `image_extension`. Then, instead of inferring the extension from the MIME type, we can directly use the provided extension and download the file as such

```
    def download_frames(
        self,
        frame_ids: Sequence[int],
        image_extension: str,
        *,
        outdir: str = ".",
        quality: str = "original",
        filename_pattern: str = "frame_{frame_id:06d}{frame_ext}",
    ) -> Optional[List[Image.Image]]:
        """
        Download the requested frame numbers for a job and save images as outdir/filename_pattern
        """

        outdir = Path(outdir)
        outdir.mkdir(parents=True, exist_ok=True)

        for frame_id in frame_ids:
            frame_bytes = self.get_frame(frame_id, quality=quality)

            im = Image.open(frame_bytes)
            im_ext = f".{image_extension.strip('.')}"

            outfile = filename_pattern.format(frame_id=frame_id, frame_ext=im_ext)
            im.save(outdir / outfile)
```
To add an extra typesafe, we can only allow a limited number of extensions. Maybe add a dropdown menu in the UI to select the image extension ?

Commit message: feature: allow downloading video frames in custom extension

Patch to review:
diff --git a/cvat-sdk/cvat_sdk/core/proxies/tasks.py b/cvat-sdk/cvat_sdk/core/proxies/tasks.py
index 0bebcc6507d4..7384beba65bb 100644
--- a/cvat-sdk/cvat_sdk/core/proxies/tasks.py
+++ b/cvat-sdk/cvat_sdk/core/proxies/tasks.py
@@ -219,31 +219,24 @@ def download_chunk(
     def download_frames(
         self,
         frame_ids: Sequence[int],
+        image_extension: str,
         *,
-        outdir: StrPath = ".",
+        outdir: str = ".",
         quality: str = "original",
         filename_pattern: str = "frame_{frame_id:06d}{frame_ext}",
     ) -> Optional[List[Image.Image]]:
         """
-        Download the requested frame numbers for a task and save images as outdir/filename_pattern
+        Download the requested frame numbers for a job and save images as outdir/filename_pattern
         """
-        # TODO: add arg descriptions in schema
 
         outdir = Path(outdir)
-        outdir.mkdir(exist_ok=True)
+        outdir.mkdir(parents=True, exist_ok=True)
 
         for frame_id in frame_ids:
             frame_bytes = self.get_frame(frame_id, quality=quality)
 
             im = Image.open(frame_bytes)
-            mime_type = im.get_format_mimetype() or "image/jpg"
-            im_ext = mimetypes.guess_extension(mime_type)
-
-            # FIXME It is better to use meta information from the server
-            # to determine the extension
-            # replace '.jpe' or '.jpeg' with a more used '.jpg'
-            if im_ext in (".jpe", ".jpeg", None):
-                im_ext = ".jpg"
+            im_ext = f".{image_extension.strip('.')}"
 
             outfile = filename_pattern.format(frame_id=frame_id, frame_ext=im_ext)
             im.save(outdir / outfile)


Code review comments:
diff_hunk: @@ -219,31 +219,24 @@ def download_chunk(
     def download_frames(
         self,
         frame_ids: Sequence[int],
+        image_extension: str,
         *,
-        outdir: StrPath = ".",
+        outdir: str = ".",
         quality: str = "original",
         filename_pattern: str = "frame_{frame_id:06d}{frame_ext}",
     ) -> Optional[List[Image.Image]]:
         """
-        Download the requested frame numbers for a task and save images as outdir/filename_pattern
+        Download the requested frame numbers for a job and save images as outdir/filename_pattern
         """
-        # TODO: add arg descriptions in schema
 
         outdir = Path(outdir)
-        outdir.mkdir(exist_ok=True)
+        outdir.mkdir(parents=True, exist_ok=True)
 
         for frame_id in frame_ids:
             frame_bytes = self.get_frame(frame_id, quality=quality)
 
             im = Image.open(frame_bytes)
-            mime_type = im.get_format_mimetype() or "image/jpg"
-            im_ext = mimetypes.guess_extension(mime_type)
-
-            # FIXME It is better to use meta information from the server
-            # to determine the extension
-            # replace '.jpe' or '.jpeg' with a more used '.jpg'
-            if im_ext in (".jpe", ".jpeg", None):
-                im_ext = ".jpg"
+            im_ext = f".{image_extension.strip('.')}"
path: cvat-sdk/cvat_sdk/core/proxies/tasks.py
text: @user1:
Why the default extension logic has been removed? Changing extension should be optional.

@author:
 I thought, instead of inferring the extension from the MIME type, we can directly use the provided extension

@user1:
Yes, but only if it's provided. Tasks can include images with different extensions, so you need to get this information from the file by default to get the expected results after downloading.

@author:
Oh okay. 

> Changing extension should be optional

I get this point now. I will make changes

@author:
Yeah. So you mean that if no extension is provided then I should infer it from the MIME type, else go ahead with the provided extension ?

@user1:
Yes, everything should be as it is now if no specific extension is requested.

@author:
done

diff_hunk: @@ -219,31 +219,24 @@ def download_chunk(
     def download_frames(
         self,
         frame_ids: Sequence[int],
+        image_extension: str,
         *,
-        outdir: StrPath = ".",
+        outdir: str = ".",
         quality: str = "original",
         filename_pattern: str = "frame_{frame_id:06d}{frame_ext}",
     ) -> Optional[List[Image.Image]]:
         """
-        Download the requested frame numbers for a task and save images as outdir/filename_pattern
+        Download the requested frame numbers for a job and save images as outdir/filename_pattern
path: cvat-sdk/cvat_sdk/core/proxies/tasks.py
text: @user1:
Why this has been changed? There should be another similar function in `proxies/job.py`.

@author:
I will fix it

diff_hunk: @@ -219,31 +219,24 @@ def download_chunk(
     def download_frames(
         self,
         frame_ids: Sequence[int],
+        image_extension: str,
         *,
-        outdir: StrPath = ".",
+        outdir: str = ".",
path: cvat-sdk/cvat_sdk/core/proxies/tasks.py
text: @user1:
Why this has been changed?

@author:
I will fix it