Repository: huggingface/dataset-viewer

Issue:
Error with RGBA images
https://huggingface.co/datasets/huggan/few-shot-skulls

```
Status code:   500
Exception:     Status500Error
Message:       cannot write mode RGBA as JPEG
```

reported by @NielsRogge 




Discussion:
Also cc @mariosasko: there is no specific code in dataset viewer related to how the images are managed/converted/written on the disk:

- in https://github.com/huggingface/datasets-preview-backend/blob/main/src/datasets_preview_backend/models/column/image.py, we just ensure each image is an instance of `PILImage.Image`
- and then we write it to the disk (to serve the image as a static asset in the dataset viewer page) just by calling the `.save()` method here: https://github.com/huggingface/datasets-preview-backend/blob/1a6eb0c53a558404db98fc110907a49a89405c80/src/datasets_preview_backend/io/asset.py#L49

Do you have any advice on how I should manage this? 
@severo This [line](https://github.com/huggingface/datasets-preview-backend/blob/a77eecca77e2f059ad246aad26fbcb5299ec0a2a/src/datasets_preview_backend/models/column/image.py#L46
) in the viewer is problematic, which attempts to "forcefully" convert every image into JPEG, and that's not supported for some formats/modes (`PNG` + `RGBA`).
ohhh, thanks, I forgot that. You're right, the format is guessed by PIL from the filename extension.

I think we could move to the following strategy: 
- specify the list of desired formats, eg. `['jpg', 'png', 'avif', 'webp']` (see https://developer.mozilla.org/en-US/docs/Web/HTML/Element/img#supported_image_formats)
- for each image cell, try to generate a file in every format
- save the list of successfully generated images (0 to 4)
- replace the `<img>` tag in https://github.com/huggingface/moon-landing/blob/main/server/views/components/DatasetSampleTable/CellImageFile.svelte#L8 with a `<picture>` tag (https://developer.mozilla.org/en-US/docs/Web/HTML/Element/picture) with all the available saved images as the sources, if any.

And possibly we could even generate "responsive" images, ie, for every format: a small one, used in the dataset viewer, and another one with the original size used when the user clicks on the image (the idea was already mentioned here: https://github.com/huggingface/datasets-preview-backend/issues/99)
Also reported here: https://github.com/huggingface/datasets/issues/4160
Also: https://huggingface.co/datasets/wikimedia/wit_base
Also: https://huggingface.co/datasets/cppe-5
Do you think you @mariosasko or @lhoestq could take this issue?
Two related errors:

- split 'train' from dataset 'Chris1/GTA5' with config 'Chris1--GTA5'
```
OSError: cannot write mode P as JPEG
```

- split 'validation' from dataset 'Chris1/cityscapes' with config 'Chris1--cityscapes'
```
OSError: cannot write mode RGBA as JPEG
```

Reported by @apsdehal on slack for WinoGround:

https://huggingface.slack.com/archives/C0311GZ7R6K/p1655776625702879

> Perhaps we can save an image in the first successfully generated format from the list ['jpg', 'png'] instead of generating a list of images + using the picture tag to avoid having to update the frontend code. And we can add optimizations such as picture, cropping, webp, etc., later. WDYT? (edited) 



Resolving patch:
diff --git a/services/worker/src/worker/models/column/image.py b/services/worker/src/worker/models/column/image.py
--- a/services/worker/src/worker/models/column/image.py
+++ b/services/worker/src/worker/models/column/image.py
@@ -41,5 +41,15 @@ def get_cell_value(self, dataset_name: str, config_name: str, split_name: str, r
         if value is None:
             return None
         check_value(value)
-        # this function can raise, we don't catch it
-        return create_image_file(dataset_name, config_name, split_name, row_idx, self.name, "image.jpg", value)
+        # attempt to generate one of the supported formats; if unsuccessful, throw an error
+        for ext in [".jpg", ".png"]:
+            try:
+                return create_image_file(
+                    dataset_name, config_name, split_name, row_idx, self.name, f"image{ext}", value
+                )
+            except OSError:
+                # if wrong format, try the next one, see https://github.com/huggingface/datasets-server/issues/191
+                #  OSError: cannot write mode P as JPEG
+                #  OSError: cannot write mode RGBA as JPEG
+                continue
+        raise ValueError("Image cannot be written as JPEG or PNG")


Test patch:
diff --git a/e2e/tests/test_api.py b/e2e/tests/test_api.py
--- a/e2e/tests/test_api.py
+++ b/e2e/tests/test_api.py
@@ -150,3 +150,27 @@ def test_timestamp_column():
     assert json["columns"][0]["column"]["type"] == "TIMESTAMP"
     assert json["columns"][0]["column"]["unit"] == "s"
     assert json["columns"][0]["column"]["tz"] is None
+
+
+def test_png_image():
+    # this test ensures that an image is saved as PNG if it cannot be saved as PNG
+    # https://github.com/huggingface/datasets-server/issues/191
+    dataset = "wikimedia/wit_base"
+    config = "wikimedia--wit_base"
+    split = "train"
+    response = requests.post(f"{URL}/webhook", json={"update": f"datasets/{dataset}"})
+    assert response.status_code == 200
+
+    response = poll_splits_until_dataset_process_has_finished(dataset, 60)
+    assert response.status_code == 200
+
+    response = poll_rows_until_split_process_has_finished(dataset, config, split, 60)
+    assert response.status_code == 200
+    json = response.json()
+    assert json["columns"][0]["column"]["type"] == "RELATIVE_IMAGE_URL"
+    assert (
+        json["rows"][0]["row"]["image"] == "assets/wikimedia/wit_base/--/wikimedia--wit_base/train/0/image/image.jpg"
+    )
+    assert (
+        json["rows"][20]["row"]["image"] == "assets/wikimedia/wit_base/--/wikimedia--wit_base/train/20/image/image.png"
+    )
diff --git a/services/worker/tests/models/test_split.py b/services/worker/tests/models/test_split.py
--- a/services/worker/tests/models/test_split.py
+++ b/services/worker/tests/models/test_split.py
@@ -158,4 +158,54 @@ def test_timestamp() -> None:
     )
 
 
+def test_image() -> None:
+    # see https://github.com/huggingface/datasets-server/issues/191
+    split = get_split(
+        "wikimedia/wit_base",
+        "wikimedia--wit_base",
+        "train",
+        HF_TOKEN,
+        rows_max_number=21,
+    )
+    assert split["rows_response"]["columns"][0]["column"]["type"] == "RELATIVE_IMAGE_URL"
+    assert (
+        split["rows_response"]["rows"][0]["row"]["image"]
+        == "assets/wikimedia/wit_base/--/wikimedia--wit_base/train/0/image/image.jpg"
+    )
+    assert (
+        split["rows_response"]["rows"][20]["row"]["image"]
+        == "assets/wikimedia/wit_base/--/wikimedia--wit_base/train/20/image/image.png"
+    )
+
+    split = get_split(
+        "Chris1/GTA5",
+        "Chris1--GTA5",
+        "train",
+        HF_TOKEN,
+        rows_max_number=1,
+    )
+    assert split["rows_response"]["columns"][0]["column"]["type"] == "RELATIVE_IMAGE_URL"
+    assert split["rows_response"]["columns"][1]["column"]["type"] == "RELATIVE_IMAGE_URL"
+    assert (
+        split["rows_response"]["rows"][0]["row"]["image"]
+        == "assets/Chris1/GTA5/--/Chris1--GTA5/train/0/image/image.jpg"
+    )
+    assert (
+        split["rows_response"]["rows"][0]["row"]["semantic_segmentation"]
+        == "assets/Chris1/GTA5/--/Chris1--GTA5/train/0/semantic_segmentation/image.png"
+    )
+
+    split = get_split(
+        "huggan/few-shot-skulls",
+        "huggan--few-shot-skulls",
+        "train",
+        HF_TOKEN,
+        rows_max_number=52,
+    )
+    assert (
+        split["rows_response"]["rows"][51]["row"]["image"]
+        == "assets/huggan/few-shot-skulls/--/huggan--few-shot-skulls/train/51/image/image.png"
+    )
+
+
 # TODO: test the truncation