[Discuss] Handling images when the number of dimensions is 2
Author: sayakpaul
Pull request: no
Labels: name: Discussion

### System Info

- `transformers` version: 4.26.0
- Platform: Linux-5.10.147+-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- TensorFlow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed

### Who can help?

Ccing @amyeroberts 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This issue thread is rather for discussing how to best handle images when the number of dimensions is two. When this is the case any image processor (subclass of `BaseImageProcessor`) will error out, complaining that this number is invalid (which it should). 

I think there are two simple ways to handle this:

* We can either discard these images. 
* Or we can handle it like so:

```python
from PIL import Image
import numpy as np 

def handle_grayscale_image(image):
    np_image = np.array(image)
    if np_image.ndim == 2:
        tiled_image = np.tile(np.expand_dims(np_image, -1), 3)
        return Image.fromarray(tiled_image)
    else:
        return Image.fromarray(np_image)
```

I was working through the [Semantic Segmentation task guide](https://huggingface.co/docs/transformers/tasks/semantic_segmentation). Depending on the shuffling, there would be cases when one would hit this situation. The task guide uses the [Scene Parse 150 dataset](https://huggingface.co/datasets/scene_parse_150). 

In this case, the grayscale (image dimension is 2) image is not worth throwing out IMO:

![original_image](https://user-images.githubusercontent.com/22957388/217417146-e4b51674-c697-438f-addb-c05cc038540d.png)

Here's the corresponding segmentation map:

![segmentation_map](https://user-images.githubusercontent.com/22957388/217417177-384bd229-773c-4e7d-ac57-f76fa6e725ce.png)

Here are my full data pipeline utilities in case someone runs into the same issue:

```python
from PIL import Image 
import numpy as np 
from datasets import load_dataset
from torchvision.transforms import ColorJitter

def handle_grayscale_image(image):
    np_image = np.array(image)
    if np_image.ndim == 2:
        tiled_image = np.tile(np.expand_dims(np_image, -1), 3)
        return Image.fromarray(tiled_image)
    else:
        return Image.fromarray(np_image)

jitter = ColorJitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.1)

def train_transforms(example_batch):
    images = [jitter(handle_grayscale_image(x)) for x in example_batch["image"]]
    labels = [x for x in example_batch["annotation"]]
    inputs = image_processor(images, labels)
    return inputs


def val_transforms(example_batch):
    images = [handle_grayscale_image(x) for x in example_batch["image"]]
    labels = [x for x in example_batch["annotation"]]
    inputs = image_processor(images, labels)
    return inputs

ds = load_dataset("scene_parse_150", split="train[:150]")

ds = ds.train_test_split(test_size=0.1)
train_ds = ds["train"]
test_ds = ds["test"]

train_ds.set_transform(train_transforms)
test_ds.set_transform(val_transforms)
```

### Expected behavior

Maybe suggest some potential alternatives from the error message that is thrown by the image processor? 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.