TorchFormatter images (H, W, C) instead of (C, H, W) format
State: closed | Author: Modexus (CONTRIBUTOR)

### Describe the bug

Using .set_format("torch") leads to images having shape (H, W, C), the same as in numpy. 
However, pytorch normally uses (C, H, W) format.

Maybe I'm missing something but this makes the format a lot less useful as I then have to permute it anyways.
If not using the format it is possible to directly use torchvision transforms but any non-transformed value will not be a tensor.

Is there a reason for this choice?

### Steps to reproduce the bug

```python
from datasets import Dataset, Features, Audio, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features) 
ds = ds.with_format("torch")
ds[0]["image"].shape
```
```python
torch.Size([512, 512, 4])
```

### Expected behavior

```python
from datasets import Dataset, Features, Audio, Image
images = ["path/to/image.png"] * 10
features = Features({"image": Image()})
ds = Dataset.from_dict({"image": images}, features=features) 
ds = ds.with_format("torch")
ds[0]["image"].shape
```
```python
torch.Size([4, 512, 512])
```

### Environment info

- `datasets` version: 2.14.6
- Platform: Linux-6.5.9-100.fc37.x86_64-x86_64-with-glibc2.31
- Python version: 3.11.6
- Huggingface_hub version: 0.18.0
- PyArrow version: 14.0.1
- Pandas version: 2.1.2

Comments: Here's a PR for that. https://github.com/huggingface/datasets/pull/6402

It's not backward compatible, unfortunately. 

Just ran into this working on data lib that's attempting to achieve common interfaces across hf datasets, webdataset, native torch style datasets. The defacto standards for image tensors are numpy == HWC, torch.Tensor == CHW. 

I had to drop use of 'torch' formatting because as is (H, W, C) makes it incompatible with pretty much all standard torch vision processing (torchvision, etc) including model inputs themselves... not sure what the breakage scope would be, but might be worth considering a breaking change since I'm not aware of many use cases where a torch.Tensor image is expected to be in HWC form. And if I set the format to 'torch', I'd expect to be able to apply torchvision transforms, etc directly to the output...

EDIT: For 'torch' output to be compatible with torch conventions (namely torchvision for images), should follow this https://pytorch.org/vision/0.17/transforms.html#supported-input-types-and-conventions

attn @lhoestq 



We can define something like `.with_format("torch", image_data_format="channels_first")` and recommend using this in the docs maybe ? also cc @NielsRogge 

Sounds good to me. I guess it's not allowed to use the channels first format by default for backwards compatibility purposes?

This works, but am wondering how widespread the use of the function is for image datasets? My hunch would be that it's not used widely enough with image datasets to favour backwards compat (keeping default channels_last) over clumsiness of needing this to be 'correct' for typical use.. but don't have the data to back that up.

I see. I just checked in the HF libraries and it shouldn't break anything. And to be consistent with them we should actually use C H W. For example `transformers` image processors use C H W by default too.

So I'm ok with doing a breaking change to make it consistent with `transformers`, `torchvision`, etc.

Since it is quite connected, the proposed PR #6402 will not work for monochrome `PIL` images since they only have 2 dimensions as `numpy `arrays. [Torchvision ](https://pytorch.org/vision/stable/_modules/torchvision/transforms/functional.html#pil_to_tensor) adds a channel before permuting. Would that make sense here as well?

@Modexus yes, indeed that would make sense as torch expects 1, H, W for monochrome, not H,W as you'd often see in numpy (via PIL), OpenCV, etc.

The reference should be the torchvision fn https://pytorch.org/vision/main/_modules/torchvision/transforms/functional.html#pil_to_tensor

My PR now should handle monochrome PIL image. Thanks for the heads up :)