Add imageArray
Author: amyeroberts
Pull request: yes

# What does this PR do?

Adds a new class `ImageArray` for use as part of the image processing pipeline. It acts as an array container, which we can use to store information about the image e.g. the data format. This is the recommended way to create 'array-like' numpy objects with persistent attributes: https://numpy.org/doc/stable/user/basics.dispatch.html

The intention is to enable users to explicitly set information about the image e.g. data_format and have that carried through the processing pipeline in a stateful way. This addresses issues where the input image(s) information is repeatedly inferred unnecessarily in functions, or when it's ambiguous e.g. image of shape `(3, 3, 3)`.  See:
* #21981
* #21638
* #22577

Defining `__array_ufunc__` and `__array_function__` means `ImageArray` can have numpy operations e.g. 

```python 
>>> from transformers.image_utils import ImageArray
>>> import numpy as np

>>> x = np.random.randint(0, 256, (2, 2, 3))
>>> img = ImageArray(x)
>>> img
ImageArray([[[ 20 232 120]
  [197 244 147]]

 [[ 47 241  95]
  [ 73 251 140]]], data_format=channels_last, num_channels=3, shape=(2, 2, 3))

# Standard array operations - multiplication, addition etc. are possible
>>> img * 2
ImageArray([[[ 40 464 240]
  [394 488 294]]

 [[ 94 482 190]
  [146 502 280]]], data_format=channels_last, num_channels=3, shape=(2, 2, 3))

>>> img + img
ImageArray([[[ 40 464 240]
  [394 488 294]]

 [[ 94 482 190]
  [146 502 280]]], data_format=channels_last, num_channels=3, shape=(2, 2, 3))

# Numpy functions and array methods can be used
>>> np.mean(img, axis=-1)
ImageArray([[124.         196.        ]
 [127.66666667 154.66666667]], data_format=none, num_channels=0, shape=(2, 2))

>>> img.mean(axis=-1)
ImageArray([[124.         196.        ]
 [127.66666667 154.66666667]], data_format=none, num_channels=0, shape=(2, 2))

# Supports slicing
>>> img[:, :, 1]
ImageObject([[232 244]
 [241 251]], data_format=none, num_channels=0, shape=(2, 2))

# Supports type casting
>>> img.astype(np.float32)
ImageObject([[[ 20. 232. 120.]
  [197. 244. 147.]]

 [[ 47. 241.  95.]
  [ 73. 251. 140.]]], data_format=channels_last, num_channels=3, shape=(2, 2, 3))

# Can be cast back as a numpy array
>>> np.array(img)
array([[[ 20, 232, 120],
        [197, 244, 147]],

       [[ 47, 241,  95],
        [ 73, 251, 140]]])

# Is a numpy array isinstance
>>> isinstance(img, np.ndarray)
True
```

## 🔪 🔪 🔪  Tricky bits 🔪 🔪 🔪 

Although this enables the ImageArray to be used directly in existing numpy logic, it does create issues when interfacing between other frameworks like `torch` or `PIL`. The following operations fail:

```
PIL.Image.fromarray(img)
torch.from_numpy(img)
```

This is because these libraries directly access the underlying memory using python's buffer protocol. As far as I can tell, there is no direct way of exposing this on the Python side, and it would require writing c code to enable. This seems like overkill to me. The only case I know this to cause an issue, is in the pix2struct image processor which uses some [torch specific logic](https://github.com/huggingface/transformers/blob/6eedfa6dd15dc1e22a55ae036f681914e5a0d9a1/src/transformers/models/pix2struct/image_processing_pix2struct.py#L260) (which ideally would be removed). 

As image processor are almost exclusively used with direct calls i.e. `image_processor(img, return_tensors="pt")`, and the torch.tensor batch conversion still works, I don't expect this to cause many issues. 

One way of getting this to work is to return numpy arrays when array methods are called: 
* `np.mean(arr) would return an ImageArray, `image_array.mean(...)` would return a numpy array.

Tbh, I wasn't able to completely figure out the interplay between this functionality as `torch.from_numpy` seems to be just calling C code. 

## Next steps
- [ ] Adapt functionality in `image_transforms` to use the 
- [ ] Add some logic for array operations to remove repeatedly finding e.g. `num_channels` when resulting array is created

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_24577). All of your documentation changes will be reflected on that endpoint.

@ydshieh Adding you as a first reviewer as you're always able to give good SWE advice :) This is quite a big design decision, so will be following up with requests from reviews from others once we've ironed out details here, if that's OK. 

@amyeroberts Thanks for requesting me for the first review. I would like to hear a bit more from you despite not looking through the changes yet 🙏 

- So `ImageObject` will be only used inside the methods of image processor/processing, and the arguments/return values will remain as numpy array?

- From the changes in image processor files, I don't see how this new class helps reducing/simplifying the logic of processing images. Am I missing anything ...?

@ydshieh 

> So ImageObject will be only used inside the methods of image processor/processing, and the arguments/return values will remain as numpy array?

Mostly. In most cases, this won't be seen by users because the image processors are called with a framework specified e.g.  `return_tensors="pt"`

If the user doesn't specify `return_tensors`, then the returned objects would be a list of `ImageObject`. Currently it `return_tensors=None` returns a list of numpy arrays. 

I could make sure to return numpy arrays at the end of processing with something like:

```py
images = [image.numpy() for image in images]
``` 

before passing to `BatchFeature`. 

In the future. once the image transformers have been adapted to use the `ImageObject` attributes, then users will see `ImageObject` returned if they called methods directly, either on the image processor or from the transforms library:

```py
from transformers.image_transforms import resize

# Returned resized_image is an ImageObject object
resized_image = image_processor.resize(image, size={"height": h, "width": w})
resized_image = resize(image, size={"height": h, "width": w})
```

> From the changes in image processor files, I don't see how this new class helps reducing/simplifying the logic of processing images. Am I missing thing ...?

It doesn't yet. This is just introducing the object and replacing the current numpy arrays to ensure everything still works as-is. Part of the next steps is updating logic in e.g. `image_transforms` and in some of the array logic to simplifying things are reduce repeated calculations. 


OK Thank you.

In this PR, `image` argument of `preprocess` remains as numpy array, which is ✅ . The return values should keep as numpy array (if not returning tensor) for which you will update in this PR ✅ .

In the next PR, you will update the file `src/transformers/image_transforms.py` to use `ImageObject` ✅ .

The only thing I am a bit worried is if the input/output of methods in that file will be changed: Those are still considered as public methods (right? We should discuss this with @sgugger  anyway.) and  we should keep them still accepting numpy array input, and return numpy array if it is currently. This might cause the conversion between numpy array <--> ImageObject several times, for which I am not 100% sure if you would love.

However,  this is a question for the next PR, not for this PR.



Hi @amyeroberts 

Could you remind me the reason to remove `ImageObject` and only use `ImageArray`. I just need to refresh my memory, thank you 🙏 

@ydshieh Of course :) Sorry, I should have added some explanatory comments. 

I actually just renamed `ImageObject` to `ImageArray` - the class hasn't been removed.

I did remove casting inputs to `ImageObject` / `ImageArray` in the image processors as it make the PR big and required tackling a few parts of the processing logic which I believe is out of scope. 

@sgugger Yes, I understand. Tbh, I'd rather not have this class. Originally I wanted just a wrapper around the array that could be passed along with the image instead of additional arguments everywhere in the processing methods and functions. Unfortunately, it's necessary to wrap the array like this to have the state persist with numpy array operations and not needing tonnes of extra handling code. 

I'm going to quickly write up an alternative with passing arguments around and compare the two. 








Closing as superseded for #25464, a bit uglier but simpler solution :) 

@ydshieh @rafaelpadilla @sgugger Thank you for your extensive reviews and help on this PR.