OneFormer processor does not return correctly formatted class_labels tensors
Author: rbavery
Pull request: no

### System Info

- `transformers` version: 4.29.0.dev0
- Platform: Linux-5.19.0-41-generic-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.0+cu117 (True)
- Tensorflow version (GPU?): 2.11.1 (False)
- Flax version (CPU?/GPU?/TPU?): 0.5.3 (cpu)
- Jax version: 0.3.6
- JaxLib version: 0.3.5
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts

I'm trying to finetune `shi-labs/oneformer_ade20k_swin_tiny` on my own dataset. I've hit two problems, one with the docs and one with the actual library code for image processing.

1. There are no docs on fine-tuning or training the OneFormer model on this page: https://huggingface.co/docs/transformers/model_doc/oneformer . So I relied on investigating this test
https://github.com/huggingface/transformers/blob/main/tests/models/oneformer/test_modeling_oneformer.py#L364
2. The train test doesn't actually use the OneFormer Processor that is used in all of the inference examples in  https://huggingface.co/docs/transformers/model_doc/oneformer

I think this is because the OneFormer Processor and the train test produce differently formatted class labels. In the train test, the class_labels are created from scratch here: https://github.com/huggingface/transformers/blob/main/tests/models/oneformer/test_modeling_oneformer.py#L106

When training, it's expected that class_labels is a Tensor shaped like [batch_size, num_classes], where a particular element in a batch would have [1,0,0,0] to represent the 0th class.

But the OneFormer processor returns a list of tensors with values greater than 1:  [tensor([0, 3])]

This eventually leads to an error here https://github.com/huggingface/transformers/blob/v4.28.1/src/transformers/models/oneformer/modeling_oneformer.py#L306 where we get index out of bounds but it's a CUDA assert error.

I think this should be rectified by including a training example in the docs and changing the test and OneFormer Processor so that they work when training the model.

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. adapt test_training to create class_labels with OneFormerProcessor instead of from scratch
2. run the adapted test_training

I have a failing example at https://github.com/developmentseed/slickformer but it takes quite a bit of setup and the datasets in't public yet.

### Expected behavior

I'd expect OneFormer Processor to return a class_labels entry in the dict that has the same expected output as `class_labels` in `test_training`. To summarize, the class_labels need to be one hot encoded for training, but OneFormerProcessor isn't doing this.

Comments:
Hi,

I'd recommend taking a look at the MaskFormer/Mask2Former notebooks regarding fine-tuning on custom data: https://github.com/NielsRogge/Transformers-Tutorials/tree/master/MaskFormer. As the API of OneFormer is identical, except that it has one additional `task_inputs` input which you need to prepare as well.

Hi @NielsRogge @amyeroberts 
Actually I  was following your MaskFormer and Mask2Former tutorials 
and my task was to finetune on Semantic information and have an instance level prediction. Where Mask2Former was performing well.
Still Problems I have mentioned on Closed Issue : https://github.com/huggingface/transformers/issues/21644
My main request was Can you also Write Training Tutorials for OneFormer in MaskFormer, Text input part is creating the problem and the segmentation_maps parameter. 

```
    779 annotation_classes = label["classes"]
    780 annotation_masks = label["masks"]
--> 782 texts = ["a semantic photo"] * self.num_text
    783 classes = []
    784 masks = []

TypeError: can't multiply sequence by non-int of type 'NoneType'

```
```
preprocessor = OneFormerImageProcessor.from_pretrained(config.MODEL_PATH)
preprocessor.num_text = 2
preprocessor.num_classes = 2
preprocessor.ignore_index=3 
preprocessor.do_reduce_labels=False 
preprocessor.do_resize=False
preprocessor.from_json_filedo_rescale=False
preprocessor.do_normalize=True
preprocessor.image_mean = config.MEAN
preprocessor.image_std = config.STD
```
after introducing num_text:
```

 972     num_class_obj[cls_name] = 0
    974 for i, label in enumerate(annotations):
--> 975     task = task_inputs[i]
    976     if task == "semantic":
    977         classes, masks, texts = self.get_semantic_annotations(label, num_class_obj)

IndexError: list index out of range
```

```
def collate_fn(batch):
    inputs = list(zip(*batch))
    images = inputs[0]
    segmentation_maps = inputs[1]
    # this function pads the inputs to the same size,
    # and creates a pixel mask
    # actually padding isn't required here since we are cropping
   
    batch = preprocessor(
        images,
        task_inputs=["semantic"],
        segmentation_maps=segmentation_maps,

         return_tensors="pt",
    )
    
    return batch
```

Any solution? I'm having the same or similar issues when changing the MaskFormerImageProcessor to OneFormerProcessor in the tutorial https://github.com/NielsRogge/Transformers-Tutorials/blob/master/MaskFormer/Fine-tuning/Fine_tuning_MaskFormer_on_a_panoptic_dataset.ipynb

@praeclarumjj3 As you implemented the model, could you confirm the desired structure for the inputs when training the model. Based on [this docstring](https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/oneformer/image_processing_oneformer.py#L948) and [this comment](https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/oneformer/image_processing_oneformer.py#L1049), this indicates to me that only a single image could be passed in at a time when training the model, as the number of labels for each image is different and hence the inputs cannot be batched. Is this correct? 

Hi @amyeroberts, Although the number of labels for each image differs, we can still batch the inputs (`image_inputs` and `tasks_inputs`) to OneFormer because they will have the same dimension. As for the labels, we pass those as a `List[Tensor]` to the loss function anyway, so there is no need to batch those as the `OneFormerProcessor` returns labels as a list already.
https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/oneformer/modeling_oneformer.py#L475
https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/oneformer/modeling_oneformer.py#L530  

The `_pad_images_to_max_in_batch` method takes the list `mask_labels` as input and outputs a batched tensor with each mask_label padded to a fixed shape. So, it's possible to batch inputs to OneFormer.
https://github.com/huggingface/transformers/blob/18ee1fe76295239335bf1528c744fe1cfba21cc8/src/transformers/models/oneformer/modeling_oneformer.py#L411

The inputs to the models would be the same as those to a `MaskFormer model for Panoptic Segmentation` with an addition of a `task_inputs` field. So, if you have a list (of length `N`) of images: `input_images`, their mask form labels: `segmentation_maps`, you need to pass a list (of length 'N') of strings of the form ["panoptic", "semantic", .... {length `N`}] as the `task_inputs` arguments.
https://github.com/huggingface/transformers/blob/04ab5605fbb4ef207b10bf2772d88c53fc242e83/src/transformers/models/oneformer/processing_oneformer.py#L78

So, the input preparation would look like:

```python
# processor is an object of the `OneFormerProcessor` class
encoded_inputs = processor(images=input_images, task_inputs=task_inputs, segmentation_maps=segmentation_maps, return_tensors="pt")
```

Please let me know if you have any other questions or issues.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.