Processor does not load optional attributes
Author: AmitMY
Pull request: no
Labels: name: bug

name: Vision

### System Info

- `transformers` version: 4.55.0.dev0
- Platform: macOS-15.6.1-arm64-arm-64bit
- Python version: 3.12.2
- Huggingface_hub version: 0.34.3
- Safetensors version: 0.5.3
- Accelerate version: 1.9.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.7.1 (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

maybe @Rocketknight1 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Minimal reproduction:
if `image_processor` is in `attributes` it works, but if it is in `optional_attributes` it fails.

```python
import tempfile

from transformers import AutoImageProcessor, ProcessorMixin


class TextImageProcessor(ProcessorMixin):
    name = "text-image-processor"

    attributes = []
    image_processor_class = "AutoImageProcessor"
    optional_attributes = ["image_processor"]

    def __init__(self,
                 image_processor: AutoImageProcessor = None):
        super().__init__(image_processor=image_processor)

        self.image_processor = image_processor

        # Not used in this processor, but necessary for "from_pretrained" compatibility
        self.chat_template = None
        self.audio_tokenizer = None


with tempfile.TemporaryDirectory() as temp_dir:
    image_processor = AutoImageProcessor.from_pretrained("WinKawaks/vit-tiny-patch16-224")
    processor = TextImageProcessor(image_processor=image_processor)
    processor.save_pretrained(save_directory=temp_dir, push_to_hub=False)
    new_processor = TextImageProcessor.from_pretrained(temp_dir)

    assert new_processor.image_processor is not None
```

### Expected behavior

Both attributes AND optional attributes should be loaded and initialized.
Right now it seems like it would only load optional attributes if they are called `audio_tokenizer`

Comments:
cc @zucchini-nlp maybe? I'm not sure what behaviour we expect here

The `optional_attributes` field isn't meant to support processor attributes that are pre-processing classes. Originally it was added for chat templates and the idea was to use for other `kwargs` in processing config dict. Personally, i believe `audio_tokenizer` also has to be an `attribute`, not an `optional` one. cc @eustlb and @vasqu who added Dia

@AmitMY do you have any specific use case where a preprocessor class might be optional and missing in hub repo?

For reference see https://github.com/huggingface/transformers/pull/38405#discussion_r2158986617

I avoided the attribute class in this instance as the saving/loading logic would've gone into saving the full model. On second thought, it's probably possible to rewrite it to avoid it but that was the fastest way I went with, without much headache 😅 

Tbh, we should let attributes be attributes and it's more likely that the audio tokenizer will be refactored than that we add other optional attributes.

Yes, we might need a better API to attributes with weights, as the current processors aren't designed to load any models. 

Thanks all. 
@zucchini-nlp my use case is as follows right now: I have this processor https://github.com/sign/image-latent-transformer/blob/main/image_latent_transformer/processor.py

And in this repository, the model encodes both bytes (as tokens, with a tokenizer) AND images (as pixels, with an image processor).

I want to maintain a single code base, but allow in training to set up the model without the image embedding model, so it only relies on bytes. That way, we can make a direct comparison between bytes only, image only, and both.

It seems to make sense that what we need is to make the attributes optional, or alternatively, what I could do but really don't want to, is to create a `NoopTokenizer` and a `NoopImageProcessor` that do nothing, to initialize with (but `None` is just so elegant!)

@AmitMY iiuc then you can simply pass `images=None` or `text=None` to indicate that the current input should be testing only one of the inputs. There is no need to set image processor to `None` in that case, and creating a data collator that suits train setting will work fine

I see that it's a custom processor - usually we allow to pass each modality or all modalities and return the respective output(s). In your case, you could make your return also optional, no? (or even monkey patch your calls based on your needs)

thanks for the suggestions. the `return` is not actually the problem, i can decide what to return at will.
the problem comes from the initialization of the processor. it uses a specific `image_processor` based on the model it was initialized with (`CLIPImageProcessor`, for example, or `YolosImageProcessor` etc)

But when I do not have an image model, there should also not be an image processor. It would be a bit funny to save a model that does not use any images, but then the processor also loads an image processor - and which one?

---

All this to say - yes, i can patch and hack my way to get the behavior I want, but it is ugly. 
Same for example, if you look in the `__init__` function, I have to set the following, otherwise `from_pretrained` doesn't work
```py
        # Not used in this processor, but necessary for "from_pretrained" compatibility
        self.chat_template = None
        self.audio_tokenizer = None
```

Hmm, i see. Then it is indeed the only way currently to patch and create a NoOpImageProcessor because `AutoImageProcessor` expects that there is an image processing config available. I still don't think we need to allow optional attributes in processors, as it was not requested until now. Let's see if the issue gets more community interest 

I am wondering about the second part with chat templates, why do we need to force set it to None and what is the error being thrown. Those attributes are not required and thus should be easily skipped if not found in hub repo. 

Thanks @zucchini-nlp . For now I will set up a Noop.

As for the arguments, feel free to try it without the arguments and see the error is (same code as above, just commented out the lines)
```py
import tempfile

from transformers import AutoImageProcessor, ProcessorMixin


class TextImageProcessor(ProcessorMixin):
    name = "text-image-processor"

    attributes = []
    image_processor_class = "AutoImageProcessor"
    optional_attributes = ["image_processor"]

    def __init__(self,
                 image_processor: AutoImageProcessor = None):
        super().__init__(image_processor=image_processor)

        self.image_processor = image_processor

        # # Not used in this processor, but necessary for "from_pretrained" compatibility
        # self.chat_template = None
        # self.audio_tokenizer = None


with tempfile.TemporaryDirectory() as temp_dir:
    image_processor = AutoImageProcessor.from_pretrained("WinKawaks/vit-tiny-patch16-224")
    processor = TextImageProcessor(image_processor=image_processor)
    processor.save_pretrained(save_directory=temp_dir, push_to_hub=False)
    new_processor = TextImageProcessor.from_pretrained(temp_dir)
```

```py
Traceback (most recent call last):
  File "/Users/amitmoryossef/dev/sign/visual-text-decoder/image_latent_transformer/example.py", line 49, in <module>
    processor.save_pretrained(save_directory=temp_dir, push_to_hub=False)
  File "/opt/homebrew/Caskroom/miniconda/base/envs/visual_text_decoder/lib/python3.12/site-packages/transformers/processing_utils.py", line 738, in save_pretrained
    if self.chat_template is not None:
       ^^^^^^^^^^^^^^^^^^
AttributeError: 'TextImageProcessor' object has no attribute 'chat_template'
```

Ah it is because you have `optional_attributes` overwritten and keeping it with default values (or default + you own values) solves the issue

```python
import tempfile

from transformers import AutoImageProcessor, ProcessorMixin


class TextImageProcessor(ProcessorMixin):
    name = "text-image-processor"

    attributes = ["image_processor"]
    image_processor_class = "AutoImageProcessor"
    # optional_attributes = ["chat_template", "audio_tokenizer"]  # Mixin defaults so can be commented out

    def __init__(self,
                 image_processor: AutoImageProcessor = None):
        super().__init__(image_processor=image_processor)

        self.image_processor = image_processor

with tempfile.TemporaryDirectory() as temp_dir:
    image_processor = AutoImageProcessor.from_pretrained("WinKawaks/vit-tiny-patch16-224")
    processor = TextImageProcessor(image_processor=image_processor)
    processor.save_pretrained(save_directory=temp_dir, push_to_hub=False)
    new_processor = TextImageProcessor.from_pretrained(temp_dir)
```

that's also strange... `optional_attributes` does not at all behave how you'd expect

Agreed, the naming is a bit misleading

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

For my use case, I created some `Noop` huggingface classes - https://github.com/sign/image-latent-transformer/blob/main/image_latent_transformer/noop.py

This way, if I don't want an image processor, I create a `NoopImageProcessor` and everything works.

I will think of renaming or entirely deleting the `optional_attributes` field for the coming v5 release, since it is not really doing anything special that can;t be done by other `**kwargs`