Incompatibility single-modality AutoProcessor and PEFT Adapter
Author: tomaarsen
Pull request: no

Hello!

## Bug report overview
* I've encountered an incompatibility between `AutoProcessor` and PEFT Adapters

## Details
Straight to the point:
```python
from transformers import AutoProcessor, AutoModel
from peft import LoraConfig, TaskType

model_name = "google-bert/bert-base-uncased"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoProcessor.from_pretrained(model_name)

peft_config = LoraConfig(
    task_type=TaskType.FEATURE_EXTRACTION
)
model.add_adapter(peft_config)

save_path = "peft_processor_test_path"
model.save_pretrained(save_path)
tokenizer.save_pretrained(save_path)

loaded_model = AutoModel.from_pretrained(save_path)
loaded_tokenizer = AutoProcessor.from_pretrained(save_path)
print(loaded_model)
print(loaded_tokenizer)
```
Initially, I run into #4273, which is being resolved in #41604. When I use that branch, an issue appears:

```
Traceback (most recent call last):
  File "c:\code\transformers\demo_test_peft_processor.py", line 21, in <module>
    loaded_tokenizer = AutoProcessor.from_pretrained(save_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\code\transformers\src\transformers\models\auto\processing_auto.py", line 360, in from_pretrained
    config = AutoConfig.from_pretrained(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\code\transformers\src\transformers\models\auto\configuration_auto.py", line 1383, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in peft_processor_test_path. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: aimv2, aimv2_vision_model, albert, align, altclip, apertus, arcee, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, bitnet, blenderbot, blenderbot-small, blip, blip-2, blip_2_qformer, bloom, blt, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, cohere2_vision, colpali, colqwen2, conditional_detr, convbert, convnext, convnextv2, cpmant, csm, ctrl, cvt, cwm, d_fine, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deepseek_v2, deepseek_v3, deepseek_vl, deepseek_vl_hybrid, deformable_detr, deit, depth_anything, depth_pro, deta, detr, dia, diffllama, dinat, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, distilbert, doge, donut-swin, dots1, dpr, dpt, edgetam, edgetam_video, edgetam_vision_model, efficientformer, efficientloftr, efficientnet, electra, emu3, encodec, encoder-decoder, eomt, ernie, ernie4_5, ernie4_5_moe, ernie_m, esm, evolla, exaone4, falcon, falcon_h1, falcon_mamba, fastspeech2_conformer, fastspeech2_conformer_with_hifigan, flaubert, flava, flex_olmo, florence2, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_audio, gemma3n_text, gemma3n_vision, git, glm, glm4, glm4_moe, glm4v, glm4v_moe, glm4v_moe_text, glm4v_text, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gpt_oss, gptj, gptsan-japanese, granite, granite_speech, granitemoe, granitemoehybrid, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hgnet_v2, hiera, hubert, hunyuan_v1_dense, hunyuan_v1_moe, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, internvl, internvl_vision, jamba, janus, jetmoe, jukebox, kosmos-2, kosmos-2.5, kyutai_speech_to_text, layoutlm, layoutlmv2, layoutlmv3, led, levit, lfm2, lfm2_moe, lfm2_vl, lightglue, lilt, llama, llama4, llama4_text, llava, llava_next, llava_next_video, llava_onevision, longcat_flash, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, metaclip_2, mgp-str, mimi, minimax, ministral, mistral, mistral3, mixtral, mlcd, mllama, mm-grounding-dino, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, modernbert-decoder, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmo3, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovis2, owlv2, owlvit, paligemma, parakeet_ctc, parakeet_encoder, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, perception_lm, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_omni, qwen2_5_vl, qwen2_5_vl_text, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen2_vl_text, qwen3, qwen3_moe, qwen3_next, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe, qwen3_vl_moe_text, qwen3_vl_text, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam2, sam2_hiera_det_model, sam2_video, sam2_vision_model, sam_hq, sam_hq_vision_model, sam_vision_model, seamless_m4t, seamless_m4t_v2, seed_oss, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip2_vision_model, siglip_vision_model, smollm3, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, t5gemma, table-transformer, tapas, textnet, time_series_transformer, timesfm, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, vaultgemma, video_llama_3, video_llama_3_vision, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, vjepa2, voxtral, voxtral_encoder, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xcodec, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xlstm, xmod, yolos, yoso, zamba, zamba2, zoedepth
```

In short:
1. `AutoProcessor` loads a tokenizer as the model is single-modal.
2. A PEFT adapter is added.
3. The model and tokenizer are saved: 
    3a. due to the PEFT adapter we only save `adapter_config.json` and `adapter_model.safetensors`, and 
    3b. due to the tokenizer being single-modal we only save `tokenizer_config.json` etc.
4. When re-loading, the AutoProcessor 1) can't find a `processor_config.json` as it wasn't saved and 2) can't find a `config.json` as it wasn't saved.

This is bottlenecking my ability to use `AutoProcessor` as a catch-all processor initialization. Note that if the model is multimodal, then this issue does not happen, as then a `processor_config.json` is saved. 

Perhaps a solution is to:
1. Wrap [this `config` initialization](https://github.com/huggingface/transformers/blob/bb6028cb7938e33ec9e75a0b47b27a6b75584151/src/transformers/models/auto/processing_auto.py#L345-L349) with a try-except 
2. Use checks that `config` is not None [here](https://github.com/huggingface/transformers/blob/bb6028cb7938e33ec9e75a0b47b27a6b75584151/src/transformers/models/auto/processing_auto.py#L351-L354) and [here](https://github.com/huggingface/transformers/blob/bb6028cb7938e33ec9e75a0b47b27a6b75584151/src/transformers/models/auto/processing_auto.py#L360-L385)
3. so that we can reach the fallback section [here](https://github.com/huggingface/transformers/blob/bb6028cb7938e33ec9e75a0b47b27a6b75584151/src/transformers/models/auto/processing_auto.py#L387-L406)

But perhaps there's room for a more structural/permanent fix rather than more try-excepts to get to a fallback.

cc @BenjaminBossan for PEFT

- Tom Aarsen

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.