Add Granite 4.1 Vision (granite4_vision)
Author: artem-spector
Pull request: yes
Labels: name: New model

## What does this PR do?

Adds built-in support for **Granite 4.1 Vision** (`granite4_vision`), IBM's multimodal vision-language model for enterprise document understanding.

### Architecture highlights
- **Vision encoder:** SigLIP2 (`google/siglip2-so400m-patch16-384`), tiled 384×384 patches
- **Window Q-Former projector:** 4×4 patch windows compressed to 2×2 query tokens via cross-attention (`downsample_rate="4/8"`)
- **DeepStack feature injection:** 8 vision-to-LLM injection points across two mechanisms:
  - *LayerDeepstack:* features from 4 vision encoder depths injected at 4 LLM layers (reversed order — deepest vision → earliest LLM)
  - *SpatialDeepstack:* deepest features split into 4 spatial offset groups (TL/TR/BL/BR), injected at 4 later LLM layers
- **Language model:** GraniteForCausalLM (3.5B) with a rank-256 LoRA adapter (same-repo, LM-only)

### Files added
| File | Purpose |
|---|---|
| `modular_granite4_vision.py` | Source of truth — inherits from LLaVA-Next, overrides novel components |
| `configuration_granite4_vision.py` | Config (generated) |
| `modeling_granite4_vision.py` | Model (generated) |
| `processing_granite4_vision.py` | Unified processor (generated) |
| `image_processing_granite4_vision.py` | Torchvision-based image processor |
| `image_processing_pil_granite4_vision.py` | PIL/NumPy image processor |
| `tests/models/granite4_vision/` | Modeling, image processing, and processor tests |
| `docs/source/en/model_doc/granite4_vision.md` | Model documentation |

### Auto-registration
- Config: auto-generated via `configuration_granite4_vision.py` model_type
- Modeling: `MODEL_MAPPING_NAMES` + `MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES`
- Processing + image processing: registered in respective auto files

### Tests
- Unit tests pass locally (`pytest tests/models/granite4_vision/ -x -q`)
- `@slow` integration tests load real checkpoint and assert outputs within tolerance
- `make style` and `make check-repo` pass (3 remaining failures are pre-existing upstream issues: `mlinter` version mismatch and `Sam3Lite` incomplete model)

## Before submitting
- [x] This PR is not a duplicate
- [x] I have read the [contributor guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)
- [x] The documentation reflects the changes
- [x] The tests pass

## Related
- vLLM built-in support: https://github.com/vllm-project/vllm/pull/40282 (merged)
- Model: https://huggingface.co/ibm-granite/granite-vision-4.1-4b

Comments:
I've traced the root cause of the check_repository_consistency and tests_torch failures to a specific upstream commit:                                                                                   
                                                                                                                                                                                                                                                 
  [Sam3LiteText] Remove unnecessary modules/configs (#45535) (7439ac0459)
                                                                                                                                                                                                                                                 
  This commit removed Sam3LiteTextViTConfig and Sam3LiteTextVisionConfig from the modeling file but left them referenced in auto_mappings.py, causing:                                                                                           
  - AttributeError: module transformers has no attribute Sam3LiteTextViTConfig (357 test failures)                                                                                                                                               
  - check_repo failure: `Sam3LiteTextVisionConfig` appears in CONFIG_MAPPING_NAMES but is not defined                                                                                                                                            
                                                                                                     
  This is reproducible on main independently of our PR.                                                                                                                                                                                          
                                                                                                                                                                                                                                                 
  Question for reviewers: Should we include a fix for this in our PR (removing the stale entries from auto_mappings.py), or would you prefer to handle it in a separate hotfix? Happy to do either.                                              


Opened a dedicated issue for the upstream regression: #45600

LMK when ready for review, and ig this PR supersedes https://github.com/huggingface/transformers/pull/45350?

@zucchini-nlp - yes, this PR supersedes https://github.com/huggingface/transformers/pull/45350. Its our team that is responsible for producing/release IBM vision models.
This PR is ready for review from my side.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45597). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp I'm ready for the second round :)

Hi @zucchini-nlp, sorry for delay - I was chasing a performance problem, which turn out to be not real.

Summary of changes:
- Granite4VisionImageFeaturesOutput now inherits BaseModelOutputWithPooling instead of bare ModelOutput, which unblocks the common image-features tests                                                                                            
- Added @capture_outputs to Granite4VisionTextModel.forward so output_hidden_states propagates correctly through generation                                                                                                                        
- Removed the manual DynamicCache init from forward — GenerationMixin._prepare_cache_for_generation handles this                                                                                                                                   
- get_image_features now properly passes output_hidden_states through (via kwarg or config fallback)                                                                                                                                               
- qformer_config is now fully specified at __init__ time rather than patched post-super().__init__()                                                                                                                                               
- _can_record_outputs and _deepstack_inject moved up to Granite4VisionPreTrainedModel                                                                                                                                                              
- One-letter variable names renamed for clarity; unused inherited config attrs now raise AttributeError with a clear message

Open question — test_can_init_all_missing_weights:                                                                                                                                                                                                                                                                                                                                                                                                                                                    

This test verifies that every weight can be re-initialized from meta device via _init_weights. It fails because WindowQFormerDownsampler embeds a Blip2QFormerModel (a full PreTrainedModel from a different family). When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized. 
Is there a recommended approach here, or is skipping acceptable for embedded third-party PreTrainedModel submodules?

> When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.

That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass

> > When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.
> 
> That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass

will re-check too

> > > When from_pretrained(None, state_dict={}) walks submodules, our _init_weights has no isinstance branches for QFormer's internal layers (attention, embeddings, etc.), so they stay uninitialized.
> > 
> > 
> > That is weird, usually our initializer walks down each sub-module and initializes its weights. Unless Qformer has special buffers that need to be init, it should be fine. I can check it on Monday and do another review pass
> 
> will re-check too

you right, `test_can_init_all_missing_weights` works

run-slow: granite4_vision

@zucchini-nlp , thanks for the approval! 
All suggestions have been implemented, except one for _init_weights: the Embedding/LayerNorm/RMSNorm branches are necessary, since the parent LlavaNextPreTrainedModel._init_weights doesn't handle these module types.                                                                                                                                                                          
  

@artem-spector not the parent llava-next, it is handled in base `PreTrainedModel` class :)

A couple comments only, I think we had a misunderstanding with prev round. And requested review from a core maintainer (@vasqu) , then we can merge

@zucchini-nlp thanks for clarifying! All three addressed and pushed.

@artem-spector we are releasing 5.8.0 today so if you could address all comments until the evening, we are very glad to add granite-vision in the release

If you need help, or you are afk, just ping me and I can fix the rest. It is all just nits about styling and better modular usage

@zucchini-nlp, thank you! I'm working on it, hope to finish soon. BTW what hour is "until evening"? 

@zucchini-nlp Few questions:

  Re __post_init__ super order:
  The dict→typed conversion for qformer_config must happen before super().__post_init__ because the super call triggers the _attn_implementation setter which walks sub_configs — if qformer_config is still a raw dict at that point it would fail. 
  We could clean this up by removing qformer_config from sub_configs (it doesn't support SDPA anyway so propagating _attn_implementation into it is a no-op) — happy to do that if preferred.                                                       
                                                                                                                                                                                                                                                     
  Re Fraction and torch-compile:
  Fraction is only used during __init__ to parse the config string once — the result is stored as plain int attributes (query_side, window_side) that are used in forward. So it's not in the compiled path.                                         
                                                                                                                                                                                                                                                     
  Re _init_weights super call:                                                                                                                                                                                                                       
  The fix requires touching LlavaNextPreTrainedModel — just want to confirm you're okay with us touching LlavaNext code as part of this PR.                                                                                                          


> Re post_init super order:

Already fixed no? All subconfigs have to be an obj before calling super(), and we can do so by completely overriding `post-init`, i.e. no copy from llava. Copying had weird side-effects


>  you're okay with us touching LlavaNext 

Yes, ofc. I should've also fixed that but feel free if anything is left :)

Just pushed some small fixes for CI but feel to add onto them if it did not catch everything

@artem-spector can you wait a second, the last force push deleted the last commits. Let me push them

Should be good now

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, granite4_vision, llava_next, llava_next_video, llava_onevision, ovis2