Shieldgemma2
Author: RyanMullins
Pull request: yes
Labels: name: New model

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @eustlb
- graph models: @clefourrier

Library:

- flax: @gante and @Rocketknight1
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @Rocketknight1
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

@RyanMullins Submit a PR for this branch as bug fixes.
https://github.com/RyanMullins/transformers/pull/2
- Fix a bug in readme example snippet (`return_tensors="pt"` for `Processor` call)
- Use public image link as example (from https://huggingface.co/google/shieldgemma-2-4b-it)
- Fix typo

There is still an logging issue that outputs incorrect info:

```python3
from PIL import Image
import requests
from transformers import AutoProcessor, ShieldGemma2ForImageClassification

model_id = "google/shieldgemma-2-4b-it"
model = ShieldGemma2ForImageClassification.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(url, stream=True).raw)

custom_policies = {
    "key_a": "descrition_a",
    "key_b": "descrition_b",
}

inputs = processor(
    images=[image],
    custom_policies=custom_policies,
    policies=["dangerous", "key_b"],
    return_tensors="pt",
).to(model.device)

output = model(**inputs)
print(output.probabilities)

```

```
$ python test.py
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:06<00:00,  3.12s/it]
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.50, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
Keyword argument `custom_policies` is not a valid argument for this processor and will be ignored.
Keyword argument `policies` is not a valid argument for this processor and will be ignored.
tensor([[5.3613e-11, 1.0000e+00],
        [3.7379e-09, 1.0000e+00]], device='cuda:1', grad_fn=<ToCopyBackward0>)
```

As you can see above, the argument `custom_policies` and `policies` are already functioning (the output tensor is only 2 classes now), but still get
```
Keyword argument `custom_policies` is not a valid argument for this processor and will be ignored.
Keyword argument `policies` is not a valid argument for this processor and will be ignored.
```

And currently it's confusing about the output tensor.
- What's the meaning of each dimension for the output tensor shape?
- What's the meaning of each number and which harmful category is it mapped to? I assume the second column is not necessary since it seems to be `1 - first_column`

It would be nice if we have something for explanation in the readme / API doc.

@ain-soph thanks for the contribs. For some reason I can't reply to your comments, but I've added docstrings to address [your questions](https://github.com/huggingface/transformers/pull/36678#issuecomment-2730504872) about the shape of the output tensors, and I added an explicit `ShieldGemma2ProcessorKwargs` to address the [logs you saw](https://github.com/huggingface/transformers/pull/36678#issuecomment-2730493230).

When running the default example, an IndexError is raised. Passing in custom_policies fixes that.

#### Example

```python
from PIL import Image
import requests
from transformers import AutoProcessor, ShieldGemma2ForImageClassification

model_id = "google/shieldgemma-2-4b-it"
model = ShieldGemma2ForImageClassification.from_pretrained(model_id, device_map="auto")
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg"
image = Image.open(requests.get(url, stream=True).raw)

inputs = processor(images=[image], return_tensors="pt").to(model.device)

output = model(**inputs)
print(output.probabilities)
```

#### Stack Trace
```python
Loading checkpoint shards: 100%
 2/2 [00:06<00:00,  2.86s/it]
WARNING:accelerate.big_modeling:Some parameters are on the meta device because they were offloaded to the cpu.
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-26-290947fc3325>](https://localhost:8080/#) in <cell line: 0>()
     10 image = Image.open(requests.get(url, stream=True).raw)
     11 
---> 12 inputs = processor(images=[image], return_tensors="pt").to(model.device)
     13 
     14 output = model(**inputs)

1 frames
[/usr/local/lib/python3.11/dist-packages/transformers/processing_utils.py](https://localhost:8080/#) in apply_chat_template(self, conversation, chat_template, **kwargs)
   1326 
   1327         if isinstance(conversation, (list, tuple)) and (
-> 1328             isinstance(conversation[0], (list, tuple)) or hasattr(conversation[0], "content")
   1329         ):
   1330             is_batched = True

IndexError: list index out of range
```

@ghunkins thanks for the bug report! 

The problem was that the files on the Hub were out of date and the processor.json was missing the `policy_definitions` property. I was using updated files locally, which is why I didn't see this until I repro'd your example in Colab. The files are now [updated](https://huggingface.co/google/shieldgemma-2-4b-it/blob/main/processor_config.json) on the Hub. 

@ArthurZucker accidentally [suggested a local solution](https://github.com/huggingface/transformers/pull/36678#discussion_r2003461909) to the problem by providing a default set of policies. So this should be fully fixed now and ready for merging. 

Brilliant, thanks for the amazing work on this @RyanMullins !

Do you need help on my side for the last remaining tests? 

nit: I just tested and there is the same warning for `policy_definitions`.
```Some kwargs in processor config are unused and will not have any effect: policy_definitions.```