Need to get hidden features from Siglip but ValueError: You have to specify input_ids
Author: thotasu
Pull request: no

Hi, I need to get hidden features from Siglip but I am stuck with: ValueError - You have to specify input_ids. Please help.

## this is my code
ckpt = "google/siglip2-base-patch16-224"

model = AutoModel.from_pretrained(ckpt, device_map="cpu",
								output_hidden_states=True).eval()

image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")
inputs = processor(images=[image], return_tensors="pt").to(model.device)

with torch.no_grad():
    model_output = model(**inputs)  #### throws error

## my intention is to access hidden states later
print(model_output.hidden_states)


## This is the error:
'''
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[16], line 3
      1 # run infernece
      2 with torch.no_grad():
----> 3     model_output = model(**inputs)  

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/transformers/models/siglip2/modeling_siglip2.py:1443, in Siglip2Model.forward(self, input_ids, pixel_values, pixel_attention_mask, spatial_shapes, attention_mask, position_ids, return_loss, output_attentions, output_hidden_states, return_dict)
   1432 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
   1434 vision_outputs = self.vision_model(
   1435     pixel_values=pixel_values,
   1436     attention_mask=pixel_attention_mask,
   (...)
   1440     return_dict=return_dict,
   1441 )
-> 1443 text_outputs = self.text_model(
   1444     input_ids=input_ids,
   1445     attention_mask=attention_mask,
   1446     position_ids=position_ids,
   1447     output_attentions=output_attentions,
   1448     output_hidden_states=output_hidden_states,
   1449     return_dict=return_dict,
   1450 )
   1452 image_embeds = vision_outputs[1]
   1453 text_embeds = text_outputs[1]

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/torch/nn/modules/module.py:1532, in Module._wrapped_call_impl(self, *args, **kwargs)
   1530     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1531 else:
-> 1532     return self._call_impl(*args, **kwargs)

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/torch/nn/modules/module.py:1541, in Module._call_impl(self, *args, **kwargs)
   1536 # If we don't have any hooks, we want to skip the rest of the logic in
   1537 # this function, and just call forward.
   1538 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1539         or _global_backward_pre_hooks or _global_backward_hooks
   1540         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1541     return forward_call(*args, **kwargs)
   1543 try:
   1544     result = None

File /Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/transformers/models/siglip2/modeling_siglip2.py:928, in Siglip2TextTransformer.forward(self, input_ids, attention_mask, position_ids, output_attentions, output_hidden_states, return_dict)
    925 return_dict = return_dict if return_dict is not None else self.config.use_return_dict
    927 if input_ids is None:
--> 928     raise ValueError("You have to specify input_ids")
    930 input_shape = input_ids.size()
    931 input_ids = input_ids.view(-1, input_shape[-1])

ValueError: You have to specify input_ids
'''

Comments:
Hey @thotasu, If you need only vision features, you can use `model = SiglipVisionModel.from_pretrained(...)`, this one will not require input_ids

Hi @qubvel, This worked thank you! Curious now: what are input_ids? Can you share any docs?

Hi @thotasu here is the transformers docs for [siglip](https://huggingface.co/docs/transformers/en/model_doc/siglip)
Essentially Siglip is trained getting the similarity between pairs of texts and images. So the full model has a text component and a vision component.
If you pass the inputs ids from the description of the image, the vectors of both components should be similar

Hi @kiansierra Got it! So input ids are for the text descriptions. I just found another research question to pursue. Yay!