SigLIP2 loss is incomplete. Add the other loss.
Author: SuperHotDogCat
Pull request: no
Labels: name: Feature request

### Feature request

<img width="1368" height="1212" alt="Image" src="https://github.com/user-attachments/assets/fd6da07d-23ce-42fb-a7d0-0ff646ae918c" />

As far as I know, original SigLIP2's loss contains three parts, Sigmoid loss, LocCa loss, and SILC/TIPS loss.
However, in the Transformers library, the loss for SigLIP2 is currently implemented only as a sigmoid loss, just like the original [SigLIP.](https://github.com/huggingface/transformers/blob/abbed7010be490a24c46aa90dedaa7aa48d78487/src/transformers/models/siglip2/modeling_siglip2.py#L1061)
So, why not implementing all the loss for SigLIP2?

```python
if return_loss:
    # Adapted from https://github.com/google-research/big_vision/blob/01edb81a4716f93a48be43b3a4af14e29cdb3a7f/big_vision/trainers/proj/image_text/siglip2.py#L287
    eye = torch.eye(logits_per_text.size(0), device=logits_per_text.device)
    m1_diag1 = -torch.ones_like(logits_per_text) + 2 * eye
    loglik = torch.nn.functional.logsigmoid(m1_diag1 * logits_per_text)
    nll = -torch.sum(loglik, dim=-1)
    loss = nll.mean()
```

### Motivation

With this limitation, it may not fully support the need for fine-tuning SigLIP2.
Do you have any plans to address this issue in the Transformers library in the future?

### Your contribution

I would like to help with the implementation.

Comments:
cc @qubvel

I’d like to take this up

Hey @SuperHotDogCat, you are right, the loss is not the same as in the original paper. We did not include it because it's a much more difficult framework compared to V1, with the student-teacher and autoregressive decoder (as noted in the picture above). Not sure if Transformers Trainer would support it; if anyone wants to contribute, it would be nice to start with a simple notebook, and then see if some parts can be included into the Transformers library itself.

Hi @SuperHotDogCat and @qubvel,

Following @qubvel's guidance, I've started implementing the complete SigLIP2 loss using the notebook approach as suggested.

**Current Progress (~20% complete):**
 **SILC/TIPS Loss Component**: Successfully implemented the self-distillation framework with:
- EMA teacher architecture for stable knowledge distillation targets
- Masked patch prediction (15% masking ratio following BERT convention)  
- Proper 20% weighting as specified in the SigLIP2 architecture diagram
- GPU-optimized performance (0.7ms overhead per iteration on Tesla T4)
- Full gradient flow support for training integration

**Colab Research Notebook**: [SigLIP2 Complete Loss Implementation](https://colab.research.google.com/drive/1ZhQX4ifLLptYeB0NaE8u5CA6tq-DeShl?usp=sharing)

**Technical Validation:**
- Mathematical verification: Total loss correctly combines sigmoid + weighted SILC/TIPS
- Performance benchmarking shows acceptable computational overhead
- Modular design ready for LocCa integration
- Compatible with existing HuggingFace ecosystem

**Next Phase Plan:**
Working on the LocCa loss implementation (captioning, dense captioning, referring expressions) with autoregressive decoder integration.

@SuperHotDogCat - Thank you for raising this important issue! Your identification of the missing loss components was spot-on. Would you be interested in collaborating on this implementation? Given your background at University of Tokyo, your insights on the autoregressive decoder components would be valuable.

@qubvel - Would appreciate your feedback on the current SILC/TIPS approach before proceeding with LocCa integration. The notebook demonstrates the student-teacher framework you mentioned, and I believe it can be successfully integrated into the Transformers Trainer through custom loss override.

**Background:** I have experience with multimodal AI systems and am committed to completing this implementation to enable proper SigLIP2 research reproducibility for the community.


Hi @Shiv-Expert2503 
Thank you for implementing the SILC and TIPS losses instead of me.
As for LocCa loss, I think there are some important points to be careful about regarding the implementation of captioning loss.
On page 4 of the [LocCa paper](https://proceedings.neurips.cc/paper_files/paper/2024/file/d303b4f1ef8d8274ae6b152df70f5406-Paper-Conference.pdf), they suggested `Parallel prediction` inspired by prior work.
Here is the prior work, [Image Captioners Are Scalable Vision Learners Too](https://arxiv.org/pdf/2306.07915)
In this paper on page 2, Parallel prediction is proposed as shown in the following figure.

<img width="1595" height="477" alt="Image" src="https://github.com/user-attachments/assets/dfdf8eb7-282b-46ad-9993-beabe6ea65fd" />

In other words, part of the training uses the standard causal mask for autoregressive learning, but during parallel prediction, autoregressive token prediction is performed without a mask, with all tokens replaced by [MASK]. I believe we should carefully implement its part.


Thank you so much @SuperHotDogCat for the detailed technical guidance! Your expertise on the LocCa implementation details is exactly what this project needs.

You're right about the parallel prediction complexity - I hadn't fully considered the dual masking strategy from the "Image Captioners Are Scalable Vision Learners" paper. 

**Proposal**: Would you be interested in being a technical collaborator on this implementation? Your research background and paper knowledge would be incredibly valuable for:

1. **LocCa Architecture Design**: Ensuring we implement the parallel prediction correctly
2. **Research Validation**: Cross-referencing our implementation against the academic literature  
3. **Technical Review**: Your Experience would strengthen the contribution, I'm pretty sure :)

**Next Steps**: I'll research the parallel prediction mechanism you mentioned and would love your input on the architectural approach before diving into implementation.

What do you think? This could be a stronger contribution with both our expertise combined.

@qubvel - Would collaborative development be welcomed on this issue?


@Shiv-Expert2503 
I’m willing to help with the architecture.
Regarding the implementation, in the prior work “Image Captioners Are Scalable Vision Learners Too”, the decoder is a standard Transformer decoder with several modifications. Specifically:
The bias terms are removed from the attention layers, MLP layers, and LayerNorms.
The MLP activation function is changed from ReLU to GELU.
The decoder has the same width (d_model), number of attention heads, and MLP dimension as the encoder, but its depth is half that of the encoder (decoder_depth = encoder_depth // 2).

However, this might be implemented differently in LocCa or SigLIP2, so I will check and share the details later. 
I suspect that they might be using Pre-LayerNorm (PreLN), as recent models typically adopt PreLN for better convergence.

<img width="1225" height="500" alt="Image" src="https://github.com/user-attachments/assets/37863ed9-bf79-4b3a-b506-7b5915fac479" />

Also, this is just my personal opinion, but for implementing masking, I think using HuggingFace’s `__prepare_4d_attention_mask` helps reduce implementation errors.
```python
from transformers.modeling_attn_mask_utils import (
    _prepare_4d_attention_mask,
    _prepare_4d_causal_attention_mask,
)
```

Usage is as follows (sorry, this is copied from code I wrote some time ago):

```python
decoder_mask = _prepare_4d_causal_attention_mask(
    attention_mask=text_attention_mask,
    input_shape=(batch_size, tgt_len),
    inputs_embeds=text_embeds,
    past_key_values_length=0,
).to(text_embeds.device)

decoder_mask = decoder_mask.repeat(1, self.config.n_head, 1, 1)
decoder_mask = decoder_mask.reshape(-1, tgt_len, tgt_len)
```
“tgt” comes from the PyTorch documentation for [torch.nn.TransformerDecoderLayer](https://docs.pytorch.org/docs/stable/generated/torch.nn.TransformerDecoderLayer.html#torch.nn.TransformerDecoderLayer.forward)



@SuperHotDogCat , It's great to be collaborating with you on this and Thanks for sharing the recipe and implementation details :)


Here’s my complete plan moving forward:

-  I will start by studying the papers you provided to fully understand the "Parallel Prediction" mechanism. My design approach and key insights will be documented.

- Implementation: I will then build the AR Decoder according to your architectural specifications and implement the LocCa loss.

- Will try perform a smoke test using a small slice of the COCO dataset to validate the end-to-end pipeline and confirm the loss decreases as expected.

All work will be done transparently in my public repository here, which will serve as our workshop:
[Repository](https://github.com/Shiv-Expert2503/SigLIP2-CompleteLoss)

Once the initial implementation is stable and tested, I will ping you for review. You’re welcome to review progress at any time, as I plan to push updates regularly to the [repo](https://github.com/Shiv-Expert2503/SigLIP2-CompleteLoss) .

I'll share it here for your final review as well. I'll be sure to raise any complex architectural questions on this thread as they come up.

**Estimated timeline:**
- Phase 1 (Research & Planning): 2-4 days
 
- Phase 2 (Implementation & Debugging): 5-10 days
 
- Phase 3 (Testing & Validation): 2-3 days


This will provide a complete proof-of-concept for your review, @qubvel  .


cc @yonigozlan as well

@Shiv-Expert2503 
Sorry for late reply.
After my last reply, I carefully reviewed the ARDecoder architecture while also checking the SigLIP2 and LocCa papers. All of them adopt a standard transformer architecture. Therefore, I believe the overall decoder architecture remains the same as in my previous reply (https://github.com/huggingface/transformers/issues/40798#issuecomment-3303980742
). So, you don’t need to change your implementation approach for the ARDecoder architecture. Is there anything else you’re having trouble with regarding the implementation now?

@SuperHotDogCat,

Please don't be Sorry, nothing more problematic for now. I have read through the respective docs you shared - just currently exploring the parallel prediction mechanism in more detail. It will surely be completed by today, and from tomorrow I will be starting the actual implementation.

Thanks for confirming the architecture approach!


Hi, I’m currently conducting research based on SigLIP2. Since the official training setup has not been open-sourced, I’ve been working on reproducing the multi-task losses. So far, I’ve implemented the sigmoid, SILC, and TIPS losses, and confirmed their effectiveness compared to the plain sigmoid loss.

I also wanted to clarify a possible misunderstanding regarding SILC/TIPS: it looks like they have been interpreted as 
> * Proper 20% weighting as specified in the SigLIP2 architecture diagram.

However, the SigLIP2 paper specifies that these losses are only activated during the final 20% of training.

I’d be glad to contribute to efforts on reproducing SigLIP2.





Hi @SuperHotDogCat ,

Just wanted to post a quick progress update.

I'm deep into the implementation of the AR Decoder. The "Parallel Prediction" logic is quite intricate, as you pointed out.

Progress was a bit slower last week due to my university mid-term exams, but I'm now back on track. My updated estimate is to have a testable version in the notebook ready for review within this week.

Thanks for your patience.

Really Sorry:)

Hi @Becomebright, you pointed out the mistake in [the @Shiv-Expert2503 's SILC/TIPS implementation](https://colab.research.google.com/drive/1ZhQX4ifLLptYeB0NaE8u5CA6tq-DeShl?usp=sharing), right? I checked the codebooks and the paper again. The SILC and TIPS losses are added when 80% of training is finished. Thanks for correcting it.

<img width="559" height="548" alt="Image" src="https://github.com/user-attachments/assets/ad5457ef-3f37-42ce-a78c-263883fd5e30" />

@Shiv-Expert2503 
Don't worry. As our instructions, please make sure to add the SILC/TIPS loss only during the final 20% of training, and pay close attention to the weights when summing the Sigmoid loss, SILC/TIPS loss, and LocCa loss. We wish you the best of luck!

Hi @SuperHotDogCat ,
I have completed the entire AR decoder architecture. Now the purely implementation testing (data pipeline + training loop) is remaining! Here is what i did in short:

Please look and tell: [locca_loss](https://colab.research.google.com/drive/1baaV1PaO8qaPt_w5uWutkA9AgJvsSEAx?usp=sharing)

## PHASE 0: Environment & Foundation Setup
- GPU environment configured - CUDA device detection, T4 GPU utilized

- Dataset loaded - Conceptual Captions (5000 samples for smoke test)

- Tokenizer configured (GPT-2) - Added custom [MASK] token for parallel prediction (vocab: 50257→50258)

- Vision encoder frozen - ViT-Base-16 (768-dim, 12 layers) from timm, pretrained on ImageNet-21k, set to eval mode with requires_grad=False

## PHASE 1: AR Decoder Architecture Built
### Core Architecture:
- TransformerDecoder with 6 layers - Half the encoder depth (12÷2=6) as per SigLIP2 spec

- Pre-LayerNorm (Pre-LN) - Uses norm_first=True instead of Post-LN for better training stability

- No bias in linear layers - bias=False in TransformerDecoderLayer for efficiency

- GELU activation - Standard for modern transformers, matches ViT encoder

- Dimension alignment - d_model=768, nhead=12, dim_feedforward=3072 (4×hidden_size)

### Key Design Choices:
- Learnable positional encodings - Uses torch.randn() * 0.02 initialization instead of static zeros or sinusoidal (allows model to learn task-specific position representations)

- Max sequence length = 77 - Follows CLIP standard for vision-language models

- Batch-first tensors - batch_first=True for cleaner tensor handling [B, Seq, Dim]

- Tied vocabulary - LM head projects to full GPT-2 vocab (50258) without bias

### Forward Pass:
- Flexible masking system - Supports tgt_mask="auto" (causal), None (bidirectional), or custom masks

- Device-agnostic - Automatically infers device from input tensors

- Positional encoding slicing - Dynamically matches input sequence length

## PHASE 2: Dual-Mode Training Logic (CapPa Method)

- Parallel prediction path (75%) - All tokens masked with [MASK], NO causal mask (full bidirectional attention) as per CapPa paper

- Autoregressive path (25%) - Standard next-token prediction with causal masking

- Proper input shifting - AR path prepends BOS token and removes last token for teacher forcing alignment

- Padding preservation - Parallel path correctly preserves pad tokens in masked inputs

- Random switching - Uses random.random() < parallel_pred_ratio for stochastic training

### Technical Correctness:
` Causal mask generation - nn.Transformer.generate_square_subsequent_mask() with float32 dtype`

`BOS token fallback - Uses bos_token_id or eos_token_id for GPT-2 compatibility`

`Logits-labels alignment - Correct shape [B, Seq, Vocab] for CrossEntropyLoss`

@SuperHotDogCat I'm running the final smoke test now. I will post an update with the results as soon as it's complete. Thanks!

Hi @Shiv-Expert2503 
Great job on the implementation! I’m looking forward to seeing your results. I took a look at your notebook, and the parallel prediction part seems fine — it matches my own understanding.
In my understanding as well, in the parallel prediction part, all tokens are mask tokens and the model predicts the labels.

I have a few comments. I assume you’re using timm since we’re currently in the notebook phase, . The following comments are things to keep in mind when we eventually integrate this into the Transformers library — you don’t need to worry about them right now.

If you plan to integrate it into Transformers, you’ll need to combine it not with timm, but with a class similar to [Siglip2VisionModel](https://github.com/huggingface/transformers/blob/d0271be18fb7a77ee2e7b984f5e4fa3148ebd9c4/src/transformers/models/siglip2/modeling_siglip2.py#L795) from the Siglip2 repository.
Don’t worry too much — essentially, you just need to feed the VisionTransformer’s output embeddings into the ARDecoder.

The ARDecoder also needs to be Hugging Face–compatible. This means it should inherit from PreTrainedModel (e.g., [example](https://github.com/huggingface/transformers/blob/d0271be18fb7a77ee2e7b984f5e4fa3148ebd9c4/src/transformers/models/siglip2/modeling_siglip2.py#L541)) and use a configuration class that inherits from PretrainedConfig (e.g., [example](https://github.com/huggingface/transformers/blob/d0271be18fb7a77ee2e7b984f5e4fa3148ebd9c4/src/transformers/models/siglip2/configuration_siglip2.py#L121)).
The model’s \_\_init\_\_ should take only the config class (e.g., [example](https://github.com/huggingface/transformers/blob/d0271be18fb7a77ee2e7b984f5e4fa3148ebd9c4/src/transformers/models/siglip2/modeling_siglip2.py#L858)).
Doing so ensures compatibility when you use the model within the Transformers Trainer class.

(I’m not an employee of Hugging Face, so the above comments are based solely on my personal experience.)

Overall, great work!

Hi @qubvel — hope you’re doing well.

Following up on the earlier discussion, I put together a short design note / gap analysis summarizing the current state of SigLIP2 support in Transformers that i and @SuperHotDogCat did, what’s missing (SILC/TIPS, LocCa, CapPa), and how this could be upstreamed incrementally in an HF-compatible way.( I did research on current codebase)

The goal was mainly to make the discussion easier to follow rather than to propose a PR yet.

If this format is useful, I’m happy to hear your wordings or adjust the scope based on your preference.

[REPORT](https://docs.google.com/document/d/1LQRjWb1ErJkxepmsiyIUysHnue0QGkH9DC588e3sGVA/edit?usp=sharing)