ViLT token_type_embeddings implemented twice
Author: niktheod
Pull request: no

I loaded `ViltModel.from_pretrained("dandelin/vilt-b32-mlm")` and I printed out the parameters of the model. I noticed that token_type_embeddings was included twice.

- Once in the `TextEmbeddings` class: [https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L239](url)

- and once in the `ViltEmbeddings` class:
[https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L98](url)

Additionaly, both of these token_type_embeddings are added to the text embeddings, which doesn't seem to be in agreement with the original paper.

- The first time is added in the `TextEmbeddings` class in this line: `embeddings = inputs_embeds + token_type_embeddings`.
[https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L280](url)

- The second time is added in the `ViltEmbeddings` class in this line: `text_embeds = text_embeds + self.token_type_embeddings(
            torch.zeros_like(attention_mask, dtype=torch.long, device=text_embeds.device)
        )`.
[https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L218C9-L220C10](url)

The image embeddings don't seem to have the same issue, as the `token_type_embeddings` are added to them only once in `ViltEmbeddings`.

Is this an intented behaviour for some reason I don't understand or is it a mistake?

Comments:
Hi,

Yes ViLT uses 2 types of token type embeddings:

* The [first](https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L239) are similar to what BERT uses: this means that 0's are used for the first sequence, and 1's for the second sequence. These are just used to indicate 2 different sequences of text, for instance when doing extractive question answering, in which case the input text would be [CLS] what is his name [SEP] his name is Niels [SEP] for instance. You indicate the question tokens "what is his name" with 0's and the context "his name is Niels" from which to extract the answer with 1's. See also the explainer [here](https://huggingface.co/docs/transformers/glossary#token-type-ids).
* The [second](https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L98) are unique to ViLT, and are used to indicate whether it's a text or image token (the modality of the token). This is because ViLT concatenates the embeddings of text tokens and image patches, so the token type embeddings are added to indicate to the model which modality they are.

 

Hi,

Thank you for your reply. 

Do you know if the [first](https://github.com/huggingface/transformers/blob/13493215abceafc1653af88b045120014fb4c1fc/src/transformers/models/vilt/modeling_vilt.py#L239) are included in the original ViLT implementation or are they an addition of the hugging face implementation? Because in the equations that describe ViLT in the original paper, the first token type embeddings are not mentioned:
![Screenshot 2024-11-17 092843](https://github.com/user-attachments/assets/7dbca0a7-b6aa-43f1-a03d-accdd6df71d1)


It's included in the original implementation [here](https://github.com/dandelin/ViLT/blob/762fd3975c180db6fc88f577cf39549983fa373a/vilt/modules/vilt_module.py#L29), so I assume that they just don't mention it in the paper.

I think the ones you mentioned are the second ones. But the first ones are included in the original implementation too within [self.text_embeddings](https://github.com/dandelin/ViLT/blob/762fd3975c180db6fc88f577cf39549983fa373a/vilt/modules/vilt_module.py#L26). So the point remains the same: both instances of `token_type_embeddings` should be included in the architecture.

Thanks for the clarification.