Add JinaBERT model
Author: joelpaulkoch
Pull request: yes
Labels: name: New model

name: run-slow

name: Text

# What does this PR do?

This PR adds `JinaBERT` to transformers.
This enables running [`jinaai/jina-embeddings-v2-base-code`](https://huggingface.co/jinaai/jina-embeddings-v2-base-code) without `trust_remote_code`.

Relevant issue and discussion is here: #27035.

Note that there are two implementations in use for Jina Embeddings v2:
This PR covers [jinaai/jina-embeddings-v2-base-code](https://huggingface.co/jinaai/jina-embeddings-v2-base-code) which uses [this implementation](https://huggingface.co/jinaai/jina-bert-v2-qk-post-norm/tree/main).
Additionally, there is [jinaai/jina-embeddings-v2-base-en](https://huggingface.co/jinaai/jina-embeddings-v2-base-en) (and variants `small-en`, `base-zh`, `base-de`, `base-es`) which use [a different implementation](https://huggingface.co/jinaai/jina-bert-implementation).

I'm not sure if we can make a single `JinaBERT` implementation that works for both `base-code` and `base-en`.
Otherwise, we probably would want to have two `JinaBERT` implementations, for instance `JinaBERT` for `base-en` and variants and `JinaBERTv2` for `base-code`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

## Tests
Quite some of the generated tests fail and I'd need help there to judge what we need or what must be updated.

I've updated the `test_inference_no_head_absolute_embedding` integration test to assert on the output that I get from the original implementation.
Moreover, I added an `test_encode` integration test to assert that we get the same results as in [the example provided by Jina AI](https://huggingface.co/jinaai/jina-embeddings-v2-base-code#usage).

## Who can review?

@ArthurZucker

@bwanglzu might be interested too

Comments:
I followed [this guide](https://huggingface.co/docs/transformers/add_new_model) and first ran `transformers-cli add-new-model-like`, then I've created `modular_jina_bert.py` to structure the code according to the modular transformers concept.

I didn't delete any of the files generated in the first step, so I think there is a lot to clean up.

Also, `modular_jina_bert.py` is mostly copy-paste from the [original implemenation](https://huggingface.co/jinaai/jina-bert-v2-qk-post-norm/blob/main/modeling_bert.py).
I only included functions that changed in comparison to BERT.
There are probably things that could be improved or removed.

Moreover, I left some TODOs in `modular_jina_bert.py` for open points.

I've updated the PR where I was somewhat certain. I would still need help, especially regarding the tests and other checks.

One thing I've noticed is that quite some tests fail with `AttributeError: Not needed for JinaBert` which is a result of what I did here (following [the modular transformers guide](https://huggingface.co/docs/transformers/en/modular_transformers#removing-attributes-and-functions) as JinaBertLMPredictionHead [does not define _tie_weights](https://huggingface.co/jinaai/jina-bert-v2-qk-post-norm/blob/main/modeling_bert.py#L854) but [BertLMPredictionHead does](https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L783)):

```python
class JinaBertLMPredictionHead(BertLMPredictionHead):
    def _tie_weights(self):
        raise AttributeError("Not needed for JinaBert")
```

Thanks for your review! I took the docs from the original repository but still applied your suggestions.
I will add a snippet in the docstrings

hey sorry for the delay! Having a look in a bit!

Hey, thanks for your response. Sorry for the even longer delay. It's not realistic for me to continue to work on this PR, and I think there are new alternatives nowadays. So I'm going to close the PR. Thank you nevertheless!