Add `Kosmos-2` model
Author: ydshieh
Pull request: yes

# What does this PR do?

Add `KOSMOS-2` model.

- I decide not to expose `Kosmos2TextModel` and `Kosmos2VisionModel` to the main `__init__` file:
  - as they are really only building blocks. Moreover, loading checkpoints of `Kosmos2ForConditionalGeneration` into those 2 models won't work.
  - loading `Kosmos2ForConditionalGeneration` into `Kosmos2Model` works.
  - (and therefore no corresponding tests for those 2 models)

TODO (follow-up PRs):

- [ ] add a checkpoint conversion script in a follow up PR. (It's there, I just need to clean up the mess code.)
- [ ] upload checkpoint to `microsoft` and change the used checkpoint repo id.


Comments:
_The documentation is not available anymore as the PR was closed or merged._

any updates?

WIP, but a bit slow pace

Hi @ArthurZucker Ready for you to take a review a again :-) 

Would be great if we can merge before Thursday 🙏 (if everything is good) as we want to have an announcement with the original Kosmos-2 authors/team. Thank you in advance!

I will transfer the checkpoint and update the repo id used in this PR - just before merge it.

Hello @ArthurZucker 

I have made all models having flat structures, so we can load models to any another one. Please see [this change](https://github.com/huggingface/transformers/pull/24709/commits/cf501abe1045535425cce3d335ab2db4c103940a#diff-149bd7fa539d3ee4998f0f0335d4f9234081289dcf30a61255dd390e670dce6d)

Unfortunately, it still require an intermediate layer `Kosmos2LMWrapper` being `Kosmos2PretrainedModel`, so we can use `generate` from `Kosmos2ForCausalLM` and `Kosmos2ForConditionalGeneration`. (ugly I know).

As @NielsRogge said [here](https://github.com/huggingface/transformers/pull/24709#discussion_r1342456143), it's probably not worth the effort.

I propose 3 approaches:

- Keep previous version (not expose `Kosmos2TextModel` and `Kosmos2ForCausalLM` in the main `__init__`): **implicitly** discourage using them
- Keep previous version, but rename `Kosmos2TextModel` and `Kosmos2ForCausalLM` to  `Kosmos2DecoderWrapper` and `Kosmos2LMWrapper`: **explicitly** discourage using them
- Keep the current version here (ugly but works in all case)

Let me know your final decision.

Regarding image processor, let's keep it as it is 🙏 

Other comments are addressed.

Hi @ArthurZucker Let me know if the change of `src/transformers/models/kosmos2/modeling_kosmos2.py` in [the last commit](https://github.com/huggingface/transformers/pull/24709/commits/cf501abe1045535425cce3d335ab2db4c103940a#diff-149bd7fa539d3ee4998f0f0335d4f9234081289dcf30a61255dd390e670dce6d) is good or if you want to keep the previous version (see the above comment), if you get some bandwidth today.

(It's not a big change - just about how layers are structured)

Thank you in advance.

Hi @ArthurZucker most comments addressed. Especially no more nested function defined.

> def remove_special_fields(text):
    return re.sub("<.*?>", "", text)

I think one liner sometimes is good to clearly indicate what it does. If you feel strong, I could remove it and just use regex. But regex is not really good for readability.

>  seems like one KEY_MAPPING and using replace could be simpler and not require regex not escaping the dots!

The conversion works well and simple enough. If you don't mind, let's just keep as it is?

> preprocess_text

I renamed it to `preprocess_examples`

Could you take another round of review, and let's meet tomorrow to discuss `_add_remove_spaces_around_tag_tokens`?


@ydshieh great work !!!! thanks again!