WIP: Added basic eos token based pooling 
Author: isamu-isozaki
Pull request: yes

# What does this PR do?
This PR is still a WIP. This is based on [this issue](https://github.com/huggingface/transformers/issues/21029). The main problem is that when new tokens are added to the tokenizer and text model and learned such as with [textual inversion](https://textual-inversion.github.io/) the clip text model pools at the wrong location as the pooling is not done at the new token location and not the eos token id location

Fixes #21029


## Before submitting
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Models:

- text models: @ArthurZucker and @younesbelkada

Comments:
@ArthurZucker Hi! Just moved to this pr. Just some git issues so switched but this is based on the pr [here](https://github.com/huggingface/transformers/pull/21096)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_21205). All of your documentation changes will be reflected on that endpoint.

Attempt fixing the bugs now

From message on old pr: @ArthurZucker Thanks for the comment! Very interesting. The reason I'm resistant to just using self.config.vocab_size-1 is that when adding new tokens for the textual inversion training, usually we increase the resize_token_embedding method. So then when loading the trained embeddings, self.config.vocab_size-1 is not the eos token id anymore.

Do you think I should change the logic for resize_token_embeddings and tokenizer instead so that the eos_token_id is always the max? The disadvantage of this is that it'll be way more code.

Hey, not really, I would say the least changes the better! 

Also for the eos_token ids, you can just set it as an argument in the clip config, and maybe raise some kind of warning if it is not the last? The default should be `config.eos_token_id`

@ArthurZucker Thanks for the comment! Good point. Will do that asap

ok seems like the bugs are coming from indexing I do when calculating the new clip pooling. I'll try fixing that within this week

@isamu-isozaki Thank you for working on this PR. 

@ArthurZucker instead of assuming the eos_token is the last (by id) or relying on the config, isn't it better to look the id up from vocab? 

Something like:
```
 self.bos_token = self.vocab["
"]
 self.eos_token = self.vocab["
"]
 self.pad_token = self.vocab["
"]
 ```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.