Make ViTPooler configurable
Author: sebbaur
Pull request: yes
Labels: name: Vision

# What does this PR do?

Make parameters of ViTPooler (activation function, output size) configurable, so that I can open-source https://arxiv.org/abs/2403.02522 more easily (the encoder slightly differs from the current implementation).

Fixes # (issue)
https://github.com/huggingface/transformers/issues/36513

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Note that the code didn't have any test to the best of my knowledge, so I have not added any. Besides, the change is trivial so it may not be needed.

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

Hey @sebbaur, feel free to ping me when it's ready for review! Btw, we have activation functions defined  in `ACT2FN` mapping, see llama for example.

Thanks for the prompt reply and the pointers! I hadn't noticed `ACT2FN` -- I have made the change

Once the CI is happy, I will ping you


I had to modify files in dpt and deit too to make CI happy

tests are not done but I suppose they will pass

@qubvel I think it is ready -- it took more work than I initially expected because I had to propagate changes to other parts of the codebase which use the same pooler. This felt a bit out of scope, but I suppose this is WAI? 

Somebody suggested here https://github.com/huggingface/transformers/issues/36513 that this change was not appropriate

I am a bit confused why -- it is a no-op and just makes some hardcoded hyperparameters configurable


Hey! That was me but on reflection I think it's okay - we do have a general rule in our philosophy against adding additional features to existing pretrained model architectures like this, but when it's compact and backward compatibility is guaranteed I think it's probably okay, especially if the PR is already complete.

@qubvel it says `2 workflows awaiting approval` -- do you know who I should reach out to to review this?

@qubvel @Rocketknight1 friendly ping :) -- do you know who should review the workflows? `2 workflows awaiting approval`

@sebbaur the additional workflows are the documentation builder - it's usually not essential to run those. I enabled them anyway, but at this point we just need core maintainer approval and we're good to merge cc @arthurzucker @cyrilvallez

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36517). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hello @ArthurZucker and @Cyrilvallez, would you be able to take a look?

friendly ping :)

Happy Monday @ArthurZucker and @Cyrilvallez! Would you please be able to take a quick look? That would unlock a few things on my side

Thanks!

Hello @Cyrilvallez ! Could you please take a look? Even if this cannot be merged, it would be very helpful for me to know this sooner than later, as I am planning to open-source a Pytorch implementation of https://huggingface.co/google/hear, which this PR allows easily. I need time to find an alternative if this is not ok

For the "Copied from", you can replace them with something such as "Originally copied from", it will stop our linter from forcing the changes, and at the same time we still keep track of the fact it was previously similar, in case we need it later

Alright, after discussions with @qubvel, let's keep the changes that were propagated to the other models, it's fine! You just need to fix the issue on flax vit then 😉 Sorry for the roller coaster! 🙃

Thanks for your review!

I have made the change to the ACT2FN dict -- good catch!

Next time I will know that the models' code shouldn't be updated... Sorry about this

Merging! Thanks!!