Support QuestionAnswering Module for ModernBert based models.
Author: bakrianoo
Pull request: yes

# What does this PR do?

This PR introduces a new feature: `ModernBertForQuestionAnswering`, which extends the `ModernBERT` model to support Question-Answering tasks. 

This addresses a gap in the current implementation of ModernBERT and enables its use for tasks requiring answer extraction from long texts.

## Evaluating

I made an experiment to fine-tune a QnA ModernBert based model. I used [this script](https://github.com/huggingface/transformers/blob/main/examples/pytorch/question-answering/run_qa.py) to produce [this model](https://huggingface.co/rankyx/ModernBERT-QnA-base-squad).

## Dependencies

- No new dependencies were introduced.
- The model was tested using the existing Hugging Face transformers library. Please check a QnA fine-tuned model here [rankyx/ModernBERT-QnA-base-squad](https://huggingface.co/rankyx/ModernBERT-QnA-base-squad)

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Current Status

Although I was be able to finetune a [QnA Modern Bert](https://huggingface.co/rankyx/ModernBERT-QnA-base-squad) model, I am still having an issue to complete the PR.

### 1)

Applying `make fixup` produces this error

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 430, in discover_pjrt_plugins
    plugin_module.initialize()
  File "/usr/local/lib/python3.10/dist-packages/jax_plugins/xla_cuda12/__init__.py", line 85, in initialize
    options = xla_client.generate_pjrt_gpu_plugin_options()
AttributeError: module 'jaxlib.xla_client' has no attribute 'generate_pjrt_gpu_plugin_options'
Traceback (most recent call last):
  File "/content/transformers/utils/check_docstrings.py", line 1060, in <module>
    check_docstrings(overwrite=args.fix_and_overwrite, check_all=args.check_all)
  File "/content/transformers/utils/check_docstrings.py", line 978, in check_docstrings
    for modified_file_diff in repo.index.diff(repo.refs.main.commit):
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 972, in __getattr__
    return list.__getattribute__(self, attr)
AttributeError: 'IterableList' object has no attribute 'main'

```

I tried to reinstall `jaxlib==0.4.13` but it does not help.

I appreciate your help here.




Comments:
Hi @bakrianoo, thanks for the PR!

1) I think the reason you can't load the model with `AutoModelForQuestionAnswering` is that you've added an extra line in the `MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING_NAMES`? Can you make sure the model is only included in the question answering mapping? Once it's in there I believe it should be loadable with the autoclass.

2) The `repo-consistency` error is because the new class isn't added to `docs/source/en/model_doc/modernbert.md`. If you add an autodoc section there, the error should go away!

Hi @Rocketknight1 
Your advice solved my issues. Thank you.
I now have a new issue, can you check the updated PR description?

---

Applying `make fixup` produces this error

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/jax/_src/xla_bridge.py", line 430, in discover_pjrt_plugins
    plugin_module.initialize()
  File "/usr/local/lib/python3.10/dist-packages/jax_plugins/xla_cuda12/__init__.py", line 85, in initialize
    options = xla_client.generate_pjrt_gpu_plugin_options()
AttributeError: module 'jaxlib.xla_client' has no attribute 'generate_pjrt_gpu_plugin_options'
Traceback (most recent call last):
  File "/content/transformers/utils/check_docstrings.py", line 1060, in <module>
    check_docstrings(overwrite=args.fix_and_overwrite, check_all=args.check_all)
  File "/content/transformers/utils/check_docstrings.py", line 978, in check_docstrings
    for modified_file_diff in repo.index.diff(repo.refs.main.commit):
  File "/usr/local/lib/python3.10/dist-packages/git/util.py", line 972, in __getattr__
    return list.__getattribute__(self, attr)
AttributeError: 'IterableList' object has no attribute 'main'

```

I tried to reinstall `jaxlib==0.4.13` but it does not help.


Hi @bakrianoo, I'm sorry for the delay! I ran `make fixup` for you, but you'll likely need to pull the latest changes to your `main` branch and then rebase the PR branch, followed by a force push. That should resolve the conflicts we're seeing here, as well as any failing tests.

Thank you @Rocketknight1 
I had `rebased + pushed` the update into the feature branch.

but when I tried to do `fixup` again, it generated many files to be changed, which are not related to my feature.

I have not yet pushed the changes of `fixup`. waiting for your reply. Thank you.

![image](https://github.com/user-attachments/assets/526f7529-c883-480e-b45a-7d633a59b1e8)




hi @bakrianoo, I fixed the code quality issues with `make fix-copies`. There are still a couple of failing tests, though - see if you can figure them out, and ping me if you're totally stuck!

 Thank you @ArthurZucker 
 Can you explain more or guide me to an example/doc to follow?

Hi @bakrianoo, you can follow these [Usage tips](https://huggingface.co/docs/transformers/model_doc/mistral#usage-tips) as an example!

Thank you @stevhliu 
I already tested this module to fine-tune a QnA model based on [ModernBERT-base](https://huggingface.co/answerdotai/ModernBERT-base)

Here's the [fine-tuned model](https://huggingface.co/rankyx/ModernBERT-QnA-base-squad)

Where can I set these details in the repo?

Hi sorry I wasn't clear! You can make a note of it in `modernbert.md` in a "Usage Tips" section similar to the one I linked [here](https://huggingface.co/docs/transformers/model_doc/mistral#usage-tips).

Hi @stevhliu 
I added some guidelines for tuning and using the tuned models here. Please check

https://github.com/bakrianoo/transformers/blob/feat-ModernBert-QnA-Support/docs/source/en/model_doc/modernbert.md?plain=1

Hi @stevhliu 
Any pending actions from my side?

Thanks for your changes! I think the only thing left to do is to rebase your branch and see if that fixes the remaining failing tests.

Hi all, is this still happening? Was looking to use this and realized it was still pending. LMK if there is anything I can do to help!

> Hi all, is this still happening? Was looking to use this and realized it was still pending. LMK if there is anything I can do to help!

Hi @orionw 
Thanks for your interest in this PR.
Actually, I had some circumstances which collapsed my capacity for this PR.

If you can help in completing this, I would appreciate it.
I just synced my feature branch now and no conflicts were raised.

Thanks @bakrianoo! I'm not able to modify your fork unfortunately - do you mind adding me? 

@ArthurZucker @Rocketknight1 @stevhliu do I just need to rebase it? Seems like the only tests failing are PaliGemma.

hi @orionw, yes, we had some issues on `main`! A rebase should fix things.

Hi @orionw 
I invited you to be a collaborator on the project.
Also, I synced the feature branch now.

Thanks @bakrianoo for the access and for doing the whole PR! @Rocketknight1 I fixed the last tests and did the rebase. 

I think we're good to go now!

All tests passing so cc @cyrilvallez @arthurzucker for core maintainer review

@ArthurZucker if you have a sec, it is a bit of a blocker for evaluations of ModernBERT style models

Hi @Rocketknight1 @ArthurZucker, any updates? Sorry to keep pinging

Would love to see this merged in! Currently the only thing preventing the ModernBERT models to be included in the [EuroEval benchmark](https://euroeval.com/).

Thanks @bakrianoo and @Cyrilvallez! It's updated now, although it's failing due to an Albert test? Seems like that might be coming from `main`

```
  File "/root/transformers/src/transformers/models/albert/modeling_flax_albert.py", line 18, in <module>
    import flax
  File "/usr/local/lib/python3.9/site-packages/flax/__init__.py", line 18, in <module>
    from .configurations import (
  File "/usr/local/lib/python3.9/site-packages/flax/configurations.py", line 92, in <module>
    flax_filter_frames = define_bool_state(
  File "/usr/local/lib/python3.9/site-packages/flax/configurations.py", line 42, in define_bool_state
    return jax_config.define_bool_state('flax_' + name, default, help)
AttributeError: 'Config' object has no attribute 'define_bool_state'
```

Yep sorry, this one is on us! 🙃 Rebasing on latest main now should fix it

@Cyrilvallez I rebased but seem to get another error that seems like it is from main?

```
Cannot access gated repo for url https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/resolve/main/preprocessor_config.json.
Access to model mistralai/Mistral-Small-3.1-24B-Instruct-2503 is restricted. You must have access to it and be authenticated to access it. Please log in.
FAILED tests/models/mistral3/test_processor_mistral3.py::Mistral3ProcessorTest::test_tokenizer_defaults_preserved_by_kwargs_video - OSError: You are trying to access a gated repo.
Make sure to have access to it at https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503.
401 Client Error. (Request ID: Root=1-67e0ecff-2e3d9f34072aaf864519fc97;3d7b3a19-a68f-4cf4-8c29-4f178c3fded3)

...
```

I can wait for a fix for it and rebase that in also, but I think the tests show everything else is passing. We seem to be playing whack-a-mole trying to fix these issues :) 

Okay, @Cyrilvallez @ArthurZucker the rebase from the newest main fixed it! I think this is good to go then!

@Rocketknight1 as well. Sorry for all the pings, just hoping to get this in so evaluation can be done on these models. 

Alright, merging! Thanks a lot!! 🤗

@orionw 
My words cannot thank you enough. Without you, this wouldn't have been done. 

Happy to help @bakrianoo — and you did pretty much all the work so it was easy! Cheers 🎉