Transformers cannot load ModernBERT for sequence classification
Author: eneko-caf
Pull request: no
Labels: name: Usage

name: bug

### System Info

I am trying to test the new ModernBER, following this notebook from the official documentation: https://github.com/AnswerDotAI/ModernBERT/blob/main/examples/finetune_modernbert_on_glue.ipynb model for sequence classification but I am getting the following error: 

```
Traceback (most recent call last):
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1038, in from_pretrained
    config_class = CONFIG_MAPPING[config_dict["model_type"]]
                   ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 740, in __getitem__
    raise KeyError(key)
KeyError: 'modernbert'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/train_full_fine_tuning.py", line 75, in <module>
    train_model_ft(tmp_folder_dataset.name, args)
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/train_full_fine_tuning.py", line 39, in train_model_ft
    orchestrate_training_with_epoch_artifacts(dataset=dataset, args=args)
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/utils/training/training_utils.py", line 153, in orchestrate_training_with_epoch_artifacts
    tokenizer, model = get_model_tokenizer(args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/caf_requirements_training/caf_requirements_training/utils/training/training_utils.py", line 46, in get_model_tokenizer
    model = AutoModelForSequenceClassification.from_pretrained(training_model_name,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ubuntu/req_datalab/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1040, in from_pretrained
    raise ValueError(
ValueError: The checkpoint you are trying to load has model type `modernbert` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
```
I am using:
- Python version: 3.12.7
- Tranformers version: 4.47.1
- Tranformers information:

```
- `transformers` version: 4.47.1
- Platform: Linux-6.8.0-1018-aws-x86_64-with-glibc2.35
- Python version: 3.12.7
- Huggingface_hub version: 0.26.3
- Safetensors version: 0.4.5
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.5.1+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?:  no
- Using GPU in script?: yes
- GPU type: NVIDIA A10G
```

The code snippet used is this:

```
model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", cache_dir=model_saving_path,
                                                               num_labels=12, compile=False)
```

Thank you very much!

### Who can help?

@ArthurZucker
@stevhliu

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just executing 
```
model = AutoModelForSequenceClassification.from_pretrained("answerdotai/ModernBERT-base", cache_dir=model_saving_path,
                                                         num_labels=12, compile=False)
```
the problem will arise 

### Expected behavior

To load the model normally

Comments:
Currently need to compile the latest transformers  

`pip install git+https://github.com/huggingface/transformers`

source: https://huggingface.co/answerdotai/ModernBERT-base/discussions/3

Any idea why I am getting this error during a `train()` of ModernBert on the latest transformers dev branch?
`PyTorch version: 2.4.1+cu118
Transformers version: 4.48.0.dev0`

`BackendCompilerFailed: backend='inductor' raised:
AssertionError: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.clone.default(tensor([...], size=(16,), dtype=torch.uint8), memory_format=torch.contiguous_format)`

upgrading to  `PyTorch version: 2.5.1` seemed to fix the issue

I am getting the following error when trying to import `transformers.trainer` on macOS:
```
RuntimeError: Failed to import transformers.trainer because of the following error (look up to see its traceback):
Failed to import transformers.integrations.integration_utils because of the following error (look up to see its traceback):
Failed to import transformers.modeling_utils because of the following error (look up to see its traceback):
cannot import name 'CompileConfig' from 'transformers.generation'
```
I am using Python 3.12.8, PyTorch 2.5.1, and also have `bitsandbytes`, `accelerate`, and `protobuf` installed in my environment.

Yep this was fixed in a patch sorry for the trouble!