Transformers 5.0.0 breaks defining and then initializing custom models in Jupyter notebooks
Author: umarbutler
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.0.0
- Platform: Linux-6.14.0-37-generic-x86_64-with-glibc2.39
- Python version: 3.12.3
- Huggingface_hub version: 1.3.5
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition

### Who can help?

@CyrilVallez

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When I try defining and initializing a custom model inside of a Jupyter notebook as shown in the provided MRE, I'm getting this traceback:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[9], line 15
     11         # Define layers.
     12         self.classifier: torch.nn.Linear = torch.nn.Linear(1024, 2)
---> 15 model = MyCustomModel(ModernBertConfig())

Cell In[9], line 9, in MyCustomModel.__init__(self, config)
      7 def __init__(self, config) -> None:
      8     # Init.
----> 9     super().__init__(config)
     11     # Define layers.
     12     self.classifier: torch.nn.Linear = torch.nn.Linear(1024, 2)

File ~/XXXXXXXXXXX/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:1300, in PreTrainedModel.__init__(self, config, *inputs, **kwargs)
   1295 self.config._attn_implementation_internal = self._check_and_adjust_attn_implementation(
   1296     self.config._attn_implementation, is_init_check=True
   1297 )
   1298 # Check the experts implementation is supported, or set it if not yet set (on the internal attr, to avoid
   1299 # setting it recursively)
-> 1300 self.config._experts_implementation_internal = self._check_and_adjust_experts_implementation(
   1301     self.config._experts_implementation
   1302 )
   1303 if self.can_generate():
   1304     self.generation_config = GenerationConfig.from_model_config(config)

File ~/XXXXXXXXXXX/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:1893, in PreTrainedModel._check_and_adjust_experts_implementation(self, experts_implementation)
   1883 def _check_and_adjust_experts_implementation(self, experts_implementation: str | None) -> str:
   1884     """
   1885     Check that the `experts_implementation` exists and is supported by the models.
   1886 
   (...)   1891         `str`: The final experts implementation to use.
   1892     """
-> 1893     applicable_experts_implementation = self.get_correct_experts_implementation(experts_implementation)
   1894     return applicable_experts_implementation

File ~/XXXXXXXXXXX/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:1942, in PreTrainedModel.get_correct_experts_implementation(self, requested_experts)
   1940 if applicable_experts == "grouped_mm":
   1941     try:
-> 1942         self._grouped_mm_can_dispatch()
   1943     except (ValueError, ImportError) as e:
   1944         if requested_experts == "grouped_mm":

File ~/XXXXXXXXXXX/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:1764, in PreTrainedModel._grouped_mm_can_dispatch(self)
   1759 def _grouped_mm_can_dispatch(self) -> bool:
   1760     """
   1761     Check the availability of Grouped MM for a given model.
   1762     """
-> 1764     if not self._can_set_experts_implementation():
   1765         raise ValueError(f"{self.__class__.__name__} does not support setting experts implementation.")
   1767     if not is_grouped_mm_available():

File ~/XXXXXXXXXXX/.venv/lib/python3.12/site-packages/transformers/modeling_utils.py:1973, in PreTrainedModel._can_set_experts_implementation(cls)
   1968 @classmethod
   1969 def _can_set_experts_implementation(cls) -> bool:
   1970     """Detect whether the class supports setting its experts implementation dynamically. It is an ugly check based on
   1971     opening the file, but avoids maintaining yet another property flag.
   1972     """
-> 1973     class_file = sys.modules[cls.__module__].__file__
   1974     with open(class_file, "r", encoding="utf-8") as f:
   1975         code = f.read()

AttributeError: module '__main__' has no attribute '__file__'
```

Here is the MRE:
```python
import torch

from transformers import PreTrainedModel, ModernBertConfig


class MyCustomModel(PreTrainedModel):
    config_class = ModernBertConfig

    def __init__(self, config: ModernBertConfig) -> None:
        # Init.
        super().__init__(config)

        # Define layers.
        self.classifier: torch.nn.Linear = torch.nn.Linear(1024, 2)


model = MyCustomModel(ModernBertConfig())
```

### Expected behavior

The model should load correctly.

Comments:
Thanks for the report! I opened https://github.com/huggingface/transformers/pull/43690 to fix it as it should indeed be a valid usage!