Problems with Training ModernBERT
Author: hyunjongkimmath
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.3
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.12.9
- Huggingface_hub version: 0.28.1
- Safetensors version: 0.5.2
- Accelerate version: 1.3.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.6.0+cu126 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: Parallel (I'm not sure; I'm using a single GPU on a single machine)
- Using GPU in script?: Yes
- GPU type: NVIDIA GeForce RTX 2060




I have also tried installing `python3.12-dev` in response to the following initial error message (included with the code snippet later)

```python
/usr/include/python3.12/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.12/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.12/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
```

but the error persists.

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction


I am essentially replicating the code from the following: 

- https://github.com/di37/ner-electrical-engineering-finetuning/blob/main/notebooks/01_data_tokenization.ipynb
- https://github.com/di37/ner-electrical-engineering-finetuning/blob/main/notebooks/02_finetuning.ipynb
- See also: https://blog.cubed.run/automating-electrical-engineering-text-analysis-with-named-entity-recognition-ner-part-1-babd2df422d8

The following is the code for setting up the training process:

```python
import os 
from transformers import BertTokenizerFast
from datasets import load_dataset

# from utilities import MODEL_ID, DATASET_ID, OUTPUT_DATASET_PATH
DATASET_ID = "disham993/ElectricalNER"
MODEL_ID = "answerdotai/ModernBERT-large"

LOGS = "logs"
OUTPUT_DATASET_PATH = os.path.join(
    "data", "tokenized_electrical_ner_modernbert"
)  # "data"
OUTPUT_DIR = "models"
MODEL_PATH = os.path.join(OUTPUT_DIR, MODEL_ID)
OUTPUT_MODEL = os.path.join(OUTPUT_DIR, f"electrical-ner-{MODEL_ID.split('/')[-1]}")

EVAL_STRATEGY = "epoch"
LEARNING_RATE = 1e-5
PER_DEVICE_TRAIN_BATCH_SIZE = 64
PER_DEVICE_EVAL_BATCH_SIZE = 64
NUM_TRAIN_EPOCHS = 5
WEIGHT_DECAY = 0.01

LOCAL_MODELS = {
    "google-bert/bert-base-uncased": "electrical-ner-bert-base-uncased",
    "distilbert/distilbert-base-uncased": "electrical-ner-distilbert-base-uncased",
    "google-bert/bert-large-uncased": "electrical-ner-bert-large-uncased",
    "answerdotai/ModernBERT-base": "electrical-ner-ModernBERT-base",
    "answerdotai/ModernBERT-large": "electrical-ner-ModernBERT-large",
}

ONLINE_MODELS = {
    "google-bert/bert-base-uncased": "disham993/electrical-ner-bert-base",
    "distilbert/distilbert-base-uncased": "disham993/electrical-ner-distilbert-base",
    "google-bert/bert-large-uncased": "disham993/electrical-ner-bert-large",
    "answerdotai/ModernBERT-base": "disham993/electrical-ner-ModernBERT-base",
    "answerdotai/ModernBERT-large": "disham993/electrical-ner-ModernBERT-large",
}

electrical_ner_dataset = load_dataset(DATASET_ID, trust_remote_code=True)
print(electrical_ner_dataset)


from datasets import DatasetDict
shrunk_train = electrical_ner_dataset['train'].select(range(10))
shrunk_valid = electrical_ner_dataset['validation'].select(range(5))
shrunk_test = electrical_ner_dataset['test'].select(range(5))

electrical_ner_dataset = DatasetDict({
    'train': shrunk_train,
    'validation': shrunk_valid,
    'test': shrunk_test
})

electrical_ner_dataset.shape


tokenizer = BertTokenizerFast.from_pretrained(MODEL_ID)

def tokenize_and_align_labels(examples, label_all_tokens=True):
    """
    Function to tokenize and align labels with respect to the tokens. This function is specifically designed for
    Named Entity Recognition (NER) tasks where alignment of the labels is necessary after tokenization.

    Parameters:
    examples (dict): A dictionary containing the tokens and the corresponding NER tags.
                     - "tokens": list of words in a sentence.
                     - "ner_tags": list of corresponding entity tags for each word.

    label_all_tokens (bool): A flag to indicate whether all tokens should have labels.
                             If False, only the first token of a word will have a label,
                             the other tokens (subwords) corresponding to the same word will be assigned -100.

    Returns:
    tokenized_inputs (dict): A dictionary containing the tokenized inputs and the corresponding labels aligned with the tokens.
    """
    tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True)
    labels = []
    for i, label in enumerate(examples["ner_tags"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)
        # word_ids() => Return a list mapping the tokens
        # to their actual word in the initial sentence.
        # It Returns a list indicating the word corresponding to each token.
        previous_word_idx = None
        label_ids = []
        # Special tokens like `
` and `<\s>` are originally mapped to None
        # We need to set the label to -100 so they are automatically ignored in the loss function.
        for word_idx in word_ids:
            if word_idx is None:
                # set –100 as the label for these special tokens
                label_ids.append(-100)
            # For the other tokens in a word, we set the label to either the current label or -100, depending on
            # the label_all_tokens flag.
            elif word_idx != previous_word_idx:
                # if current word_idx is != prev then its the most regular case
                # and add the corresponding token
                label_ids.append(label[word_idx])
            else:
                # to take care of sub-words which have the same word_idx
                # set -100 as well for them, but only if label_all_tokens == False
                label_ids.append(label[word_idx] if label_all_tokens else -100)
                # mask the subword representations after the first subword

            previous_word_idx = word_idx
        labels.append(label_ids)
    tokenized_inputs["labels"] = labels
    return tokenized_inputs

tokenized_datasets = electrical_ner_dataset.map(tokenize_and_align_labels, batched=True)
tokenized_electrical_ner_dataset = tokenized_datasets

import os 
import numpy as np
from transformers import AutoTokenizer
from transformers import DataCollatorForTokenClassification
from transformers import AutoModelForTokenClassification
from datasets import load_from_disk
from transformers import TrainingArguments, Trainer
import evaluate
import json

import pandas as pd 

label_list= tokenized_electrical_ner_dataset["train"].features["ner_tags"].feature.names
num_labels = len(label_list)

print(f"Labels: {label_list}")
print(f"Number of labels: {num_labels}")

model = AutoModelForTokenClassification.from_pretrained(MODEL_ID, num_labels=num_labels)
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)

args = TrainingArguments(
    output_dir=MODEL_PATH,
    eval_strategy=EVAL_STRATEGY,
    learning_rate=LEARNING_RATE,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    num_train_epochs=NUM_TRAIN_EPOCHS,
    weight_decay=WEIGHT_DECAY,
    push_to_hub=False
)

data_collator = DataCollatorForTokenClassification(tokenizer)

def compute_metrics(eval_preds):
    """
    Function to compute the evaluation metrics for Named Entity Recognition (NER) tasks.
    The function computes precision, recall, F1 score and accuracy.

    Parameters:
    eval_preds (tuple): A tuple containing the predicted logits and the true labels.

    Returns:
    A dictionary containing the precision, recall, F1 score and accuracy.
    """
    pred_logits, labels = eval_preds

    pred_logits = np.argmax(pred_logits, axis=2)
    # the logits and the probabilities are in the same order,
    # so we don’t need to apply the softmax

    # We remove all the values where the label is -100
    predictions = [
        [label_list[eval_preds] for (eval_preds, l) in zip(prediction, label) if l != -100]
        for prediction, label in zip(pred_logits, labels)
    ]

    true_labels = [
      [label_list[l] for (eval_preds, l) in zip(prediction, label) if l != -100]
       for prediction, label in zip(pred_logits, labels)
    ]
    metric = evaluate.load("seqeval")
    results = metric.compute(predictions=predictions, references=true_labels)
    return {
        "precision": results["overall_precision"],
        "recall": results["overall_recall"],
        "f1": results["overall_f1"],
        "accuracy": results["overall_accuracy"],
    }

trainer = Trainer(
    model,
    args,
   train_dataset=tokenized_electrical_ner_dataset["train"],
   eval_dataset=tokenized_electrical_ner_dataset["validation"],
   data_collator=data_collator,
   tokenizer=tokenizer,
   compute_metrics=compute_metrics,
)

trainer.train()
```

```python
The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization. 
The tokenizer class you load from this checkpoint is 'PreTrainedTokenizerFast'. 
The class this function is called from is 'BertTokenizerFast'.
Map: 100%|█████████████████████████████| 10/10 [00:00<00:00, 1220.41 examples/s]
Map: 100%|████████████████████████████████| 5/5 [00:00<00:00, 818.40 examples/s]
Map: 100%|███████████████████████████████| 5/5 [00:00<00:00, 1246.82 examples/s]
Some weights of ModernBertForTokenClassification were not initialized from the model checkpoint at answerdotai/ModernBERT-large and are newly initialized: ['classifier.bias', 'classifier.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
/home/hyunjong/Documents/Development/Python/trouver_personal_playground/playgrounds/ml_model_training_playground/modernbert_training_error.py:183: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.
  trainer = Trainer(
  0%|                                                    | 0/50 [00:00<?, ?it/s]In file included from /usr/include/python3.12/Python.h:12:0,
                 from /tmp/tmpjbmobkir/main.c:5:
/usr/include/python3.12/pyconfig.h:3:12: fatal error: x86_64-linux-gnu/python3.12/pyconfig.h: No such file or directory
 #  include <x86_64-linux-gnu/python3.12/pyconfig.h>
            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
Traceback (most recent call last):
  File "/home/hyunjong/Documents/Development/Python/trouver_personal_playground/playgrounds/ml_model_training_playground/modernbert_training_error.py", line 193, in <module>
    trainer.train()
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/trainer.py", line 2171, in train
    return inner_training_loop(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/trainer.py", line 2531, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/trainer.py", line 3675, in training_step
    loss = self.compute_loss(model, inputs, num_items_in_batch=num_items_in_batch)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/trainer.py", line 3731, in compute_loss
    outputs = model(**inputs)
              ^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/models/modernbert/modeling_modernbert.py", line 1349, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/models/modernbert/modeling_modernbert.py", line 958, in forward
    hidden_states = self.embeddings(input_ids=input_ids, inputs_embeds=inputs_embeds)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/transformers/models/modernbert/modeling_modernbert.py", line 217, in forward
    self.compiled_embeddings(input_ids)
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1380, in __call__
    return self._torchdynamo_orig_callable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 1164, in __call__
    result = self._inner_convert(
             ^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 547, in __call__
    return _compile(
           ^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
    return _compile_inner(code, one_graph, hooks, transform)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
    out_code = transform_code_object(code, transform)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
    transformations(instructions, code_options)
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
    tracer.run()
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 2868, in run
    super().run()
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
    while self.step():
          ^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
    self.dispatch_table[inst.opcode](self, inst)
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 3048, in RETURN_VALUE
    self._return(inst)
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/symbolic_convert.py", line 3033, in _return
    self.output.compile_subgraph(
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1101, in compile_subgraph
    self.compile_and_call_fx_graph(
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1382, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1432, in call_user_compiler
    return self._call_user_compiler(gm)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1483, in _call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/output_graph.py", line 1462, in _call_user_compiler
    compiled_fn = compiler_fn(gm, self.example_inputs())
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/repro/after_dynamo.py", line 130, in __call__
    compiled_gm = compiler_fn(gm, example_inputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/__init__.py", line 2340, in __call__
    return compile_fx(model_, inputs_, config_patches=self.config)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1863, in compile_fx
    return aot_autograd(
           ^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/backends/common.py", line 83, in __call__
    cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1155, in aot_module_simplified
    compiled_fn = dispatch_and_compile()
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 1131, in dispatch_and_compile
    compiled_fn, _ = create_aot_dispatcher_function(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 580, in create_aot_dispatcher_function
    return _create_aot_dispatcher_function(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 830, in _create_aot_dispatcher_function
    compiled_fn, fw_metadata = compiler_fn(
                               ^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 678, in aot_dispatch_autograd
    compiled_fw_func = aot_config.fw_compiler(fw_module, adjusted_flat_args)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_functorch/aot_autograd.py", line 489, in __call__
    return self.compiler_fn(gm, example_inputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1741, in fw_compiler_base
    return inner_compile(
           ^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 569, in compile_fx_inner
    return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_dynamo/repro/after_aot.py", line 102, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 685, in _compile_fx_inner
    mb_compiled_graph = fx_codegen_and_compile(
                        ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1129, in fx_codegen_and_compile
    return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 1044, in codegen_and_compile
    compiled_fn = graph.compile_to_module().call
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 2027, in compile_to_module
    return self._compile_to_module()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 2033, in _compile_to_module
    self.codegen_with_cpp_wrapper() if self.cpp_wrapper else self.codegen()
                                                             ^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/graph.py", line 1968, in codegen
    self.scheduler.codegen()
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 3477, in codegen
    return self._codegen()
           ^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/scheduler.py", line 3554, in _codegen
    self.get_backend(device).codegen_node(node)
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/codegen/cuda_combined_scheduling.py", line 80, in codegen_node
    return self._triton_scheduling.codegen_node(node)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1219, in codegen_node
    return self.codegen_node_schedule(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/codegen/simd.py", line 1263, in codegen_node_schedule
    src_code = kernel.codegen_kernel()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/codegen/triton.py", line 3154, in codegen_kernel
    **self.inductor_meta_common(),
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/_inductor/codegen/triton.py", line 3013, in inductor_meta_common
    "backend_hash": torch.utils._triton.triton_hash_with_backend(),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/utils/_triton.py", line 111, in triton_hash_with_backend
    backend = triton_backend()
              ^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/torch/utils/_triton.py", line 103, in triton_backend
    target = driver.active.get_current_target()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/runtime/driver.py", line 23, in __getattr__
    self._initialize_obj()
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/runtime/driver.py", line 20, in _initialize_obj
    self._obj = self._init_fn()
                ^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/runtime/driver.py", line 9, in _create_driver
    return actives[0]()
           ^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 450, in __init__
    self.utils = CudaUtils()  # TODO: make static
                 ^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 80, in __init__
    mod = compile_module_from_src(Path(os.path.join(dirname, "driver.c")).read_text(), "cuda_utils")
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/backends/nvidia/driver.py", line 57, in compile_module_from_src
    so = _build(name, src_path, tmpdir, library_dirs(), include_dir, libraries)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/runtime/build.py", line 50, in _build
    ret = subprocess.check_call(cc_cmd)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 415, in check_call
    raise CalledProcessError(retcode, cmd)
torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
CalledProcessError: Command '['/home/hyunjong/anaconda3/bin/x86_64-conda-linux-gnu-cc', '/tmp/tmpjbmobkir/main.c', '-O3', '-shared', '-fPIC', '-Wno-psabi', '-o', '/tmp/tmpjbmobkir/cuda_utils.cpython-312-x86_64-linux-gnu.so', '-lcuda', '-L/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/backends/nvidia/lib', '-L/lib/x86_64-linux-gnu', '-L/lib/i386-linux-gnu', '-I/home/hyunjong/Documents/Development/Python/trouver_py312_venv/lib/python3.12/site-packages/triton/backends/nvidia/include', '-I/tmp/tmpjbmobkir', '-I/usr/include/python3.12']' returned non-zero exit status 1.

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True

  0%|          | 0/50 [00:01<?, ?it/s]  
```





### Expected behavior

For the training process to happen.

Comments:
This feels like an environment issue that we probably can't debug for you, unfortunately! To investigate this issue, we need some simpler code that reproduces the problem, and ideally you should check this code on other machines (e.g. Colab) to verify that the bug occurs there as well.

I copied and ran your code [in a collab notebook](https://colab.research.google.com/drive/1zBzg1BK6iFZsIWdrLwcUNKRwHJ4wa3N5?usp=sharing) and it works fine there! 

As mentioned, probably an env issue.

Closing for now, but feel free to reopen if you can make a reproducible bug report!

Thank you @Rocketknight1 @neonwatty both for helping me to address this issue.

Just in case anyone in the future has the same issue, it seems that the issue is indeed an environment related one and removing anaconda from my machine seems to have fixed the issue for me (I suspect that the Python configuration/installation of anaconda was getting in the way of the apt installation of Python in my case)

Now I just need to figure out if I can set up a dataset with short enough max sequence length so that my 6GB VRAM GPU can handle the fine-tuning process.