OSError: Can't load tokenizer for fine-tuned model
Author: ccruttjr
Pull request: no

### System Info

**I promise you this issue isn't as long as it seems.** (It's long because I included a lot of context below just in case it was needed)

Hello! I fine-tuned a the gpt2-xl model on some custom data and saved the model. The directory for the saved model is `./saved/` and the file contents are `config.json  generation_config.json  model-00001-of-00002.safetensors  model-00002-of-00002.safetensors  model.safetensors  model.safetensors.index.json`. Let me know if there's more information I can provide other than what's below. When attempting to use the fine-tuned model for text generation, I ran into an error running this:

```python
model = AutoModelForCausalLM.from_pretrained(model_path)
```
getting
```
Traceback (most recent call last):
  File "/home/username/NCAI/inference.py", line 40, in <module>
    main()
  File "/home/username/NCAI/inference.py", line 32, in main
    model, tokenizer = load_model(model_path)
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/NCAI/inference.py", line 9, in load_model
    model = AutoModelForCausalLM.from_pretrained(model_path)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/miniconda3/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 566, in from_pretrained
    return model_class.from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/miniconda3/lib/python3.11/site-packages/transformers/modeling_utils.py", line 3371, in from_pretrained
    with safe_open(resolved_archive_file, framework="pt") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
safetensors_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization
```
I changed how I loaded up the model via
```python
config = AutoConfig.from_pretrained(model_path)
model = AutoModelForCausalLM.from_config(config)
```
which the script got through! Which is why I didn't put that issue in the title. But... I ran into an error right after with this:
```python
tokenizer = AutoTokenizer.from_pretrained(model_path)
# as well as this
tokenizer = GPT2Tokenizer.from_pretrained(model_path)
```
giving
```
Traceback (most recent call last):
  File "/home/username/NCAI/inference.py", line 40, in <module>
    main()
  File "/home/username/NCAI/inference.py", line 32, in main
    model, tokenizer = load_model(model_path)
                       ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/NCAI/inference.py", line 12, in load_model
    tokenizer = AutoTokenizer.from_pretrained(model_path)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/username/miniconda3/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 805, in from_pretrained
    return tokenizer_class_fast.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/daimyollc/miniconda3/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2012, in from_pretrained
    raise EnvironmentError(
OSError: Can't load tokenizer for './saved/'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure './saved/' is the correct path to a directory containing all relevant files for a GPT2TokenizerFast tokenizer.
```
For some extra info, here is my config.json and generation_config.json
```json
{
  "_name_or_path": "gpt2-xl",
  "activation_function": "gelu_new",
  "architectures": [
    "GPT2LMHeadModel"
  ],
  "attn_pdrop": 0.1,
  "bos_token_id": 50256,
  "embd_pdrop": 0.1,
  "eos_token_id": 50256,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "model_type": "gpt2",
  "n_ctx": 1024,
  "n_embd": 1600,
  "n_head": 25,
  "n_inner": null,
  "n_layer": 48,
  "n_positions": 1024,
  "output_past": true,
  "reorder_and_upcast_attn": false,
  "resid_pdrop": 0.1,
  "scale_attn_by_inverse_layer_idx": false,
  "scale_attn_weights": true,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "task_specific_params": {
    "text-generation": {
      "do_sample": true,
      "max_length": 50
    }
  },
  "torch_dtype": "float32",
  "transformers_version": "4.36.2",
  "use_cache": true,
  "vocab_size": 50257
}
```
```json
{
  "_from_model_config": true,
  "bos_token_id": 50256,
  "eos_token_id": 50256,
  "transformers_version": "4.36.2"
}
```
Here is the code that fine-tuned and saved the new model
```python
# 1. Have Transformer's determine the best tokenizer for the given model
# 2. Convert XML to readable dataset. Have the first GPU run it first so multiple GPUs aren't trying to edit the XML at
#    the same time
# 3. Set the max length and padding of each eConsult and how wewant to tokenize the dataset
# 4. Split dataset into training dataset and eval 80/20
# 5. Distribute tokenized datasets across multiple GPUs as to not run out of memory
# 6. Create/return dataloader with the given data for the trainer to use
def get_dataloaders(accelerator: Accelerator, batch_size, model_name, data_location):
    # 1
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    tokenizer.pad_token = tokenizer.eos_token

    # 2
    with accelerator.main_process_first():
        dataset = Dataset.from_pandas(process_dataset(data_location))

    # 3
    def tokenize_function(examples):
        return tokenizer(examples["conversation"], padding="max_length", truncation=True, max_length=256)

    with accelerator.main_process_first():
        tokenized_dataset = dataset.map(tokenize_function, batched=True)

    tokenized_dataset.set_format(
        "torch", columns=["input_ids", "attention_mask"])

    # 4
    split_datasets = tokenized_dataset.train_test_split(test_size=0.2)
    tokenized_train_dataset = split_datasets["train"]
    tokenized_eval_dataset = split_datasets["test"]

    # 5
    train_sampler = DistributedSampler(
        tokenized_train_dataset, num_replicas=accelerator.num_processes, rank=accelerator.process_index, shuffle=True
    )

    eval_sampler = DistributedSampler(
        tokenized_eval_dataset, num_replicas=accelerator.num_processes, rank=accelerator.process_index, shuffle=False
    )

    # 6
    train_dataloader = DataLoader(
        tokenized_train_dataset,
        batch_size=batch_size,
        drop_last=True,
        sampler=train_sampler
    )

    eval_dataloader = DataLoader(
        tokenized_eval_dataset,
        batch_size=batch_size*2,
        drop_last=(accelerator.mixed_precision == "fp8"),
        sampler=eval_sampler
    )

    return train_dataloader, eval_dataloader


# 1. Initialize accelerator with mixed percision and define training parameters via arguments given in command line
# 2. Sets seed (if given as a command line argument) for reproducability
# 3. Get dataloaders
# 4. Initialize more training perameters and "prepare"/optimize them via Accelerate
# 5. Train/fine-tune model with new data & set parameters using FSDP
# 6. Evaluate quality of trainer for that epoch
# 7. Have the first GPU save the newly fine-tuned dataset
def training_function(args):
    # 1
    accelerator = Accelerator(mixed_precision=args.mixed_precision)
    lr = args.lr
    num_epochs = args.num_epochs
    batch_size = args.batch_size
    num_warmup_steps = args.num_warmup_steps

    # 2
    if args.seed:
        set_seed(args.seed)

    # 3
    train_dataloader, eval_dataloader = get_dataloaders(
        accelerator, batch_size, args.model_name, args.data_location)

    # 4
    # Instantiate the model (we build the model here so that the seed also control new weights initialization)
    model = AutoModelForCausalLM.from_pretrained(args.model_name)
    model = accelerator.prepare(model)

    optimizer = AdamW(params=model.parameters(), lr=lr)

    # Instantiate scheduler
    lr_scheduler = get_linear_schedule_with_warmup(
        optimizer=optimizer,
        num_warmup_steps=num_warmup_steps,
        num_training_steps=(len(train_dataloader) *
                            num_epochs),
    )

    # Prepare everything
    # There is no specific order to remember, we just need to unpack the objects in the same order we gave them to the
    # prepare method.
    optimizer, train_dataloader, eval_dataloader, lr_scheduler = accelerator.prepare(
        optimizer, train_dataloader, eval_dataloader, lr_scheduler
    )

    # Initialize logging variables
    total_train_loss = 0
    total_eval_loss = 0

    # 5
    # Now we train the model
    for epoch in range(num_epochs):
        model.train()
        total_train_loss = 0
        for batch in tqdm(train_dataloader, desc="Training"):
            with accelerator.accumulate(model):
                # Process the batch
                inputs = {k: v.to(accelerator.device)
                          for k, v in batch.items()}
                if "labels" not in inputs:
                    inputs["labels"] = inputs["input_ids"]

                outputs = model(**inputs)
                loss = outputs.loss
                total_train_loss += loss.item()
                accelerator.backward(loss)
                optimizer.step()
                lr_scheduler.step()
                optimizer.zero_grad()

        accelerator.wait_for_everyone()

        # 6
        # Evaluation loop after each training epoch
        model.eval()
        total_eval_loss = 0
        for batch in tqdm(eval_dataloader, "Evaluating"):
            with torch.no_grad():
                inputs = {k: v.to(accelerator.device)
                          for k, v in batch.items()}
                if "labels" not in inputs:
                    inputs["labels"] = inputs["input_ids"]

                outputs = model(**inputs)
                loss = outputs.loss
                total_eval_loss += loss.item()

        # Log the average losses
        avg_train_loss = total_train_loss / len(train_dataloader)
        avg_eval_loss = total_eval_loss / len(eval_dataloader)
        print(
            f"Epoch: {epoch}, Average Training Loss: {avg_train_loss}, Average Evaluation Loss: {avg_eval_loss}")

        accelerator.wait_for_everyone()

    # 7
    accelerator.wait_for_everyone()
    accelerator.print("saving")
    accelerator.unwrap_model(model).save_pretrained(
        "./saved_1000",
        is_main_process=accelerator.is_main_process,
        save_function=accelerator.save,
        state_dict=accelerator.get_state_dict(model),
    )


def main():
    args = parse_args()
    training_function(args)


if __name__ == "__main__":
    start = time()
    main()
    print(f"Total Execution Time: {time() - start} seconds")

```
```
$ transformers-cli env
- `transformers` version: 4.36.2
- Platform: Linux-5.15.0-91-generic-x86_64-with-glibc2.35
- Python version: 3.11.5
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: 0.26.1
- Accelerate config:
    - compute_environment: LOCAL_MACHINE
    - distributed_type: FSDP
    - mixed_precision: fp16
    - use_cpu: False
    - debug: False
    - num_processes: 6
    - machine_rank: 0
    - num_machines: 1
    - rdzv_backend: static
    - same_network: True
    - main_training_function: main
    - fsdp_config: {
        'fsdp_auto_wrap_policy': 'TRANSFORMER_BASED_WRAP','
        'fsdp_backward_prefetch': 'BACKWARD_PRE',
        'fsdp_cpu_ram_efficient_loading': True,
        'fsdp_forward_prefetch': False,
        'fsdp_offload_params': False,
        'fsdp_sharding_strategy': 'FULL_SHARD',
        'fsdp_state_dict_type': 'SHARDED_STATE_DICT',
        'fsdp_sync_module_states': True,
        'fsdp_use_orig_params': True
      }
- downcast_bf16: no
- tpu_use_cluster: False
- tpu_use_sudo: False
- tpu_env: []
- PyTorch version (GPU?): 2.1.2 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```bash
$ # create XML file with data we wanna use
$ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
$ conda install transformers accelerate dataset
$ pip instal bs4 pandas tqdm
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ # https://developer.nvidia.com/cuda-zone
$ # ran fine tuning file
$ python inference.py
```

### Expected behavior

For this
```python
tokenizer = GPT2Tokenizer.from_pretrained(model_path)
```
to not fail!

Comments:
Hello, you mentioned that:
>The directory for the saved model is ./saved/ and the file contents are config.json  generation_config.json  model-00001-of-00002.safetensors  model-00002-of-00002.safetensors  model.safetensors  model.safetensors.index.json  

there does not seem to be any tokenizer files here no? 

Yep. Realized fairly quickly I needed to add something like that

```python
    if accelerator.is_main_process:
        print("saving tokenizer")
        # Saving the tokenizer
        tokenizer.save_pretrained(save_location)
        print("saved tokenizer")
```
seems to be resolved