Error Training with Deepspeed configs
Author: andysingal
Pull request: no

### System Info

Colab Notebook

### Who can help?

@pacman100  @muellerz  

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoTokenizer, AutoModelForCausalLM, Trainer, TrainingArguments
from datasets import load_dataset
from transformers import DataCollatorForLanguageModeling
from transformers import TrainerCallback
import torch
import transformers
def main():
    # Load the tokenizer
    tokenizer = AutoTokenizer.from_pretrained('TinyLlama/TinyLlama-1.1B-step-50K-105b')
    tokenizer.add_special_tokens({'pad_token': '[PAD]'})
    # Load dataset from the Hugging Face datasets library
    dataset = load_dataset("wikitext", "wikitext-2-raw-v1")


    # Tokenize the texts
    def tokenize_function(examples):
        return tokenizer(examples["text"], padding="max_length", truncation=True, max_length=512)


    tokenized_datasets = dataset.map(tokenize_function, batched=True)


    # Load the data collator
    data_collator = DataCollatorForLanguageModeling(
        tokenizer=tokenizer,
        mlm=False,  # TinyLlama uses a causal (not masked) language model, similar to GPT-2
    )


    # Load the model
    model = AutoModelForCausalLM.from_pretrained('TinyLlama/TinyLlama-1.1B-step-50K-105b')


    # Define the training arguments
    training_args = TrainingArguments(
        output_dir='./results',
        overwrite_output_dir=True,
        num_train_epochs=3,
        per_device_train_batch_size=1,
        save_steps=10_000,
        save_total_limit=2,
        fp16=True, 
        deepspeed="path_to_deepspeed_config.json",  # Path to DeepSpeed config file        
        gradient_checkpointing=True,
	report_to='wandb'    
    )


    # Initialize the Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        data_collator=data_collator,
        train_dataset=tokenized_datasets["train"],
        eval_dataset=tokenized_datasets["validation"]
    )


    # Start the training
    trainer.train()


    # Save the final model and tokenizer
    model.save_pretrained('./final_model')
    tokenizer.save_pretrained('./final_model')


if __name__ == "__main__":
    main()

```

```
{
    "zero_optimization": {
      "stage": 3,
      "offload_param": {
        "device": "cpu"
      },
      "offload_optimizer": {
        "device": "cpu"
      }
    },
    "gradient_accumulation_steps": 1,
    "train_micro_batch_size_per_gpu": 1,
    "gradient_clipping": 1.0,
    "fp16": {
      "enabled": true
    }
  }
```

gives error:
```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
[<ipython-input-3-55e1fc0e8c28>](https://localhost:8080/#) in <cell line: 68>()
     67 
     68 if __name__ == "__main__":
---> 69     main()

3 frames
[/usr/local/lib/python3.10/dist-packages/transformers/integrations/deepspeed.py](https://localhost:8080/#) in is_deepspeed_zero3_enabled()
    285 def is_deepspeed_zero3_enabled():
    286     if _hf_deepspeed_config_weak_ref is not None and _hf_deepspeed_config_weak_ref() is not None:
--> 287         return _hf_deepspeed_config_weak_ref().is_zero3()
    288     else:
    289         return False

AttributeError: 'HfTrainerDeepSpeedConfig' object has no attribute 'is_zero3'
```

### Expected behavior

it needs to run smoothly 

Comments:
Could you share the output of `transformers-cli env` that will help us as well 😉 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.