Deepspeed with Trainer RecursionError: maximum recursion depth exceeded while calling a Python object
Author: bettyballin
Pull request: no

### System Info

System Info:
- `transformers` version: 4.25.1
- Platform: Linux-5.4.0-137-generic-x86_64-with-glibc2.31
- Python version: 3.10.8
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)


### Who can help?

@stas00 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


Stack trace:
```
File "myPythonScript.py", line 230, in train
  trainer.train()
File "/miniconda3/envs/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1527, in train
  return inner_training_loop(
File "/miniconda3/envs/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1597, in _inner_training_loop
  deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
File "/miniconda3/envs/venv/lib/python3.10/site-packages/transformers/deepspeed.py", line 344, in deepspeed_init
  deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/__init__.py", line 125, in initialize
    return inner_training_loop(
  File "/miniconda3/envs/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1597, in _inner_training_loop
    deepspeed_engine, optimizer, lr_scheduler = deepspeed_init(
  File "/miniconda3/envs/venv/lib/python3.10/site-packages/transformers/deepspeed.py", line 344, in deepspeed_init
  engine = DeepSpeedEngine(args=args,
File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 348, in wrapper
    deepspeed_engine, optimizer, _, lr_scheduler = deepspeed.initialize(**kwargs)
  File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/__init__.py", line 125, in initialize
    engine = DeepSpeedEngine(args=args,
  File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 348, in wrapper
  if not hasattr(module, "_ds_child_entered"):
File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 495, in __getattr__
    if not hasattr(module, "_ds_child_entered"):
  File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 495, in __getattr__
  if name in dir(self):
File "/home/ballin/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2022, in __dir__
  parameters = list(self._parameters.keys())
File "/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 495, in __getattr__
  if name in dir(self):
....
... multiple hundred lines of the same two function calls ....
....
File "/miniconda3/envs/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2028, in __dir__
  parameters = list(self._parameters.keys())
File "/mnt/ssestorage2-data/ballin/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 495, in __getattr__
  if name in dir(self):
File "/mnt/ssestorage2-data/ballin/miniconda3/envs/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2028, in __dir__
  parameters = list(self._parameters.keys())
File "/mnt/ssestorage2-data/ballin/miniconda3/envs/venv/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 495, in __getattr__
  if name in dir(self):
File "/mnt/ssestorage2-data/ballin/miniconda3/envs/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 2026, in __dir__
  module_attrs = dir(self.__class__)
RecursionError: maximum recursion depth exceeded while calling a Python object
```

The code used in myClassifier.py:
```
MODEL = "microsoft/bloom-deepspeed-inference-fp16"
TOKENIZER = AutoTokenizer.from_pretrained(MODEL)
training_args = TrainingArguments( 
    do_train=True,
    do_eval=True,
    fp16=True,
    load_best_model_at_end=True,
    evaluation_strategy="epoch",
    save_strategy="epoch",
    deepspeed="ds_config.json",
    local_rank=os.environ.get("LOCAL_RANK")
)
config = AutoConfig.from_pretrained(MODEL)
with deepspeed.zero.Init(dtype=torch.float16):
    model = AutoModelForSequenceClassification.from_config(config=config,torch_dtype=torch.float16)
model = model.eval()
dist.barrier()
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=ds["train"],
    eval_dataset=ds["test"],
    tokenizer=TOKENIZER
)   
trainer.train()
```

I am using the deepspeed configuration file from https://huggingface.co/docs/transformers/main/en/main_classes/deepspeed#zero3-config and call deepspeed with ```deepspeed --num_gpus=4 --master_addr="myIP" --master_port=1234 --hostfile=job/hostfile myPythonScript.py``` on 2 nodes using 4x NVIDIA A100 with 80 GB each.



### Expected behavior

The training should run without an error.

Comments:
In general this belongs to https://github.com/microsoft/DeepSpeed/issues as this is a deepspeed issue. I see you file it here https://github.com/microsoft/DeepSpeedExamples/issues/84#issuecomment-1405311822 but I think this is the wrong place.

I have run into this problem myself recently - it was triggered by `zero.Init` - and I had some nested `from_pretrained` calls and multiple `zero.Init` calls. Once I recoded to have only a single `zero.Init` call the problem went away. 

So in your code sample you shouldn't do:

```
with deepspeed.zero.Init(dtype=torch.float16):
    model = AutoModelForSequenceClassification.from_config(config=config,torch_dtype=torch.float16)
```

because `from_config` already uses `zero.Init` internally! so you end up with nested `zero.Init` and it breaks.

It should be just:

```
model = AutoModelForSequenceClassification.from_config(config=config,torch_dtype=torch.float16)
```

Thank you @stas00 , this resolved the issue!