RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
Author: dshwei
Pull request: no

### System Info
```python
   **errors take place as following:**
    
    Detected CUDA files, patching ldflags
Emitting ninja build file /demo/.cache/torch_extensions/py39_cu121/fused_adam/build.ninja...
Building extension module fused_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
ninja: no work to do.
Loading extension module fused_adam...
Time to load fused_adam op: 0.30463647842407227 seconds
/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  self._dummy_overflow_buf = get_accelerator().IntTensor([0])
Loading extension module fused_adam...
Time to load fused_adam op: 0.31080079078674316 seconds
/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/ops/adam/fused_adam.py:96: UserWarning: The torch.cuda.*DtypeTensor constructors are no longer recommended. It's best to use methods such as torch.tensor(data, dtype=*, device='cuda') to create tensors. (Triggered internally at ../torch/csrc/tensor/python_tensor.cpp:83.)
  self._dummy_overflow_buf = get_accelerator().IntTensor([0])
Parameter Offload: Total persistent parameters: 266240 in 65 params
`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...

  0%|          | 0/3000 [00:00<?, ?it/s]`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...
/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/utils/checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/utils/checkpoint.py:429: UserWarning: torch.utils.checkpoint: please pass in use_reentrant=True or use_reentrant=False explicitly. The default value of use_reentrant will be updated to be False in the future. To maintain current behavior, pass use_reentrant=True. It is recommended that you use use_reentrant=False. Refer to docs for more details on the differences between the two variants.
  warnings.warn(
/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/utils/checkpoint.py:61: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
  warnings.warn(
Traceback (most recent call last):
  File "/demo/project/NL2SQL/starcoder-git/finetune_sft_mask_fp32.py", line 549, in <module>
Traceback (most recent call last):
  File "/demo/project/NL2SQL/starcoder-git/finetune_sft_mask_fp32.py", line 549, in <module>
    main(args)
  File "/demo/project/NL2SQL/starcoder-git/finetune_sft_mask_fp32.py", line 538, in main
    run_training(args, train_dataset)
  File "/demo/project/NL2SQL/starcoder-git/finetune_sft_mask_fp32.py", line 509, in run_training
    trainer.train()
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/trainer.py", line 2781, in training_step
    self.accelerator.backward(loss)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/accelerate/accelerator.py", line 1899, in backward
    self.deepspeed_engine_wrapped.backward(loss, **kwargs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/accelerate/utils/deepspeed.py", line 167, in backward
    main(args)
  File "/demo/project/NL2SQL/starcoder-git/finetune_sft_mask_fp32.py", line 538, in main
    run_training(args, train_dataset)
  File "/demo/project/NL2SQL/starcoder-git/finetune_sft_mask_fp32.py", line 509, in run_training
    trainer.train()
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/trainer.py", line 1539, in train
    return inner_training_loop(
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/trainer.py", line 1869, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/transformers/trainer.py", line 2781, in training_step
    self.accelerator.backward(loss)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/accelerate/accelerator.py", line 1899, in backward
    self.deepspeed_engine_wrapped.backward(loss, **kwargs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/accelerate/utils/deepspeed.py", line 167, in backward
        self.engine.backward(loss, **kwargs)self.engine.backward(loss, **kwargs)

  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1955, in backward
    ret_val = func(*args, **kwargs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/runtime/engine.py", line 1955, in backward
        self.optimizer.backward(loss, retain_graph=retain_graph)self.optimizer.backward(loss, retain_graph=retain_graph)

  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/utils/nvtx.py", line 15, in wrapped_fn
    ret_val = func(*args, **kwargs)    
ret_val = func(*args, **kwargs)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 2135, in backward
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/runtime/zero/stage3.py", line 2135, in backward
    self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
    self.loss_scaler.backward(loss.float(), retain_graph=retain_graph)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/deepspeed/runtime/fp16/loss_scaler.py", line 63, in backward
    scaled_loss.backward(retain_graph=retain_graph)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/_tensor.py", line 492, in backward
    scaled_loss.backward(retain_graph=retain_graph)
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/_tensor.py", line 492, in backward
    torch.autograd.backward(
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward
    torch.autograd.backward(
  File "/demo/miniconda3/envs/sqlcode/lib/python3.9/site-packages/torch/autograd/__init__.py", line 251, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

  0%|          | 0/3000 [00:01<?, ?it/s]
[2024-04-01 19:39:20,275] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2847887
[2024-04-01 19:39:20,330] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2847888
[2024-04-01 19:39:20,330] [ERROR] [launch.py:321:sigkill_handler] ['/demo/miniconda3/envs/sqlcode/bin/python', '-u', 'finetune_sft_mask_fp32.py', '--local_rank=1', '--model_path=/demo/data/CodeLlama-7b-Instruct-hf/', '--dataset_name=/demo/project/NL2SQL/NL2SQL/data_upset/text2sql_data_3_27.json', '--split=train', '--num_workers', '8', '--size_train_set', '0', '--seq_length', '2000', '--max_steps', '3000', '--num_train_epochs', '8', '--input_column_name=question', '--input_column_name2=table_schema', '--output_column_name=SQL', '--log_freq', '1', '--save_freq', '100', '--use_lora', '1', '--lora_alpha', '32', '--lora_r', '16', '--lora_dropout', '0.05', '--use_p_tuning', '0', '--num_virtual_tokens', '20', '--encoder_hidden_size', '64', '--use_prefix_tuning', '0', '--num_virtual_tokens2', '64', '--prefix_projection', '0', '--prepare_model_for_int8_training', '1', '--output_dir=/demo/data/nl2sql/checkpoints/sqlcoder2_4_hf_7_fp32/', '--deepspeed_config', 'ds_config_zero3.json', '--per_device_train_batch_size', '1'] exits with return code = 1
```

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```bash
deepspeed  --master_port 29532 finetune_sft_mask_fp32.py \
    --model_path="/demo/data/CodeLlama-7b-Instruct-hf/" \
    --dataset_name="/demo/project/NL2SQL/NL2SQL/data_upset/text2sql_data_3_27.json" \
    --split="train" \
    --num_workers 8 \
    --size_train_set 0 \
    --seq_length 2000 \
    --max_steps 3000 \
    --num_train_epochs 8 \
    --input_column_name="question" \
    --input_column_name2="table_schema" \
    --output_column_name="SQL" \
    --log_freq 1 \
    --save_freq 100 \
    --use_lora 1 \
    --lora_alpha 32 \
    --lora_r 16\
    --lora_dropout 0.05 \
    --use_p_tuning 0 \
    --num_virtual_tokens 20 \
    --encoder_hidden_size 64 \
    --use_prefix_tuning 0 \
    --num_virtual_tokens2 64 \
    --prefix_projection 0 \
    --prepare_model_for_int8_training 1 \
    --output_dir="/demo/data/nl2sql/checkpoints/sqlcoder2_4_hf_7_fp32/"  \
    --deepspeed_config ds_config_zero3.json \
    --per_device_train_batch_size 1 \
 ```
python script as following:  
```python3
 config = AutoConfig.from_pretrained(args.model_path, **config_kwargs)
    model = AutoModelForCausalLM.from_pretrained(
        args.model_path,
        config =config,
        use_auth_token=True,
        torch_dtype=torch.bfloat16,
        device_map={"": Accelerator().process_index},
        use_flash_attention_2=True
    )
```
### Expected behavior

**using  llama2-7b-instruct model,  config.json showing as following:****
```
{
  "architectures": [
    "LlamaForCausalLM"
  ],
  "attention_bias": false,
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "eos_token_id": 2,
  "hidden_act": "silu",
  "hidden_size": 4096,
  "initializer_range": 0.02,
  "intermediate_size": 11008,
  "max_position_embeddings": 16384,
  "model_type": "llama",
  "num_attention_heads": 32,
  "num_hidden_layers": 32,
  "num_key_value_heads": 32,
  "pretraining_tp": 1,
  "rms_norm_eps": 1e-05,
  "rope_scaling": null,
  "rope_theta": 1000000,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.37.2",
  "use_cache": true,
  "vocab_size": 32016
}    
```
**python script load  model as following:**
```python2
model = AutoModelForCausalLM.from_pretrained(
        args.model_path,
        config =config,
        use_auth_token=True,
        torch_dtype=torch.bfloat16,
        device_map={"": Accelerator().process_index},
        use_flash_attention_2=True
    )
    
 ```

Comments:
cc @pacman100 and @SunMarc . Before they can answer I think having the full reproducer is important @dshwei 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.