Repository: hpcaitech/ColossalAI

Issue:
[BUG]: Error during running chatgpt train_rm.sh 
### 🐛 Describe the bug

I run into a runtime error when running `train_rm.sh` under examples of chatgpt.
```
Traceback (most recent call last):
  File "/root/ColossalAI/applications/ChatGPT/examples/train_reward_model.py", line 53, in <module>
    train(args)
  File "/root/ColossalAI/applications/ChatGPT/examples/train_reward_model.py", line 36, in train
    trainer.fit(use_lora=args.lora_rank)
  File "/root/miniconda3/envs/chatgpt/lib/python3.10/site-packages/chatgpt/trainer/rm.py", line 57, in fit
    loss.backward()
  File "/root/miniconda3/envs/chatgpt/lib/python3.10/site-packages/torch/_tensor.py", line 488, in backward
    torch.autograd.backward(
  File "/root/miniconda3/envs/chatgpt/lib/python3.10/site-packages/torch/autograd/__init__.py", line 197, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
```


The model is downloaded from Huggingface `bigscience/bloom-560m`.  `lora_rank` is `16`. Is there any hints on this?


### Environment

Environment:
torch                    1.13.1+cu113

------------ Environment ------------
Colossal-AI version: 0.2.4
PyTorch version: 1.13.1
CUDA version: 11.3
CUDA version required by PyTorch: 11.7

Note:
1. The table above checks the versions of the libraries/tools in the current environment
2. If the CUDA version is N/A, you can set the CUDA_HOME environment variable to locate it

------------ CUDA Extensions AOT Compilation ------------
Found AOT CUDA Extension: x
PyTorch version used for AOT compilation: N/A
CUDA version used for AOT compilation: N/A

Note:
1. AOT (ahead-of-time) compilation of the CUDA kernels occurs during installation when the environment varialbe CUDA_EXT=1 is set
2. If AOT compilation is not enabled, stay calm as the CUDA kernels can still be built during runtime

------------ Compatibility ------------
PyTorch version match: N/A
System and PyTorch CUDA version match: x
System and Colossal-AI CUDA version match: N/A

Note:
1. The table above checks the version compatibility of the libraries/tools in the current environment
   - PyTorch version mistach: whether the PyTorch version in the current environment is compatible with the PyTorch version used for AOT compilation
   - System and PyTorch CUDA version match: whether the CUDA version in the current environment is compatible with the CUDA version required by PyTorch
   - System and Colossal-AI CUDA version match: whether the CUDA version in the current environment is compatible with the CUDA version used for AOT compilation
[tensor] fix some unittests

[tensor] fix some unittests



Discussion:
 
> Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿      
----  
**Title:** [BUG]: Error during running chatgpt train_rm.sh    
      



Resolving patch:
diff --git a/applications/ChatGPT/chatgpt/trainer/rm.py b/applications/ChatGPT/chatgpt/trainer/rm.py
--- a/applications/ChatGPT/chatgpt/trainer/rm.py
+++ b/applications/ChatGPT/chatgpt/trainer/rm.py
@@ -43,7 +43,7 @@ def fit(self, use_lora):
             # train
             if use_lora > 0:
                 print("Using Lora")
-                lora.mark_only_lora_as_trainable(self.model)
+                lora.mark_only_lora_as_trainable(self.model.model)
             else:
                 self.model.train()
             for chosen_ids, c_mask, reject_ids, r_mask in self.train_dataloader:


Test patch: