Repository: microsoft/DeepSpeed

Issue:
[BUG] GPT-Neo inference examples broken in Master
**Describe the bug**
Get error `AttributeError: module 'transformers.models' has no attribute 'opt'.` when trying to run `gpt-neo.py` from DeepSpeedExamples using master build from DeepSpeed github.

GPT-neo example from this tutorial also fails with the same error https://www.deepspeed.ai/tutorials/inference-tutorial/

**To Reproduce**
Steps to reproduce the behavior:
1. clone deepspeed from github
2. clone DeepSpeeedExamples from github
3. cd DeepSpeedExamples/inference/huggingface
4. python gpt-neo.py
5. results in the following error
```
***************** Creating model in RANK (0) with WORLD_SIZE = 1 *****************
[2022-08-22 18:06:41,060] [INFO] [logging.py:68:log_dist] [Rank -1] DeepSpeed info: version=0.7.1+a9b3bfa2, git-hash=a9b3bfa2, git-branch=master
[2022-08-22 18:06:41,060] [INFO] [logging.py:68:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups = 1
Traceback (most recent call last):
  File "/data/home/pacrook/DeepSpeed/examples/DeepSpeedExamples/inference/huggingface/gpt-neo.py", line 21, in <module>
    generator.model = deepspeed.init_inference(generator.model,
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/__init__.py", line 292, in init_inference
    engine = InferenceEngine(model,
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 140, in __init__
    self._apply_injection_policy(
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/inference/engine.py", line 333, in _apply_injection_policy
    replace_transformer_layer(
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 779, in replace_transformer_layer
    replaced_module = replace_module(model=model,
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/module_inject/replace_module.py", line 962, in replace_module
    _ = plcy(None)
  File "/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed/module_inject/replace_policy.py", line 451, in __init__
    transformers.models.opt.configuration_opt.OPTConfig):
AttributeError: module 'transformers.models' has no attribute 'opt'. Did you mean: 'dpt'?
```

**Expected behavior**
Expect gpt-neo to output inference result.

**ds_report output**
```
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_adam ............. [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
utils .................. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/torch']
torch version .................... 1.11.0
torch cuda version ............... 11.3
torch hip version ................ None
nvcc version ..................... 11.0
deepspeed install path ........... ['/data/home/pacrook/miniconda/envs/scoop_deap/lib/python3.10/site-packages/deepspeed']
deepspeed info ................... 0.7.1+a9b3bfa2, a9b3bfa2, master
deepspeed wheel compiled w. ...... torch 1.11, cuda 11.3
```

**System info (please complete the following information):**
 - OS: Ubuntu 20.04.4 LTS
 - GPU count and types: AWS, SLURM run, node with 4 x A100




Discussion:
What is exact code you used?

It work i just checked and it gives speedup of more than 2x and there are some diiferences in generation for 125M model.

Just copy the code from the example page and try it.

It need to be run from cuda, so you need CUDA_HOME. Cuda from conda if you dont have system cuda wont work. I mean cunda runtime.
I ran this example https://github.com/microsoft/DeepSpeedExamples/blob/master/inference/huggingface/gpt-neo.py
and this https://www.deepspeed.ai/tutorials/inference-tutorial/
<img width="752" alt="image" src="https://user-images.githubusercontent.com/37202747/186025622-10aa4c99-28bd-4bcc-a667-7d7dadce37ce.png">

What version of deepspeed are you using?
- It runs with `pip install deepspeed` which installs 0.7.0.  (Though output is garbled for >1 GPU.)

- It throws the above error with the latest git `master` head; 86164c48, master
```
git pull
cd DeepSpeed
./install
```
this builds 0.7.1+86164c48
@arashb ,
Can you please take a look at this issue?
Thanks,
Reza
@skiingpacman you can fix your issue by using the latest `transformers` package which supports OPT models. If you don't wish to upgrade your "transformer" package, please wait until I send a PR to make the DSPolicy class backward compatible. I will send the PR by tomorrow the latest.  
Can confirm that upgrading transformers from 4.18.0 to 4.21.1 resolves the issue for me.

@arashb thanks for looking into this so promptly.

Resolving patch:
diff --git a/deepspeed/module_inject/replace_policy.py b/deepspeed/module_inject/replace_policy.py
--- a/deepspeed/module_inject/replace_policy.py
+++ b/deepspeed/module_inject/replace_policy.py
@@ -444,13 +444,12 @@ def __init__(self, client_module, inference=True):
         try:
             import transformers
             HFOPTLayerPolicy._orig_layer_class = transformers.models.opt.modeling_opt.OPTDecoderLayer
+            if isinstance(DSPolicy.hf_model_config,
+                          transformers.models.opt.configuration_opt.OPTConfig):
+                self.pre_attn_norm = self.hf_model_config.do_layer_norm_before
         except:
             HFOPTLayerPolicy._orig_layer_class = None
 
-        if isinstance(DSPolicy.hf_model_config,
-                      transformers.models.opt.configuration_opt.OPTConfig):
-            self.pre_attn_norm = self.hf_model_config.do_layer_norm_before
-
     def get_hidden_heads(self):
         return self.client_module.self_attn.embed_dim, \
                 self.client_module.self_attn.num_heads


Test patch: