Repository: microsoft/DeepSpeed

Issue:
[BUG] FlopsProfiler does not recurse when calculating module duration
**Describe the bug**
I tried to profile a finetuned [AV-HuBERT model](https://github.com/facebookresearch/av_hubert) using the FlopsProfiler. However, the code crashes with the following error:

```
Traceback (most recent call last):
  File "measure_flops_avhubert_inference.py", line 108, in <module>
    prof.print_model_profile(profile_step=profile_step)
  File "~/mambaforge/envs/avhubert/lib/python3.8/site-packages/deepspeed/profiling/flops_profiler/profiler.py", line 366, in print_model_profile
    flops_to_string(total_flops / fwd_latency)))
ZeroDivisionError: division by zero
``` 

**To Reproduce**
I don't have a script, but I think trying to profile any model with more than one level of submodules should do. For example, I was using finetuned AV-HuBERT models like [this one](https://dl.fbaipublicfiles.com/avhubert/model/lrs3/vsr/base_lrs3_30h.pt) . Other similar models listed [here](https://facebookresearch.github.io/av_hubert/).

**Expected behavior**
I believe that `get_module_duration()` should recurse into child modules just like `get_module_macs()` and `get_module_flops()` do. In other words, [this line](https://github.com/microsoft/DeepSpeed/blob/180dd39714a79b1d30bdecfe5c1b2ffe37b30152/deepspeed/profiling/flops_profiler/profiler.py#L1142) should change to:

```
duration += get_module_duration(m)
```

**ds_report output**
```
[2023-09-14 12:01:43,772] [INFO] [real_accelerator.py:158:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
fused_lamb ............. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  please install triton==1.0.0 if you want to use sparse attention
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['~/mambaforge/envs/avhubert/lib/python3.8/site-packages/torch']
torch version .................... 1.13.1
deepspeed install path ........... ['~/mambaforge/envs/avhubert/lib/python3.8/site-packages/deepspeed']
deepspeed info ................... 0.10.2, unknown, unknown
torch cuda version ............... 11.6
torch hip version ................ None
nvcc version ..................... 11.6
deepspeed wheel compiled w. ...... torch 1.13, cuda 11.6
shared memory (/dev/shm) size .... 31.24 GB
```

**Screenshots**
None.

**System info (please complete the following information):**
 - OS: Ubuntu 20.04
 - GPU count and types: one NVIDIA GeForce RTX 2080 Ti
 - Python version: 3.8.17

**Docker context**
None.

**Additional context**
None.


Discussion:

Resolving patch:
diff --git a/deepspeed/profiling/flops_profiler/profiler.py b/deepspeed/profiling/flops_profiler/profiler.py
--- a/deepspeed/profiling/flops_profiler/profiler.py
+++ b/deepspeed/profiling/flops_profiler/profiler.py
@@ -1139,7 +1139,7 @@ def get_module_duration(module):
     duration = module.__duration__
     if duration == 0:  # e.g. ModuleList
         for m in module.children():
-            duration += m.__duration__
+            duration += get_module_duration(m)
     return duration
 
 


Test patch: