Repository: microsoft/DeepSpeed

Issue:
[BUG] Wrong time unit in flops profiler
**Describe the bug**
The elapsed time in `flops_profiler` gets by `_get_elapsed_msec` is msec now, introducing many time unit inconsistencies. #2090 and its fix #2095, and #1934 have resolved some of the misuses. 

I mainly examined the MoE part and found the remaining below. 

1. The flop profiler log print the duration as ms
https://github.com/microsoft/DeepSpeed/blob/fda63432ba67643d2f54885e45f0b2058fa2b937/deepspeed/runtime/engine.py#L1714-L1716
However it is captured and *1000 manually
https://github.com/microsoft/DeepSpeed/blob/fda63432ba67643d2f54885e45f0b2058fa2b937/deepspeed/moe/sharded_moe.py#L434

2. The time breakdown log needs ms here
https://github.com/microsoft/DeepSpeed/blob/fda63432ba67643d2f54885e45f0b2058fa2b937/deepspeed/runtime/engine.py#L2095-L2097


Similar issues: #2090 #1934


[BUG] Wrong time unit in flops profiler
**Describe the bug**
The elapsed time in `flops_profiler` gets by `_get_elapsed_msec` is msec now, introducing many time unit inconsistencies. #2090 and its fix #2095, and #1934 have resolved some of the misuses. 

I mainly examined the MoE part and found the remaining below. 

1. The flop profiler log print the duration as ms
https://github.com/microsoft/DeepSpeed/blob/fda63432ba67643d2f54885e45f0b2058fa2b937/deepspeed/runtime/engine.py#L1714-L1716
However it is captured and *1000 manually
https://github.com/microsoft/DeepSpeed/blob/fda63432ba67643d2f54885e45f0b2058fa2b937/deepspeed/moe/sharded_moe.py#L434

2. The time breakdown log needs ms here
https://github.com/microsoft/DeepSpeed/blob/fda63432ba67643d2f54885e45f0b2058fa2b937/deepspeed/runtime/engine.py#L2095-L2097


Similar issues: #2090 #1934




Discussion:

Resolving patch:
diff --git a/deepspeed/moe/sharded_moe.py b/deepspeed/moe/sharded_moe.py
--- a/deepspeed/moe/sharded_moe.py
+++ b/deepspeed/moe/sharded_moe.py
@@ -431,7 +431,7 @@ def forward(
 
         if self.wall_clock_breakdown:
             self.timers('TopKGate').stop()
-            self.gate_time = self.timers('TopKGate').elapsed(reset=False) * 1000
+            self.gate_time = self.timers('TopKGate').elapsed(reset=False)
 
         return gate_output
 
@@ -535,7 +535,7 @@ def forward(self, *input: Tensor, **kwargs: Any) -> Tensor:
 
         if self.wall_clock_breakdown:
             self.timers('falltoall').stop()
-            self.time_falltoall = self.timers('falltoall').elapsed(reset=False) * 1000
+            self.time_falltoall = self.timers('falltoall').elapsed(reset=False)
 
         # Re-shape after all-to-all: ecm -> gecm
         dispatched_input = dispatched_input.reshape(self.ep_size,
@@ -552,7 +552,7 @@ def forward(self, *input: Tensor, **kwargs: Any) -> Tensor:
 
         if self.wall_clock_breakdown:
             self.timers('salltoall').stop()
-            self.time_salltoall = self.timers('salltoall').elapsed(reset=False) * 1000
+            self.time_salltoall = self.timers('salltoall').elapsed(reset=False)
 
         # Re-shape back: gecm -> ecm
         expert_output = expert_output.reshape(self.ep_size * self.num_local_experts,
@@ -576,6 +576,6 @@ def forward(self, *input: Tensor, **kwargs: Any) -> Tensor:
 
         if self.wall_clock_breakdown:
             self.timers('moe').stop()
-            self.time_moe = self.timers('moe').elapsed(reset=False) * 1000
+            self.time_moe = self.timers('moe').elapsed(reset=False)
 
         return a
diff --git a/deepspeed/runtime/engine.py b/deepspeed/runtime/engine.py
--- a/deepspeed/runtime/engine.py
+++ b/deepspeed/runtime/engine.py
@@ -2058,8 +2058,7 @@ def step(self, lr_kwargs=None):
                     self._write_monitor()
 
                 if self.has_moe_layers:
-                    fwd_time = self.timers(FORWARD_GLOBAL_TIMER).elapsed(
-                        reset=False) * 1000
+                    fwd_time = self.timers(FORWARD_GLOBAL_TIMER).elapsed(reset=False)
                     self.print_forward_breakdown(fwd_time=fwd_time)
 
                 self.timers.log(self.engine_timers.global_timers)
@@ -2103,29 +2102,27 @@ def _write_monitor(self):
             self.summary_events = [
                 (
                     f"Train/Samples/elapsed_time_ms_forward",
-                    self.timers(FORWARD_GLOBAL_TIMER).elapsed(reset=False) * 1000.0,
+                    self.timers(FORWARD_GLOBAL_TIMER).elapsed(reset=False),
                     self.global_samples,
                 ),
                 (
                     f"Train/Samples/elapsed_time_ms_backward",
-                    self.timers(BACKWARD_GLOBAL_TIMER).elapsed(reset=False) * 1000.0,
+                    self.timers(BACKWARD_GLOBAL_TIMER).elapsed(reset=False),
                     self.global_samples,
                 ),
                 (
                     f"Train/Samples/elapsed_time_ms_backward_inner",
-                    self.timers(BACKWARD_INNER_GLOBAL_TIMER).elapsed(reset=False) *
-                    1000.0,
+                    self.timers(BACKWARD_INNER_GLOBAL_TIMER).elapsed(reset=False),
                     self.global_samples,
                 ),
                 (
                     f"Train/Samples/elapsed_time_ms_backward_allreduce",
-                    self.timers(BACKWARD_REDUCE_GLOBAL_TIMER).elapsed(reset=False) *
-                    1000.0,
+                    self.timers(BACKWARD_REDUCE_GLOBAL_TIMER).elapsed(reset=False),
                     self.global_samples,
                 ),
                 (
                     f"Train/Samples/elapsed_time_ms_step",
-                    self.timers(STEP_GLOBAL_TIMER).elapsed(reset=False) * 1000.0,
+                    self.timers(STEP_GLOBAL_TIMER).elapsed(reset=False),
                     self.global_samples,
                 ),
             ]


Test patch: