Repository: mosaicml/composer

Issue:
Composer default lr scheduler create spurious warnings when max training duration is in epochs
** Environment **
<!-- Please copy paste the output of running `composer_collect_env` below-->

Collecting system information...
---------------------------------
System Environment Report        
Created: 2024-04-22 14:18:22 UTC
---------------------------------

PyTorch information
-------------------
PyTorch version: 2.2.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: 14.0.0-1ubuntu1.1
CMake version: version 3.24.1
Libc version: glibc-2.35

Python version: 3.10.14 (main, Apr  9 2024, 09:53:48) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.10.192-183.736.amzn2.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla T4
Nvidia driver version: 535.161.07
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.2
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      46 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             48
On-line CPU(s) list:                0-47
Vendor ID:                          GenuineIntel
Model name:                         Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
CPU family:                         6
Model:                              85
Thread(s) per core:                 2
Core(s) per socket:                 24
Socket(s):                          1
Stepping:                           7
BogoMIPS:                           4999.99
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke avx512_vnni
Hypervisor vendor:                  KVM
Virtualization type:                full
L1d cache:                          768 KiB (24 instances)
L1i cache:                          768 KiB (24 instances)
L2 cache:                           24 MiB (24 instances)
L3 cache:                           35.8 MiB (1 instance)
NUMA node(s):                       1
NUMA node0 CPU(s):                  0-47
Vulnerability Gather data sampling: Unknown: Dependent on hypervisor status
Vulnerability Itlb multihit:        KVM: Mitigation: VMX unsupported
Vulnerability L1tf:                 Mitigation; PTE Inversion
Vulnerability Mds:                  Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Meltdown:             Mitigation; PTI
Vulnerability Mmio stale data:      Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Retbleed:             Vulnerable
Vulnerability Spec rstack overflow: Not affected
Vulnerability Spec store bypass:    Vulnerable
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, STIBP disabled, RSB filling
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] flake8==7.0.0
[pip3] hs-infrastructure-flake8-plugins==0.2.0
[pip3] mypy==1.9.0
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] onnx==1.16.0
[pip3] onnxruntime==1.17.3
[pip3] pytorch-ranger==0.1.1
[pip3] torch==2.2.2
[pip3] torch-optimizer==0.3.0
[pip3] torchaudio==2.2.2
[pip3] torchmetrics==1.0.3
[pip3] torchvision==0.17.2
[pip3] triton==2.2.0
[conda] Could not collect


Composer information
--------------------
Composer version: 0.21.2
Composer commit hash: None
Host processor model name: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
Host processor core count: 24
Number of nodes: 1
Accelerator model name: Tesla T4
Accelerators per node: 1
CUDA Device Count: 1

<!-- if your bug is CUDA related, please also provide your CUDA version -->

** To reproduce

When launching a training with a maximum duration passed to the fit method, if the maximum duration is in TimeUnit.EPOCH
the training spams warning messages with the following shape

`Since max_duration xxep with units TimeUnit.EPOCH and t_max yyyyba with units TimeUnit.BATCH are not comparable, make sure that your LR schedule is defined at all points in the training duration.`

This comes from the method _raise_if_max_duration_exceeds_t_max because the lr scheduler converts its passed max duration with _convert_time, which is then deemed incompatible with the maximum duration in epochs. This is definitely not desirable, because it forces the max duration of the training to be set in batches if you don't want to be spammed with logs.

Steps to reproduce the behavior:

1. Have any LR scheduler from composer, with max duration in epochs (gets internally converted to batches)
2. Have a composer trainer using that scheduler
3. Fit a model with a max duration in epochs

## Expected behavior

Not having warnings if the maximum duration of the fit is in epochs.



Discussion:
Hey, will take a look today!

Resolving patch:
diff --git a/composer/optim/scheduler.py b/composer/optim/scheduler.py
--- a/composer/optim/scheduler.py
+++ b/composer/optim/scheduler.py
@@ -537,30 +537,39 @@ def _raise_if_max_duration_exceeds_t_max(t_max: Union[str, Time], state: State):
         max_dur = Time.from_timestring(max_dur)
 
     max_dur_exceeds_t_max = False
-    if t_max.unit == max_dur.unit and t_max.value < max_dur.value:
-        max_dur_exceeds_t_max = True
-    elif (
-        t_max.unit == TimeUnit.BATCH and max_dur.unit == TimeUnit.EPOCH and state.dataloader_len is not None and
-        t_max.value < max_dur.value * int(state.dataloader_len)
-    ):
-        max_dur_exceeds_t_max = True
-    elif (
-        t_max.unit == TimeUnit.EPOCH and max_dur.unit == TimeUnit.BATCH and state.dataloader_len is not None and
-        t_max.value * int(state.dataloader_len) < max_dur.value
-    ):
-        max_dur_exceeds_t_max = True
-    elif t_max.unit != max_dur.unit:
-        log.info(
-            f'Since max_duration {max_dur} with units {max_dur.unit} and t_max {t_max} with units {t_max.unit} are not '
-            'comparable, make sure that your LR schedule is defined at all points in the training duration.',
-        )
+    if t_max.unit == max_dur.unit:
+        if t_max.value >= max_dur.value:
+            # Time units are comparable, and t_max is valid.
+            return
+        else:
+            max_dur_exceeds_t_max = True
+    elif (t_max.unit == TimeUnit.BATCH and max_dur.unit == TimeUnit.EPOCH and state.dataloader_len is not None):
+        if t_max.value >= max_dur.value * int(state.dataloader_len):
+            # Batches are comparable to epochs through the dataloader length, and t_max is valid.
+            return
+        else:
+            max_dur_exceeds_t_max = True
+    elif (t_max.unit == TimeUnit.EPOCH and max_dur.unit == TimeUnit.BATCH and state.dataloader_len is not None):
+        if t_max.value * int(state.dataloader_len) >= max_dur.value:
+            # Batches are comparable to epochs through the dataloader length, and t_max is valid.
+            return
+        else:
+            max_dur_exceeds_t_max = True
 
     if max_dur_exceeds_t_max:
+        # None of the checks above passed. Time units are comparable, but t_max is invalid since it's less than max_dur.
         raise ValueError(
             f't_max {t_max} must be greater than or equal to max_duration {max_dur}. Otherwise, the LR schedule will '
             'not be defined for the entire training duration.',
         )
 
+    if t_max.unit != max_dur.unit:
+        # Units are not comparable, so we cannot check if t_max is valid. Log this and return.
+        log.debug(
+            f'Since max_duration {max_dur} with units {max_dur.unit} and t_max {t_max} with units {t_max.unit} are not '
+            'comparable, make sure that your LR schedule is defined at all points in the training duration.',
+        )
+
 
 def _raise_if_warmup_and_max_incompatible(t_warmup: Time[int], t_max: Time[int]):
     """Checks that t_warmup and t_max have the same units.


Test patch:
diff --git a/tests/optim/test_scheduler.py b/tests/optim/test_scheduler.py
--- a/tests/optim/test_scheduler.py
+++ b/tests/optim/test_scheduler.py
@@ -440,6 +440,7 @@ def test_warmup_schedulers_fail_fast(
     max_duration_no_epoch = state.max_duration
     if max_duration_unit == 'ep':
         max_duration_no_epoch = Time.from_timestring(max_duration_unit_to_str['ba'])
+
     error_context = contextlib.nullcontext()
     if (
         hasattr(scheduler, 't_max') and