Repository: facebookresearch/fairscale

Issue:
Add the ability to turn off trainable params changed warning
## ❓ Questions and Help
I'm posing this as a question, but it is actually a (very small) feature request.

To my understanding, when using `ShardedDDP` with `auto_refresh_trainable=True` (which is the default), I do not have to manually call `refresh_trainable()` when changing the model into `eval()` mode. Yet, while training, whenever I change the model to eval mode, to evaluate the model's validation set performance, I get this warning:

```
WARNING : root::_detect_train_change : ShardedDDP detected that the trainable params changed, either because of eval/train mode or parameter freezing/unfreeze.
```

Although I might be able to suppress any warning logs it is not what I want; I only would like to suppress this specific warning as it is superfluous and occurs every time I switch to eval model to evaluate my model during training.

Would it be possible to add a parameter to the `ShardedDDP` constructor that allows me to disable this warning?
E.g. a `warn_on_trainable_params_changed`, which could be `True` by default.

 



Discussion:
That makes sense! I suspect we can't figure out when the user has intentionally changed the trainability of params. 

Resolving patch:
diff --git a/fairscale/nn/data_parallel/sharded_ddp.py b/fairscale/nn/data_parallel/sharded_ddp.py
--- a/fairscale/nn/data_parallel/sharded_ddp.py
+++ b/fairscale/nn/data_parallel/sharded_ddp.py
@@ -63,6 +63,9 @@ class ShardedDataParallel(nn.Module):
         reduce_fp16 (bool):
             cast the grads to fp16 before reducing. Not needed if the model is already fp16, but will probably improve performance
             for multi node jobs using PyTorch AMP. The effect is similar to DDP's fp16_compress_hook_ and will also save some memory.
+        warn_on_trainable_params_changed (bool):
+            When set to False no warning will be logged whenever a parameter trainability change has been detected.
+            Default is True.
 
     .. _fp16_compress_hook: https://pytorch.org/docs/1.8.0/ddp_comm_hooks.html?highlight=fp16#torch.distributed.algorithms.ddp_comm_hooks.default_hooks.fp16_compress_hook
 
@@ -100,6 +103,7 @@ def __init__(
         reduce_buffer_size: int = 2 ** 23,
         auto_refresh_trainable: bool = True,
         reduce_fp16: bool = False,
+        warn_on_trainable_params_changed: bool = True,
     ):
         super().__init__()
 
@@ -116,6 +120,8 @@ def __init__(
                 "fp16 gradient reduction is not compatible with reduction buffers, which are requested. fp16 grad reduction is deactivated."
             )
 
+        self._warn_on_trainable_params_changed = warn_on_trainable_params_changed
+
         # Handle a no_sync() context which prevents the gradient synchronization,
         # accumulate in place
         self._should_accumulate_grads = False
@@ -654,9 +660,10 @@ def _detect_train_change(self) -> bool:
             # - the whole model is not trainable but we still have grad hooks
             trainability_changed |= not self.training and len(self._grad_hooks) > 0
 
-            if trainability_changed:
+            if self._warn_on_trainable_params_changed and trainability_changed:
                 logging.warning(
-                    "ShardedDDP detected that the trainable params changed, either because of eval/train mode or parameter freezing/unfreeze."
+                    "ShardedDDP detected that the trainable params changed, "
+                    "either because of eval/train mode or parameter freezing/unfreeze."
                 )
                 self._reference_trainable_mask = trainable_mask
 


Test patch: