Repository: NVIDIA/apex

Issue:
AttributeError: module 'torch.distributed' has no attribute 'deprecated'
Hi! 

I get this error on Windows 10 and torch=1.2.0 when just `import apex`

I think my system just not supports this, but it is not good behavior. 




Discussion:
Hi @metya,

could you post the complete stack trace, so that we can have a look, where this `deprecated` attribute is coming from?
sure! 
excuse me that i didn't think about it.

```python
>>> import apex
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "D:\metya\Anaconda3\lib\site-packages\apex\__init__.py", line 4, in <module>
    from . import parallel
  File "D:\metya\Anaconda3\lib\site-packages\apex\parallel\__init__.py", line 8, in <module>
    ReduceOp = torch.distributed.deprecated.reduce_op
AttributeError: module 'torch.distributed' has no attribute 'deprecated'
```
Thanks for letting us know!
We'll look into it.
I found this thread when getting the same error upgrading to torch=1.2.0. Seems that for some (weird) reason torch.distributed does not have attribute ReduceOp (even though documentation states it). Error goes away with downgrading to torch=1.1.0 
@asbe Do you get this error on a Linux or Windows machine?
Win10, Cuda 10.  Torch installed using the pip wheel suggested by the selector on pytorch.org 
Same here.

win10, cuda10.0, vs2017, pytorch1.2.0
Same here. 
pytorch 1.3.0 built from source, windows 10 pro, cuda 10.1, vs2019.
@metya @asbe @helson73 
Could you try to build apex from this branch and see, if this error disappears?
https://github.com/ptrblck/apex/tree/apex_no_distributed
same here,cuda10.0.1  vs2017 pytorch1.2.0 win10
D:\MachineLearning\Maskrcnn_benchMark\lib\site-packages\apex\__init__.py in <module>
      3 import warnings
      4 
----> 5 from . import parallel
      6 from . import amp
      7 from . import fp16_utils

D:\MachineLearning\Maskrcnn_benchMark\lib\site-packages\apex\parallel\__init__.py in <module>
      6     ReduceOp = torch.distributed.reduce_op
      7 else:
----> 8     ReduceOp = torch.distributed.deprecated.reduce_op
      9 
     10 from .distributed import DistributedDataParallel, Reducer

AttributeError: module 'torch.distributed' has no attribute 'deprecated'
 problem still
@tuboxin did you try to build from my branch or the current master branch?
Same error here while trying the dcgan example.

python 3.7.3, pyTorch master (torch-1.3.0a0+6ce6939), centos7, cuda10.1, cudnn 7.6.1
@mfuntowicz Could you try to build apex using this branch: https://github.com/NVIDIA/apex/issues/429#issuecomment-522604591 ? 
@ptrblck  yes,I followed your apex branch!
@ptrblck 
Hi, I have tried the branch but got following error: 

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user\Anaconda3\envs\tc1src\lib\site-packages\apex\__init__.py", line 7, in <module>
    from . import amp
  File "C:\Users\user\Anaconda3\envs\tc1src\lib\site-packages\apex\amp\__init__.py", line 1, in <module>
    from .amp import init, half_function, float_function, promote_function,\
  File "C:\Users\user\Anaconda3\envs\tc1src\lib\site-packages\apex\amp\amp.py", line 2, in <module>
    from .handle import AmpHandle, NoOpHandle
  File "C:\Users\user\Anaconda3\envs\tc1src\lib\site-packages\apex\amp\handle.py", line 11, in <module>
    from ..parallel.LARC import LARC
  File "C:\Users\user\Anaconda3\envs\tc1src\lib\site-packages\apex\parallel\__init__.py", line 8, in <module>
    ReduceOp = torch.distributed.deprecated.reduce_op
AttributeError: module 'torch.distributed' has no attribute 'deprecated'
```


@ptrblck Something went wrong with pytorch I compiled leading to distributed not being included.

It's fixed and it works as expected. Sorry for the false positive.
@ptrblck I think the error I shown above is not caused by apex. 
For some reason, pytorch I installed on my windows machine does not contain reduce_op at all. (both 1.3 complied from source or 1.2 installed from conda)
I guess for parallel usage of multi-gpus, pytorch needs NCCL, but NCCL does not support windows, so how pytorch in windows could run at multi-gpu mode at the first place? It's so weird. 
@helson73, @tuboxin, @asbe, @metya 
I've updated my branch with some more fixes. Could you try it again please?
https://github.com/ptrblck/apex/tree/apex_no_distributed
Hey @ptrblck, same setup as the others, except for python3.6
I think 
```
ReduceOp = torch.distributed.deprecated.reduce_op
``` 
will always fail in PyTorch 1.2.0, since they've removed torch.distributed.deprecated. That being said, it's unclear why torch.distributed does not have a ReduceOp method when imported.

Here's a pretty minimal check on the command line for the things you try to grab in __init__.py for apex.
![image](https://user-images.githubusercontent.com/54115240/63811158-5119e780-c8ec-11e9-81e6-df527afa12ae.png)

Looks like @helson73 may be right about it being a PyTorch problem.
This OP is missing in the PyTorch binaries for Windows, since (if I'm not mistaken) Windows does not support (some) distributed setups.
Therefore I've built PyTorch from source, manually disabling the distributed option, so that I can run into the same errors.
My current branch should guard these imports with `if torch.distributed.is_available()`, so that this operator shouldn't be visible at all.

@jacob-mink This would of course also mean that you won't be able to use the distributed package on your machine, but should be able to use other utilities like mixed precision training etc.
Let me know, if my current branch still tries to import this operator and I'll have another look.
Right, I understand. I think your current branch is still trying to import that operator. Specifically, in `apex\parallel\__init__.py` I'm seeing an attempt to create a ReduceOp variable that throws the same error.
Thanks for the information!
Could you post the whole stack trace, so that I can fix it?
@ptrblck here you go!
```
Traceback (most recent call last):
  File "train.py", line 23, in <module>
    from apex.parallel import DistributedDataParallel
  File "C:\Users\user\Anaconda3\envs\ml-pt-apex-test\lib\site-packages\apex\parallel\__init__.py", line 8, in <module>
    ReduceOp = torch.distributed.deprecated.reduce_op
AttributeError: module 'torch.distributed' has no attribute 'deprecated'
```
OK, I see.
Unfortunately, if your system does not support distributed training, you won't be able to use DDP.
Could you try to remove this import and all occurrences of DDP in your script?
Is the `apex` and `amp` import working so far?
Ah, got it. So, at that level, it's a different issue.
Now when I use your branch w/ PyTorch 1.2.0, removing all DDP, it works!
@ptrblck I installed apex from your [repository](https://github.com/ptrblck/apex/tree/apex_no_distributed). It works correctly now. Thanks.
Windows 10, Python 3.6, PyTorch 1.2, cuda10.0
@ptrblck  ~~Tried your branch, not work:~~ works!

```
Python 3.5.2 (default, Nov 12 2018, 13:43:14) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import apex
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.5/dist-packages/apex/__init__.py", line 5, in <module>
    from . import parallel
  File "/usr/local/lib/python3.5/dist-packages/apex/parallel/__init__.py", line 8, in <module>
    ReduceOp = torch.distributed.deprecated.reduce_op
AttributeError: module 'torch.distributed' has no attribute 'deprecated'
>>> 

```


I am using pytorch1.3 build from latest source.
@jinfagang [Line5](https://github.com/ptrblck/apex/blob/73f3a18257944c5250e4f1b4f3b239947d9cdad3/apex/__init__.py#L5) in my repository points to a check using `torch.distributed.is_avaibale()`, not the import which raised the error from your stack trace.
Could you make sure to use my branch and run the code again?
Sorry, it works!
If you've already installed apex, remove it:
```
pip uninstall apex
rm -rf apex
```

Reinstall from @ptrblck 's fork of apex `apex_no_distributed` branch
```
git clone https://github.com/ptrblck/apex.git
cd apex
git checkout apex_no_distributed
pip install -v --no-cache-dir ./
```

Resolving patch:
diff --git a/apex/__init__.py b/apex/__init__.py
--- a/apex/__init__.py
+++ b/apex/__init__.py
@@ -2,7 +2,9 @@
 import torch
 import warnings
 
-from . import parallel
+if torch.distributed.is_available():
+    from . import parallel
+
 from . import amp
 from . import fp16_utils
 
diff --git a/apex/amp/_initialize.py b/apex/amp/_initialize.py
--- a/apex/amp/_initialize.py
+++ b/apex/amp/_initialize.py
@@ -2,6 +2,7 @@
 from torch._six import string_classes
 import functools
 import numpy as np
+import sys
 import warnings
 from ._amp_state import _amp_state, warn_or_err, container_abcs
 from .handle import disable_casts
@@ -10,8 +11,10 @@
 from apex.fp16_utils import convert_network
 from ..fp16_utils import FP16_Optimizer as FP16_Optimizer_general
 from ..contrib.optimizers import FP16_Optimizer as FP16_Optimizer_for_fused
-from ..parallel import DistributedDataParallel as apex_DDP
-from ..parallel.LARC import LARC
+
+if torch.distributed.is_available():
+    from ..parallel import DistributedDataParallel as apex_DDP
+    from ..parallel.LARC import LARC
 
 
 def to_type(dtype, t):
@@ -62,7 +65,7 @@ def check_models(models):
         parallel_type = None
         if isinstance(model, torch.nn.parallel.DistributedDataParallel):
             parallel_type = "torch.nn.parallel.DistributedDataParallel"
-        if isinstance(model, apex_DDP):
+        if ('apex_DDP' in sys.modules) and isinstance(model, apex_DDP):
             parallel_type = "apex.parallel.DistributedDataParallel"
         if isinstance(model, torch.nn.parallel.DataParallel):
             parallel_type = "torch.nn.parallel.DataParallel"
@@ -139,11 +142,10 @@ def __call__(self, module, state_dict, prefix, local_metadata):
 
 
 def _initialize(models, optimizers, properties, num_losses=1, cast_model_outputs=None):
-    from apex.parallel import DistributedDataParallel as apex_DDP
     from .amp import init as amp_init
 
     optimizers_was_list = False
-    if isinstance(optimizers, torch.optim.Optimizer) or isinstance(optimizers, LARC):
+    if isinstance(optimizers, torch.optim.Optimizer) or ('LARC' in sys.modules and isinstance(optimizers, LARC)):
         optimizers = [optimizers]
     elif optimizers is None:
         optimizers = []
diff --git a/apex/amp/handle.py b/apex/amp/handle.py
--- a/apex/amp/handle.py
+++ b/apex/amp/handle.py
@@ -1,12 +1,15 @@
 import contextlib
 import warnings
+import sys
 import torch
 
 from . import utils
 from .opt import OptimWrapper
 from .scaler import LossScaler
 from ._amp_state import _amp_state, master_params, maybe_print
-from ..parallel.LARC import LARC
+
+if torch.distributed.is_available():
+    from ..parallel.LARC import LARC
 
 
 # There's no reason to expose the notion of a "handle". Everything can happen through amp.* calls.
@@ -84,7 +87,7 @@ def scale_loss(loss,
         yield loss
         return
 
-    if isinstance(optimizers, torch.optim.Optimizer) or isinstance(optimizers, LARC):
+    if isinstance(optimizers, torch.optim.Optimizer) or ('LARC' in sys.modules and isinstance(optimizers, LARC)):
         optimizers = [optimizers]
 
     loss_scaler = _amp_state.loss_scalers[loss_id]


Test patch: