Repository: horovod/horovod

Issue:
Bug in _DistributedOptimizer: named_parameters is None and multiple param groups
**Environment:**
1. Framework: PyTorch
2. Framework version: 1.7.1
3. Horovod version: master
4. MPI version:
5. CUDA version:
6. NCCL version:
7. Python version:
8. Spark / PySpark version:
9. OS and version:
10. GCC version:
11. CMake version:

**Checklist:**
1. Did you search issues to find if somebody asked this question before?
2. If your question is about hang, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/running.rst)?
3. If your question is about docker, did you read [this doc](https://github.com/horovod/horovod/blob/master/docs/docker.rst)?
4. Did you check if you question is answered in the [troubleshooting guide](https://github.com/horovod/horovod/blob/master/docs/troubleshooting.rst)?

**Bug report:**

I have an impression that this part of code is a bit wrong if multiple param_groups are used:
https://github.com/horovod/horovod/blob/6916985c9df111f36864724e2611827f64de8e11/horovod/torch/optimizer.py#L46-L48
and if checked then for duplicates:
https://github.com/horovod/horovod/blob/6916985c9df111f36864724e2611827f64de8e11/horovod/torch/optimizer.py#L55

For example:
```python
param_groups = [
    {"params": ["a", "b", "c"]},
    {"params": ["e", "f", "g"]},    
]

named_parameters = [('allreduce.noname.%s' % i, v)
                    for param_group in param_groups
                    for i, v in enumerate(param_group['params'])]

print("named_parameters:", named_parameters)

def find_duplicates(lst):
    seen = set()
    dups = set()
    for el in lst:
        if el in seen:
            dups.add(el)
        seen.add(el)
    return dups

dups = find_duplicates([k for k, _ in named_parameters])
print("duplicates:", dups)
```
Output:
```
named_parameters: [('allreduce.noname.0', 'a'), ('allreduce.noname.1', 'b'), ('allreduce.noname.2', 'c'), ('allreduce.noname.0', 'e'), ('allreduce.noname.1', 'f'), ('allreduce.noname.2', 'g')]

duplicates: {'allreduce.noname.0', 'allreduce.noname.1', 'allreduce.noname.2'}
```

Here is a complete code snippet to repro the issue:
```python
    hvd.init()
    torch.cuda.set_device(hvd.local_rank())

    model = nn.Sequential(nn.Linear(10, 10), nn.Linear(10, 10)).cuda()

    optimizer = optim.SGD(
        [{"params": model[0].parameters()}, {"params": model[1].parameters()},],
        lr=0.001,
    )

    optimizer = hvd.DistributedOptimizer(
        optimizer,
        # named_parameters=model.named_parameters()
    )

> ValueError: Parameter names in named_parameters must be unique. Found duplicates: allreduce.noname.0, allreduce.noname.1
```





Discussion:
Thanks for the repro @vfdev-5!  I'll take a look.
@tgaddair maybe, it can be easily fixed by adding param group index like : 
```python
named_parameters = [('allreduce.noname.%s.%s' % (j, i), v)
                    for j, param_group in enumerate(param_groups)
                    for i, v in enumerate(param_group['params'])]
```
@tgaddair any updates on that ?

Resolving patch:
diff --git a/horovod/torch/optimizer.py b/horovod/torch/optimizer.py
--- a/horovod/torch/optimizer.py
+++ b/horovod/torch/optimizer.py
@@ -43,9 +43,9 @@ def __init__(self, params, named_parameters, compression,
         if named_parameters is not None:
             named_parameters = list(named_parameters)
         else:
-            named_parameters = [('allreduce.noname.%s' % i, v)
-                                for param_group in self.param_groups
-                                for i, v in enumerate(param_group['params'])]
+            named_parameters = [(f'allreduce.noname.{i}.{j}', v)
+                                for i, param_group in enumerate(self.param_groups)
+                                for j, v in enumerate(param_group['params'])]
         # make sure that named_parameters are tuples
         if any([not isinstance(p, tuple) for p in named_parameters]):
             raise ValueError('named_parameters should be a sequence of '


Test patch:
diff --git a/test/parallel/test_torch.py b/test/parallel/test_torch.py
--- a/test/parallel/test_torch.py
+++ b/test/parallel/test_torch.py
@@ -2370,6 +2370,25 @@ def check_file(fname, check_cycle=True):
 
         hvd.shutdown()
 
+    def test_optimizer_no_named_parameters(self):
+        hvd.init()
+
+        model = nn.Sequential(nn.Linear(10, 10), nn.Linear(10, 10))
+        optimizer = torch.optim.SGD(
+            [{"params": model[0].parameters()}, {"params": model[1].parameters()}, ],
+            lr=0.001,
+        )
+        optimizer = hvd.DistributedOptimizer(optimizer)
+
+        params = optimizer._parameter_names
+        self.assertEqual(len(params), len(set(params.values())))
+
+        # Make sure all workers have the same set of parameter names
+        all_param_names = hvd.allgather_object(set(params.values()))
+        self.assertEqual(len(all_param_names), hvd.size())
+        for param_names in all_param_names:
+            self.assertEqual(all_param_names[0], param_names)
+
 
 if __name__ == "__main__":
    unittest.main()