Repository: pytorch/ignite

Issue:
Metrics objects are not pickleable
Pickling Metrics objects fails due to infinite loop.

The reason for that is the following:

To make metrics composable, the base class Metric has [methods](https://github.com/pytorch/ignite/blob/master/ignite/metrics/metric.py#L86) to return a `MetricsLambda` object. However, this means that pickling `Metrics` requires pickling `MetricsLambda`, but since `MetricsLambda` depends on `Metrics` we enter an infinite loop.




Discussion:
@Darktex thanks for the feedback ! A naive question about why would you like to pickle metrics ?
I'm working on making a friendly interface to DDP by having an API that looks like the following:

```
engine = DistributedEngine(engine, gpus=[0, 1, 2, 3])
engine.run()
```

The way this is implemented, `DistributedEngine` does all the DDP os.environ sync in its `__init__`, and `run` becomes just the caller for `torch.mp.spawn(train_one)`.

`train_one` needs to get a pointer to the dataloader and to spawn copies of the engine, so I want to pickle the engine to send it over :)
@Darktex thanks for the context, I see better. That's true that in my case I used `torch.distrib.launch` utility that just runs several processes as is. For example, https://github.com/pytorch/ignite/tree/distrib/examples/contrib/cifar10

In your case I imagine that there could be some cases too when pickling dataloader can be impossible if there are some lambdas involved.

If you have some ideas of how enable this, we can discuss it here. Let me investigate this too.




Couple of ideas on this one :)

- To pickle the lambdas we could simply use `dill` or `cloudpickle`. Yes, it's one more external dep but it could be optional (if the dataloader has a lambda and you didn't install the package, just raise and explain what to do)
- Ideally, I think it's best to not pass the DataLoader around. The reason is that by pickling the dataloader you have to also pickle the Dataset. Not a problem if you are on a vision dataset (it's just a pointer to a folder anyway) but a big deal if you are dealing with text or tabular data, where your dataset holds actual data (so if you have 8 GPUs you would clone it 8 times!). Even if you shard it first, sending it down the wire via Pickle takes a super long time. FairSeq has [handled the problem](https://github.com/pytorch/fairseq/commit/439ead5a7738bc5080d1d4643ae4bf6dfc78b8ca) by leveraging [Plasma](https://arrow.apache.org/docs/python/plasma.html) in Apache Arrow


I think using `dill` here would be a good idea, since this would be only optional and is a rather lightweight dependency.

On the other hand I really don't like introducing plasma here. There are a couple of reasons for that:
If you really do it in a distributed manner that you are on different workstations, there is no other option than passing around data in some way. From my point of view, this wouldn't matter much, if we send it down the wire with pickle or plasma, because the data has to be stored somewhere. Otherwise you could also add the a separate instance which would be only responsible for data loading and send single samples down the wire. But this is equally slow, no matter which backend you choose (at least that's what I think logically).

My way to go here would be to include a warning to the docs and make the user aware of this problem, that they shouldn't preload all the data (or at least do it during the first `__getitem__` or something similar), since then the pointers would have already been passed through the connection.
So far I was not able to find a real USP for having a new dependency on plasma here.
@Darktex sorry for late answer on this issue, seems like we found out the probleme with metrics, we had to `dill` them (https://github.com/pytorch/ignite/pull/884/files#diff-eb79c89d36d424b0ace62f61e7c25dd4R295).
Could you please provide the code of your `DistributedEngine`
```
engine = DistributedEngine(engine, gpus=[0, 1, 2, 3])
engine.run()
```
such that we could test our fix ? Thanks !

Resolving patch:
diff --git a/ignite/metrics/metric.py b/ignite/metrics/metric.py
--- a/ignite/metrics/metric.py
+++ b/ignite/metrics/metric.py
@@ -292,6 +292,12 @@ def __getitem__(self, index: Any):
 
         return MetricsLambda(lambda x: x[index], self)
 
+    def __getstate__(self):
+        return self.__dict__
+
+    def __setstate__(self, d):
+        self.__dict__.update(d)
+
 
 def sync_all_reduce(*attrs) -> Callable:
     def wrapper(func: Callable) -> Callable:


Test patch:
diff --git a/tests/ignite/metrics/test_dill.py b/tests/ignite/metrics/test_dill.py
new file mode 100644
--- /dev/null
+++ b/tests/ignite/metrics/test_dill.py
@@ -0,0 +1,40 @@
+import dill
+
+from ignite.metrics import Metric
+
+
+class Accumulation(Metric):
+    def __init__(self):
+        self.value = 0
+        super(Accumulation, self).__init__()
+
+    def reset(self):
+        self.value = 0
+
+    def compute(self):
+        return self.value
+
+    def update(self, output):
+        self.value += output
+
+
+def test_metric():
+    def _test(m, values, e):
+        for v in values:
+            m.update(v)
+        assert m.compute() == e
+
+    metric = Accumulation()
+
+    m1 = dill.loads(dill.dumps(metric))
+
+    values = list(range(10))
+    expected = sum(values)
+
+    _test(m1, values, expected)
+
+    metric.update(5)
+
+    m2 = dill.loads(dill.dumps(metric))
+
+    _test(m2, values, expected + 5)