Repository: pytorch/ignite

Issue:
Increase flexibility of setting seed
I was running some code that has randomness during evaluation. However I found that my, supposedly, random preprocessing was always done in exactly the same way.

This is related to setting the random seed by default to 12: https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L583

An alternative mechanism, that respects the user's choice of setting a random seed is to change `seed =12` to:

```seed = torch.randint(100)```

which reads from the manual seed if it was set. 

On Slack it was suggested to use `int(time.time() * 100 % 1000)`, however this still surprises a user that wants to run the same script with two different random seeds and have both seeds reproducible.

I think it's okay if  `torch.randint` changes the random state, because many things in the dataloader already do as well.


Discussion:
Aside from using `torch.randint` there are a bunch of other options: https://pytorch.org/docs/stable/torch.html#random-sampling

Spelling out the two cases:

1. users sets no random seed: need seed from ignite for proper restarting
2. users sets random seed: ignite shouldn't overwrite using default seed, it is unexpected behaviour for a training framework. Instead it should support restarting and respect the user seed (so it can be reproduced).

if we do `torch.randint` then 1) is safeguarded, but not reproducible between runs. Also 2) is safeguarded, but we do advance the random state by sampling from `torch.randint`.

I think ignite shouldn't care about reproducibility between runs if user doesn't set a seed (model parameters are different).

@y0ast maybe we can even do something like this I think:
```python
torch.manual_seed(1)

g = torch.Generator()
g.seed()
seed = torch.randint(100, size=(1,), generator=g)

print(seed, torch.rand(1))
```
Hmm, so this solution allows us to not touch the random state for getting the initial seed. 

However in https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L616 and https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L671 you overwrite the global random state anyway, so have you really gained anything?

I would argue this is worse, because the newly set seed is random, which means any generated randomness after https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L616 is not reproducible.

> However in https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L616 and https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L671 you overwrite the global random state anyway, so have you really gained anything?

Well, actually, `self._setup_engine()` is called after `Events.STARTED` and `Events.EPOCH_STARTED` handlers where user can do something based on his random state...
Good point, we would break backwards reproducibility of scripts that use randomness in those two events.

We could check if `self.state.seed` is `None` in `setup_seed` and do `torch.randint` in there instead of in `run()`
another question is what happens in the case, when the evaluation only runs for one epoch after each training loop and is thus always seeded to 12. Is there a simple way to circumvent this?
>  Is there a simple way to circumvent this?

@CDitzel you can still pass `seed` argument to `run` function : https://pytorch.org/ignite/engine.html#ignite.engine.Engine.run
yes but if I dont want any seed?
> yes but if I dont want any seed?

@CDitzel Sorry, I do not understand what do you mean by this. Could you please details your use-case and explain why the seed being 12 is not working for you if you do not want to set a seed in run function ? Thanks 
> We could check if self.state.seed is None in setup_seed and do torch.randint in there instead of in run()

@y0ast You mean to delay the setup in where it is needed ? Why you do not like to use another generator as proposed previously ?
 So using another generator leads to any randomness used after https://github.com/pytorch/ignite/blob/master/ignite/engine/engine.py#L616 to be not reproducible, because the global manual_seed is set using the generator created seed (which is fully random).

No run will be reproducible.
@y0ast sorry, I still do not understand your point about lack of reproducibility using another generator. Let me show you an example
```python
import torch

def random_data_generator():
    while True:
        yield torch.randint(0, 100, size=(10, ))

from ignite.engine import Engine

def print_data(engine, batch):
    i = engine.state.iteration
    print(i, batch)
    

engine = Engine(print_data)

torch.manual_seed(34)

# This will be done inside `run` where instead of seed = 12
g = torch.Generator()
g.seed()
seed = torch.randint(100, size=(1,), generator=g).item()

print("seed:", seed)
print("Start the training:")
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed);

print("\nNow let's reproduce the training with that seed:")
torch.manual_seed(34)
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed);
```
It gives the following
```
seed: 28
Start the training:
1 tensor([77,  9, 33, 78, 88,  4, 11, 63, 96, 48])
2 tensor([63, 63, 96, 32, 61, 52, 32, 86, 58,  3])
3 tensor([80,  8, 91, 13, 98, 86,  9, 68, 10, 11])
4 tensor([80, 92, 88, 21,  2, 44, 68, 92,  5, 87])
5 tensor([18, 48, 71, 32, 40, 44, 85, 45, 12, 35])
6 tensor([ 0, 17, 92, 56, 21, 88, 42,  9,  9,  2])
7 tensor([73, 18,  5, 22, 38, 94,  1, 79, 96, 30])
8 tensor([25, 10,  8, 12, 46, 75, 58, 87, 81, 23])
9 tensor([65, 92, 42, 86, 15, 20, 20,  7, 31, 71])
10 tensor([37, 67, 27, 40, 21,  0, 25, 12, 61, 91])

Now let's reproduce the training with that seed:
1 tensor([77,  9, 33, 78, 88,  4, 11, 63, 96, 48])
2 tensor([63, 63, 96, 32, 61, 52, 32, 86, 58,  3])
3 tensor([80,  8, 91, 13, 98, 86,  9, 68, 10, 11])
4 tensor([80, 92, 88, 21,  2, 44, 68, 92,  5, 87])
5 tensor([18, 48, 71, 32, 40, 44, 85, 45, 12, 35])
6 tensor([ 0, 17, 92, 56, 21, 88, 42,  9,  9,  2])
7 tensor([73, 18,  5, 22, 38, 94,  1, 79, 96, 30])
8 tensor([25, 10,  8, 12, 46, 75, 58, 87, 81, 23])
9 tensor([65, 92, 42, 86, 15, 20, 20,  7, 31, 71])
10 tensor([37, 67, 27, 40, 21,  0, 25, 12, 61, 91])
```
Please hint me, why this could not work as a solution ?
 I am not sure if I am doing cross-posting here, but relating to my previous post. The issue is that usually the run method of an evaluation engine is called once every time one epoch of the training engine is finished. 

Every epoch I do some random-related business in my dataloader.

But the validation engine's seed is always reinitialized to 12 and thus the validation sequence is always the same although in the respective dataloader I do some shuffling. 

This does not affect the train dataloader, as the training engine"s run function is usually only called once and then its seed is incremented
@CDitzel let me be more explicit with an example and please comment out what is the issue with that for you.
```python
import torch
from torch.utils.data import DataLoader
from ignite.engine import Engine, Events

def random_train_data_loader(size):
    data = torch.arange(0, size)
    return DataLoader(data, batch_size=4, shuffle=True)

def random_val_data_loader(size):
    data = torch.arange(size, 2 * size)
    return DataLoader(data, batch_size=4, shuffle=True)
        
def print_train_data(engine, batch):
    i = engine.state.iteration
    e = engine.state.epoch
    print("train", e, i, batch)
    
def print_val_data(engine, batch):
    i = engine.state.iteration
    print("val", i, batch)
    
trainer = Engine(print_train_data)
evaluator = Engine(print_val_data)

@trainer.on(Events.EPOCH_COMPLETED)
def run_evaluation(_):
    seed = torch.seed() % 100
    evaluator.run(random_val_data_loader(20), seed=seed)

trainer.run(random_train_data_loader(40), max_epochs=4, epoch_length=5);
```
This gives
```
train 1 1 tensor([22, 25, 33, 30])
train 1 2 tensor([19, 18,  3, 37])
train 1 3 tensor([ 6, 38, 32, 16])
train 1 4 tensor([29, 11, 10, 27])
train 1 5 tensor([ 9, 39, 26, 20])
val 1 tensor([37, 28, 25, 27])
val 2 tensor([24, 31, 29, 38])
val 3 tensor([30, 20, 35, 36])
val 4 tensor([23, 39, 33, 22])
val 5 tensor([34, 26, 32, 21])
train 2 6 tensor([14, 17,  1,  2])
train 2 7 tensor([23, 36,  5, 31])
train 2 8 tensor([35, 24,  4, 13])
train 2 9 tensor([21,  7, 34, 28])
train 2 10 tensor([15,  8,  0, 12])
val 1 tensor([39, 27, 30, 24])
val 2 tensor([23, 22, 25, 21])
val 3 tensor([28, 35, 32, 29])
val 4 tensor([36, 26, 20, 33])
val 5 tensor([34, 38, 31, 37])
train 3 11 tensor([34, 21, 30,  7])
train 3 12 tensor([28, 29, 24,  3])
train 3 13 tensor([11,  4, 20, 35])
train 3 14 tensor([18, 16,  6, 14])
train 3 15 tensor([ 1, 17, 27, 37])
val 1 tensor([32, 31, 28, 38])
val 2 tensor([35, 20, 37, 39])
val 3 tensor([36, 29, 21, 22])
val 4 tensor([25, 26, 23, 24])
val 5 tensor([30, 27, 34, 33])
train 4 16 tensor([ 9, 23, 15, 26])
train 4 17 tensor([ 5, 19, 39, 36])
train 4 18 tensor([33, 32,  8, 22])
train 4 19 tensor([31,  2, 13,  0])
train 4 20 tensor([12, 10, 38, 25])
val 1 tensor([27, 20, 22, 26])
val 2 tensor([33, 31, 35, 32])
val 3 tensor([23, 21, 30, 39])
val 4 tensor([29, 25, 36, 37])
val 5 tensor([24, 34, 38, 28])
```

Right now, we need to set the seed for evaluator `seed = torch.seed() % 100; evaluator.run(random_val_data_loader(20), seed=seed)`, and your data coming to evaluator is randomly sampled and does not repeat between the calls. Otherwise it will be randomly sampled, but will be the same between the calls. When this issue is closed we wont need this anymore.
@vfdev-5 ah, I now see why we didn't understand each other.

In your example the manual seed of the user is not at all respected. It only looks the same because you pass `seed=seed` twice and `seed` is sampled just once.

If you append these lines:

```
print("\nNow verify that it does something different")
torch.manual_seed(42)
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed)
```

```
$ python test.py 
seed: 61
Start the training:
1 tensor([11,  6, 70, 96, 75, 50, 14, 91, 66, 85])
2 tensor([23, 53, 53, 79,  4, 35, 55,  4, 73, 66])
3 tensor([11, 95, 76, 62, 77, 10, 67, 18, 37, 26])
4 tensor([20, 54, 57, 82, 43, 98, 87, 33, 74, 67])
5 tensor([ 1, 57, 69, 46, 93, 98, 20, 74, 26, 90])
6 tensor([54, 73, 43, 64, 97, 19, 81, 57, 20, 40])
7 tensor([45, 20, 92, 93, 14, 42, 79,  4, 11, 75])
8 tensor([48,  8, 24,  0, 33, 55, 55,  2, 22, 15])
9 tensor([83, 57, 92,  5, 53,  5, 57, 94, 18, 58])
10 tensor([50, 52,  7, 30, 48, 90, 20, 55, 67, 11])

Now let's reproduce the training with that seed:
1 tensor([11,  6, 70, 96, 75, 50, 14, 91, 66, 85])
2 tensor([23, 53, 53, 79,  4, 35, 55,  4, 73, 66])
3 tensor([11, 95, 76, 62, 77, 10, 67, 18, 37, 26])
4 tensor([20, 54, 57, 82, 43, 98, 87, 33, 74, 67])
5 tensor([ 1, 57, 69, 46, 93, 98, 20, 74, 26, 90])
6 tensor([54, 73, 43, 64, 97, 19, 81, 57, 20, 40])
7 tensor([45, 20, 92, 93, 14, 42, 79,  4, 11, 75])
8 tensor([48,  8, 24,  0, 33, 55, 55,  2, 22, 15])
9 tensor([83, 57, 92,  5, 53,  5, 57, 94, 18, 58])
10 tensor([50, 52,  7, 30, 48, 90, 20, 55, 67, 11])

Now verify that it does something different
1 tensor([11,  6, 70, 96, 75, 50, 14, 91, 66, 85])
2 tensor([23, 53, 53, 79,  4, 35, 55,  4, 73, 66])
3 tensor([11, 95, 76, 62, 77, 10, 67, 18, 37, 26])
4 tensor([20, 54, 57, 82, 43, 98, 87, 33, 74, 67])
5 tensor([ 1, 57, 69, 46, 93, 98, 20, 74, 26, 90])
6 tensor([54, 73, 43, 64, 97, 19, 81, 57, 20, 40])
7 tensor([45, 20, 92, 93, 14, 42, 79,  4, 11, 75])
8 tensor([48,  8, 24,  0, 33, 55, 55,  2, 22, 15])
9 tensor([83, 57, 92,  5, 53,  5, 57, 94, 18, 58])
10 tensor([50, 52,  7, 30, 48, 90, 20, 55, 67, 11])
```

It has the same output as the other seeds! This isn't correct and comes from the call to `manual_seed` inside ignite.

if instead you would have resampled from the generator between run1 and run2, then the results would be different for 1 and 2 (while user sets same manual seed) and same for 2 and 3 (while user sets different seed)!

This is I think the opposite of what we want :)

```
seed: 85
Start the training:
1 tensor([26, 80, 81, 64, 59, 17, 84,  3,  5, 43])
2 tensor([ 2, 89, 48, 29, 69, 47, 91, 22, 79, 99])
3 tensor([99, 78, 46, 40, 15, 44, 97, 21, 63, 41])
4 tensor([89, 73, 14, 35, 53, 68, 67, 82, 80, 25])
5 tensor([34, 54, 82, 28,  1, 61, 20, 31, 11,  2])
6 tensor([42, 69,  7,  6, 63, 33, 39, 56, 56, 88])
7 tensor([94, 61, 95, 17, 48, 20, 80, 99, 71, 64])
8 tensor([93, 77, 48, 43,  3,  6, 54, 47,  0, 31])
9 tensor([34, 16, 98, 29, 37, 54, 96, 38, 21, 64])
10 tensor([ 5, 20, 54, 77, 98,  5, 42, 98, 30, 31])

Now let's reproduce the training with that manual seed (but resampled generator):
1 tensor([ 1, 17, 50, 18, 23, 91, 16, 85, 41, 90])
2 tensor([13, 79, 34, 21, 36, 52, 77, 26, 72, 30])
3 tensor([35, 28,  6, 69, 68, 76, 30, 63,  3, 90])
4 tensor([60, 36,  7,  8, 58, 82,  5, 26, 91, 29])
5 tensor([78, 82, 93, 77, 51, 36,  7,  2, 10, 93])
6 tensor([92, 51, 47,  9, 28, 87, 60, 27, 48, 82])
7 tensor([47, 42, 90,  0, 47, 78, 80, 12, 41, 50])
8 tensor([80,  2, 29,  3, 78, 30, 15, 42, 94, 39])
9 tensor([51, 57,  7, 57, 67,  3, 35,  6, 90, 16])
10 tensor([26, 47, 40, 14, 28, 99, 65, 71, 51, 80])

Now verify that it does something different
1 tensor([ 1, 17, 50, 18, 23, 91, 16, 85, 41, 90])
2 tensor([13, 79, 34, 21, 36, 52, 77, 26, 72, 30])
3 tensor([35, 28,  6, 69, 68, 76, 30, 63,  3, 90])
4 tensor([60, 36,  7,  8, 58, 82,  5, 26, 91, 29])
5 tensor([78, 82, 93, 77, 51, 36,  7,  2, 10, 93])
6 tensor([92, 51, 47,  9, 28, 87, 60, 27, 48, 82])
7 tensor([47, 42, 90,  0, 47, 78, 80, 12, 41, 50])
8 tensor([80,  2, 29,  3, 78, 30, 15, 42, 94, 39])
9 tensor([51, 57,  7, 57, 67,  3, 35,  6, 90, 16])
10 tensor([26, 47, 40, 14, 28, 99, 65, 71, 51, 80])
```
@y0ast I think this is a correct behaviour : 
```python
torch.manual_seed(42)
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed)
```
as IMO it is Engine's role to setup correct random state once run is called. And yes, `torch.manual_seed(42)` is just ignored.


> if instead you would have resampled from the generator between run1 and run2

What is exact code to produce the last snippet, please ?
See below for the script.

I think seeding the generator using the user's manual seed would be an option, which is very similar to my proposed `torch.randint(100)` solution but fixed for the epoch/iteration started events.

```
import torch
from ignite.engine import Engine


def random_data_generator():
    while True:
        yield torch.randint(0, 100, size=(10,))


def print_data(engine, batch):
    i = engine.state.iteration
    print(i, batch)


engine = Engine(print_data)

torch.manual_seed(34)

# This will be done inside `run` where instead of seed = 12
g = torch.Generator()
g.seed()
seed = torch.randint(100, size=(1,), generator=g).item()

print("seed:", seed)
print("Start the training:")
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed)

g = torch.Generator()
g.seed()
seed = torch.randint(100, size=(1,), generator=g).item()

print("\nNow let's reproduce the training with that seed:")
torch.manual_seed(34)
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed)

print("\nNow verify that it does something different")
torch.manual_seed(42)
engine.run(random_data_generator(), max_epochs=2, epoch_length=5, seed=seed)
```
In this code, after the run1 you regenerated another seed which was then used with the Engine. So, it is obvious that run2 and run3 wont give the same results as run1.


I understand, however consider a normal users workflow with your proposed generator change:

script.py trains a model, the user sets a manual seed and gets a particular output

the user reruns script.py to train a new model, with the same manual seed, now that user gets a different output.

The difference comes from the generator inside `run` which produced a different seed in both occasions.

Instead, that generator should be seeded from the global random state. In that case the user will get the *same* output twice, as expected.
According to our slack discussion with @y0ast 
It would be nice to have the following behaviour:
1) without setting a seed to `run`
```python
torch.manual_seed(123);
engine.run(random_data_generator(), max_epochs=2, epoch_length=5);
```
have reproducible run if everthing is executed multiple times. (still the case now)
2) randomness to be controlled by `torch.manual_seed(123)`
```python
torch.manual_seed(234);
engine.run(random_data_generator(), max_epochs=2, epoch_length=5);
```
has same reproducibility as in 1) but all random dataflow etc is different.

Joost's solution is to simply setup inside `run`
```python
if seed is None:
     seed = torch.randint(100, size=(1,)).item()
```
covers the requirements.



Resolving patch:
diff --git a/ignite/engine/engine.py b/ignite/engine/engine.py
--- a/ignite/engine/engine.py
+++ b/ignite/engine/engine.py
@@ -564,8 +564,8 @@ def run(
             epoch_length (int, optional): Number of iterations to count as one epoch. By default, it can be set as
                 `len(data)`. If `data` is an iterator and `epoch_length` is not set, an error is raised.
                 This argument should be `None` if run is resuming from a state.
-            seed (int, optional): Seed to setup at each epoch for reproducible runs.
-                This argument should be `None` if run is resuming from a state.
+            seed (int, optional): Seed to use for dataflow consistency, by default it
+                will respect the global random state. This argument should be `None` if run is resuming from a state.
 
         Returns:
             State: output state.
@@ -594,7 +594,7 @@ def switch_batch(engine):
             if max_epochs is None:
                 max_epochs = 1
             if seed is None:
-                seed = 12
+                seed = torch.randint(0, int(1e9), (1,)).item()
             if epoch_length is None:
                 if hasattr(data, "__len__"):
                     epoch_length = len(data)


Test patch:
diff --git a/tests/ignite/engine/test_engine.py b/tests/ignite/engine/test_engine.py
--- a/tests/ignite/engine/test_engine.py
+++ b/tests/ignite/engine/test_engine.py
@@ -6,6 +6,7 @@
 import torch
 
 from ignite.engine import Engine, Events, State
+from ignite.metrics import Average
 
 
 def test_terminate():
@@ -529,3 +530,35 @@ def foo(e, b):
     engine.run(data_loader, epoch_length=10, max_epochs=5)
 
     assert counter[0] == 50
+
+
+def test_engine_random_state():
+    def random_data_generator():
+        while True:
+            yield torch.randint(0, 100, size=(5,))
+
+    def sum_data(engine, batch):
+        result = torch.sum(batch)
+        return result
+
+    def get_engine():
+        engine = Engine(sum_data)
+        average = Average()
+        average.attach(engine, "average")
+        return engine
+
+    torch.manual_seed(34)
+    engine = get_engine()
+    state1 = engine.run(random_data_generator(), max_epochs=2, epoch_length=2)
+
+    torch.manual_seed(34)
+    engine = get_engine()
+    state2 = engine.run(random_data_generator(), max_epochs=2, epoch_length=2)
+
+    torch.manual_seed(42)
+    engine = get_engine()
+    state3 = engine.run(random_data_generator(), max_epochs=2, epoch_length=2)
+
+    assert state1.metrics["average"] == pytest.approx(state2.metrics["average"])
+    assert state1.metrics["average"] != pytest.approx(state3.metrics["average"])
+    assert state2.metrics["average"] != pytest.approx(state3.metrics["average"])