Repository: GPflow/GPflow

Issue:
Reproducibility using seeds
Been running some experiments, and it seems I cannot get the same reproducible result even when setting numpy and tensorflow seeds at the top of the script like

```
np.random.seed(72)
tf.set_random_seed(72)
```

Naturally, either I'm setting them wrong or I'm missing a RNG. I call Param.randomize() and Model.optimize(), and they seem to be the only sources of randomness at the moment (and I'm not even sure about the latter) so one of them is likely the culprit. Is there any way to either set the seed appropriately without messing with the source code or maybe adding an option allowing seeds to be set in the future?


Discussion:
A shot in the dark: `Parameterized.randomize` depends on the order of parameters in `sorted_params` which, in turn, depends on the order in [`Parameterized.__dict__`](https://github.com/GPflow/GPflow/blob/master/GPflow/param.py#L790). I do not think that python makes a guarantee about the order of members in `__dict__` (maybe in python 3.6?) and therefore it might happen that while the seed is the same, `Param.randomize()` gets called in a different order each time, thus resulting in different initializations.
uh, that could definitely be problematic. But running some tests it does not seem to pose any immediate problems - I can set the seed and run randomize, quit the session, repeat the process, and get the same result. 
It almost seems like the problem occurs in `Model.predict_f`, but I don't think there's anything random going on there.
No, wait, I think you're right, it just only occurs in the lowest level of my model hierarchy! Good catch!

edit: eh, any fixes you can think of?
There is a call to `sorted` in the function I linked which uses the `id` as a key. This `id` will not be the same for every run. Thinking about it, maybe the order of items in `__dict__` does not actually matter.
Maybe the Params could be sorted by their long names instead? I guess @jameshensman will have thought about this when implementing `sorted_params`.

As a quick fix without patching GPflow: You could iterate over the Params yourself and call randomize more carefully?
seems like sorting by `Param.name` does the trick - I can submit a micro-PR if the sorting key is a bit arbitrary anyway?
hey although randomize does not use Tensorflow(?) so you're okay setting only the numpy seed I think `tf.set_random_seed` works on a graph level, so you want to call it in a particular graph context rather than at the start of the script.

More info here: https://www.tensorflow.org/api_docs/python/tf/set_random_seed
@Bonnevie , are you calling `m.randomize()`? Or are you doing `m.set_state(np.random.randn(...))`?

'long_name' should always be unique, so I don't think it will be harmful to sort by that. I originally used `id()` because it was guaranteed to be unique and very fast. If switching to long_name has no adverse effects, it might be a good choice. Maybe just check that it's no abhorrently slow though. 
@john-bradshaw: yeah, saw there were some potential issues with the TF seed, luckily it turned out I wasn't really using TF's randomness. Since GPflow takes care of graph instantiation and the like, I'm not sure what could be done to allow a seed to be set in the future. Maybe use the settings interface?
  
@jameshensman: just went with 'name', but you're right that it might be relevant to use long_name to avoid  issues with duplicates. Do you think it's going to be a lot slower because of the lexical ordering? Maybe we can use some kind of hash, since we just a fixed order and not necessarily the alphatical one. 

Agree that being able to set the tf random seed from the GPflow setting would be good. 

`name` won't work, there will be clashes: only `long_name` will be guaranteed to be unique. As for speed, remember this isn;'t happening inside an optimization loop, so so long as it 'feels' fast to the user, I'm okay. 

hmm, what function could provide a nice hash of an object? maybe `id()`? 😛 
Running a timing experiment in IPython (using %magic) of the form

```
class A():
    def __init__(self, N = 20):
        self.long_name = ''.join(random.choice(string.ascii_uppercase +
                                 string.digits) for _ in range(N))

L = [A() for _ in range(100)]
test_id = lambda: sorted(L, key=id)
test_name = lambda: sorted(L, key=lambda x: x.long_name)
%timeit test_id
%timeit test_name
```
results in timings with almost no difference, and if anything, the timings are in favor of the long_name key.

edit: e.g. one run gives 
```
10000000 loops, best of 3: 29.1 ns per loop
10000000 loops, best of 3: 24.8 ns per loop

```

Resolving patch:
diff --git a/gpflow/training/optimizer.py b/gpflow/training/optimizer.py
--- a/gpflow/training/optimizer.py
+++ b/gpflow/training/optimizer.py
@@ -43,7 +43,8 @@ def minimize(self, model, session=None, var_list=None, feed_dict=None,
     @staticmethod
     def _gen_var_list(model, var_list):
         var_list = var_list or []
-        return list(set(model.trainable_tensors).union(var_list))
+        all_vars = list(set(model.trainable_tensors).union(var_list))
+        return sorted(all_vars, key=lambda x: x.name)
 
     @staticmethod
     def _gen_feed_dict(model, feed_dict):


Test patch: