Issue 1:
System information

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 10.13.4

TensorFlow installed from (source or binary): Binary

TensorFlow version (use command below): v1.8.0-0-g93bc2e2072 1.8.0

Python version: 3.5.4

Bazel version (if compiling from source): 0.11.1

GCC/Compiler version (if compiling from source): 9.1.0

CUDA/cuDNN version: N/A

GPU model and memory: N/A

Exact command to reproduce: N/A

Describe the problem

I'm running two versions of a NN which I expect to produce identical results, but they don't.

The first version contains as part of the computation:

x = [0.0*a + 1.0*b, 1.0*a + 0.0*b]

The second contains instead:

x = [b, a]

The two runs produce different results. (a and b are both matrices).

I'm using a constant seed for the random noise input, and I've verified that multiple runs of either of the two above cases produce identical results. Just changing from one case to the other produces different results. The results are not broken in either case (i.e. both cases produce reasonable output), just different.

This seems like a bug to me.

Thanks

EDIT:

I've also found that doing:

x = 1.0 * a

x = 1.0 * (1.0 * a)

x = tf.identity(a)

All give different results.

Looking at this: #14675

Apparently the problem is I've made a change to the computation graph.

I still think this is a bug so I'm going to leave this here. It's very frustrating when obvious things don't work as expected.

Issue 2:
I'm running TensorFlow 1.2 with CUDA.

Error case:

I call tf.set_random_seed(2017) and then build a graph g1 that includes trainable variables and an optimizer. I create a session, run the tf.global_variables_initializer() op, and then immediately fetch the value of a scalar variable (without running any train steps, so this is the initial value of the variable). As expected, this value stays the same if I launch this process multiple times.

I then build a graph g2 that is identical to g1 except for a slight change. g1 contained mu = tf.reduce_sum(x) / tf.size(x)[0], and g2 contains mu = tf.reduce_mean(x). g2 is seeded the same way as g1 and has the same variable names and shapes as g1. The only differently named tensors are those relating to the modification mentioned above. When I fetch the initial value of the same scalar variable from g2, there is a completely different value from when fetched from g1.

I've tried to isolate this into a small test case but have not been successful yet. I will continue to work on this. Apologies for bug report without test case.

My current intention is to workaround this with a Numpy based initialization scheme.

Questions:

(1) Is this expected behavior? Ideally the variables would be initialized in the same way to help make results more reproducible. In my case, the different variable initialization makes it more difficult to test that g1 and g2 produce the same values. If variables were initialized the same way, would be easy to see that refactoring the mean computation in the graph did not break anything.

(2) Any idea why this occurs? Perhaps relevantly, tf.make_template is used within this graph. My current (evidence-free) hunch is the small change in graph causes a variable to move from CPU resident to GPU resident and caused a different PRNG kernel (provided with the same seed) to be used.