Issue 1:
Ubuntu 14.04

commit used: a154b6a

Simply changed num_gpus to 2. I have 2 GPUs. A 1060 and 970. Shows this error:

ValueError: Variable tower_1/tower_1/conv1/weight_loss/avg/biased does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?

Issue 2:
Environment info

Operating System: Ubuntu

Installed version of CUDA and cuDNN: 8.0 and 5

The commit hash (git rev-parse HEAD): 3d41cf7

The output of bazel version:

Build label: 0.3.2

Build target: bazel-out/local-fastbuild/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar

Build time: Fri Oct 7 17:25:10 2016 (1475861110)

Build timestamp: 1475861110

Build timestamp as int: 1475861110

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

python cifar10_multi_gpu_train.py --num_gpus=2

Both cifar10_train.py and cifar10_multi_gpu_train.py (without specifying num_gpus, so running on a single GPU) work.

Logs or other output that would be helpful

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally

Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.

Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.

Traceback (most recent call last):

File "cifar10_multi_gpu_train.py", line 280, in

tf.app.run()

File "/data/github/tensorflow/_python_build/tensorflow/python/platform/app.py", line 43, in run

sys.exit(main(sys.argv[:1] + flags_passthrough))

File "cifar10_multi_gpu_train.py", line 276, in main

train()

File "cifar10_multi_gpu_train.py", line 180, in train

loss = tower_loss(scope)

File "cifar10_multi_gpu_train.py", line 92, in tower_loss

loss_averages_op = loss_averages.apply(losses + [total_loss])

File "/data/github/tensorflow/_python_build/tensorflow/python/training/moving_averages.py", line 391, in apply

self._averages[var], var, decay, zero_debias=zero_debias))

File "/data/github/tensorflow/_python_build/tensorflow/python/training/moving_averages.py", line 70, in assign_moving_average

update_delta = _zero_debias(variable, value, decay)

File "/data/github/tensorflow/_python_build/tensorflow/python/training/moving_averages.py", line 177, in _zero_debias

trainable=False)

File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 1024, in get_variable

custom_getter=custom_getter)

File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 850, in get_variable

custom_getter=custom_getter)

File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 346, in get_variable

validate_shape=validate_shape)

File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 331, in _true_getter

caching_device=caching_device, validate_shape=validate_shape)

File "/data/github/tensorflow/_python_build/tensorflow/python/ops/variable_scope.py", line 650, in _get_single_variable

"VarScope?" % name)

ValueError: Variable tower_1/tower_1/conv1/weight_loss/avg/biased does not exist, or was not created with tf.get_variable(). Did you mean to set reuse=None in VarScope?