Issue 1:
Hi,

I'm trying to run this model on a system with GeForce GTX 960M:

python /usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py

and get the following output:

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcublas.so.7.0 locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcudnn.so.6.5 locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcufft.so.7.0 locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcuda.so locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcurand.so.7.0 locally

I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 4

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 0 with properties:

name: GeForce GTX 960M

major: 5 minor: 0 memoryClockRate (GHz) 1.176

pciBusID 0000:01:00.0

Total memory: 2.00GiB

Free memory: 1.69GiB

I tensorflow/core/common_runtime/gpu/gpu_init.cc:127] DMA: 0

I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 0: Y

I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0)

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Allocating 1.50GiB bytes.

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:52] GPU 0 memory begins at 0x501a40000 extends to 0x5616a0000

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00GiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00GiB

I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 4

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity/_79 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_344_Identity", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Variable/initial_value/_81 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1185_Variable/initial_value", tensor_type=DT_INT32, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv1/biases/Assign/_67 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_304_conv1/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_2/_63 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_293_Identity_2", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_3/_55 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_275_Identity_3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv2/biases/Assign/_51 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_273_conv2/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_4/_47 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_262_Identity_4", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_5/_39 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_233_Identity_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_6/_31 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_220_Identity_6", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local3/biases/Assign/_35 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_231_local3/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_7/_23 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_191_Identity_7", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local4/biases/Assign/_19 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_189_local4/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_8/_15 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_178_Identity_8", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_9/_7 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_160_Identity_9", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: softmax_linear/biases/Assign/_3 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_158_softmax_linear/biases/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Identity_1/_71 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_306_Identity_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv1/weights/Assign/_75 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_342_conv1/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: softmax_linear/weights/Assign/_11 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_176_softmax_linear/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv1/biases/_64 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_303_conv1/biases", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv2/biases/_48 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_272_conv2/biases", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local3/biases/_32 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_230_local3/biases", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local4/biases/_16 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_188_local4/biases", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: softmax_linear/biases/_0 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_157_softmax_linear/biases", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: init/NoOp_1/_88 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1192_init/NoOp_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Variable/Assign/_84 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_1186_Variable/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv1/weights/Assign/_76 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_342_conv1/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv1/biases/Assign/_68 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_304_conv1/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv2/weights/Assign/_60 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_291_conv2/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv2/biases/Assign/_52 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_273_conv2/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local3/weights/Assign/_44 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_260_local3/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local3/biases/Assign/_36 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_231_local3/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local4/weights/Assign/_28 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_218_local4/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local4/biases/Assign/_20 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_189_local4/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: softmax_linear/weights/Assign/_12 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_176_softmax_linear/weights/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: softmax_linear/biases/Assign/_4 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_158_softmax_linear/biases/Assign", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: conv2/weights/Assign/_59 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_291_conv2/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local4/weights/Assign/_27 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_218_local4/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x322d680 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: local3/weights/Assign/_43 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_260_local3/weights/Assign", _device="/job:localhost/replica:0/task:0/cpu:0"]]

W tensorflow/core/common_runtime/executor.cc:1076] 0x32983e0 Compute status: Failed precondition: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

[[Node: Variable/initial_value/_80 = _HostSendT=DT_INT32, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_1185_Variable/initial_value", _device="/job:localhost/replica:0/task:0/gpu:0"]]

Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.

Traceback (most recent call last):

File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py", line 138, in

tf.app.run()

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/default/_app.py", line 30, in run

sys.exit(main(sys.argv))

File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py", line 134, in main

train()

File "/usr/local/lib/python2.7/dist-packages/tensorflow/models/image/cifar10/cifar10_train.py", line 94, in train

sess.run(init)

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 368, in run

results = self._do_run(target_list, unique_fetch_targets, feed_dict_string)

File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 444, in _do_run

e.code)

tensorflow.python.framework.errors.FailedPreconditionError: Attempting to use uninitialized value conv1/weights

[[Node: conv1/weights/_72 = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_341_conv1/weights", _device="/job:localhost/replica:0/task:0/cpu:0"]]

Can somebody tell me what's wrong?

The mnist model: tensorflow/models/image/mnist/convolutional.py

seems to work well:

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcublas.so.7.0 locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcudnn.so.6.5 locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcufft.so.7.0 locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcuda.so locally

I tensorflow/stream_executor/dso_loader.cc:101] successfully opened CUDA library libcurand.so.7.0 locally

I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 4

I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 0 with properties:

name: GeForce GTX 960M

major: 5 minor: 0 memoryClockRate (GHz) 1.176

pciBusID 0000:01:00.0

Total memory: 2.00GiB

Free memory: 1.71GiB

I tensorflow/core/common_runtime/gpu/gpu_init.cc:127] DMA: 0

I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 0: Y

I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 960M, pci bus id: 0000:01:00.0)

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Allocating 1.51GiB bytes.

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:52] GPU 0 memory begins at 0x501a40000 extends to 0x562600000

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.0KiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 4.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 8.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 16.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 32.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 64.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 128.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 256.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 512.00MiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 1.00GiB

I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:66] Creating bin of max chunk size 2.00GiB

I tensorflow/core/common_runtime/direct_session.cc:58] Direct session inter op parallelism threads: 4

Extracting data/train-images-idx3-ubyte.gz

Extracting data/train-labels-idx1-ubyte.gz

Extracting data/t10k-images-idx3-ubyte.gz

Extracting data/t10k-labels-idx1-ubyte.gz

Initialized!

Epoch 0.00

Minibatch loss: 12.054, learning rate: 0.010000

Minibatch error: 90.6%

Validation error: 84.6%

Epoch 0.12

Minibatch loss: 3.297, learning rate: 0.010000

Minibatch error: 6.2%

Validation error: 7.5%

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.4 x64

TensorFlow installed from (source or binary): source master and source r1.9

TensorFlow version (use command below): 1.9.0rc0 and 1.9.0rc2

Python version: 3.6.6

Bazel version (if compiling from source): 0.15.0

GCC/Compiler version (if compiling from source): 5.4.0

CUDA/cuDNN version: 9.2/7.1

GPU model and memory: GTX1080Ti 11GB X 2

Exact command to reproduce:

checkout to r1.9 or master and ./configure

after configure cuda and cudnn

and configure NCCL ( default 1.3 or 2.2, whatever I choose, same )

the process hangs forever