TensorFlow > ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10000,32,28,28] > CPUで処理した

```txt:動作環境
Ubuntu 14.04 LTS desktop amd64
GeForce GTX 750 Ti
ASRock Z170M Pro4S [Intel Z170chipset]
TensorFlow v0.11
cuDNN v5.1 for Linux
CUDA v7.5
```

http://blog.brainpad.co.jp/entry/2016/02/25/153000
に記載のコードを実行してみた。

関連記事　http://qiita.com/ikki8412/items/d50a6200b2d317fe0a68

### watch nvidia-smi

`watch nvidia-smi`の状況は以下。
メモリが2GBなのは仕方がない。

```
Every 2.0s: nvidia-smi                                  Sun Oct  9 11:36:02 2016

Sun Oct  9 11:36:02 2016
+------------------------------------------------------+
| NVIDIA-SMI 352.63     Driver Version: 352.63         |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 750 Ti  Off  | 0000:01:00.0      On |                  N/A |
| 30%   53C    P0    34W /  38W |   1943MiB /  2047MiB |     93%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1202    G   /usr/bin/X                                     223MiB |
|    0      1894    G   compiz                                         104MiB |
|    0      2273    G   ...el=2178.0.77079571 --mojo-application-cha    84MiB |
|    0      9050    C   python                                        1520MiB |
+-----------------------------------------------------------------------------+
```

### ログの冒頭

```txt:冒頭
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: 
name: GeForce GTX 750 Ti
major: 5 minor: 0 memoryClockRate (GHz) 1.202
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.56GiB
I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 750 Ti, pci bus id: 0000:01:00.0)
Extracting MNIST_data/train-images-idx3-ubyte.gz
Extracting MNIST_data/train-labels-idx1-ubyte.gz
Extracting MNIST_data/t10k-images-idx3-ubyte.gz
Extracting MNIST_data/t10k-labels-idx1-ubyte.gz
step 0, training accuracy 0.12
step 100, training accuracy 0.86
step 200, training accuracy 0.9
```

### エラー部分 (長い)

```txt:エラー部分
step 11800, training accuracy 1
step 11900, training accuracy 0.98
step 12000, trainingI tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (256): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (512): 	Total Chunks: 1, Chunks in use: 0 768B allocated for chunks. 200B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1024): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2048): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4096): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8192): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16384): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (32768): 	Total Chunks: 1, Chunks in use: 0 36.8KiB allocated for chunks. 200B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (65536): 	Total Chunks: 1, Chunks in use: 0 78.5KiB allocated for chunks. 3.1KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (131072): 	Total Chunks: 1, Chunks in use: 0 200.0KiB allocated for chunks. 153.1KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (262144): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (524288): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (1048576): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (2097152): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (4194304): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (8388608): 	Total Chunks: 1, Chunks in use: 0 11.86MiB allocated for chunks. 390.6KiB client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (16777216): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (33554432): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (67108864): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (134217728): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:639] Bin (268435456): 	Total Chunks: 0, Chunks in use: 0 0B allocated for chunks. 0B client-requested for chunks. 0B in use in bin. 0B client-requested in use in bin.
I tensorflow/core/common_runtime/bfc_allocator.cc:656] Bin for 957.03MiB was 256.00MiB, Chunk State: 
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a40000 of size 1280
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a40500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a40600 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a40700 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a40800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a40900 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a41900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a41a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a41b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a41c00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a42900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a42a00 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a74a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x701a74b00 of size 12845056
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026b4b00 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026b5b00 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026bfb00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026bfc00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026bfd00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026bfe00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026bff00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0000 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0100 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0200 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0300 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0400 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0500 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c0a00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026c1700 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026cb700 of size 80128
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7026f2a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x702724b00 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x703332b00 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x703333b00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333db00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333dc00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333dd00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333de00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333eb00 of size 3328
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333f800 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333f900 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x70333fa00 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x703371a00 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7033a3a00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7033a3b00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7033a3c00 of size 12845056
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x703fe3c00 of size 12845056
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c23c00 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c24c00 of size 4096
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c25c00 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c2fc00 of size 40960
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c39c00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c39d00 of size 256
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x704c39e00 of size 13053184
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x7058acb00 of size 31360000
I tensorflow/core/common_runtime/bfc_allocator.cc:674] Chunk at 0x707694f00 of size 1366651136
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x7026c0600 of size 768
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x7026df000 of size 80384
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x7026f2b00 of size 204800
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x702756b00 of size 12435456
I tensorflow/core/common_runtime/bfc_allocator.cc:683] Free at 0x703334800 of size 37632
I tensorflow/core/common_runtime/bfc_allocator.cc:689]      Summary of in-use Chunks by size: 
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 31 Chunks of size 256 totalling 7.8KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 1280 totalling 1.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 5 Chunks of size 3328 totalling 16.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 5 Chunks of size 4096 totalling 20.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 4 Chunks of size 40960 totalling 160.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 80128 totalling 78.2KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 4 Chunks of size 204800 totalling 800.0KiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 3 Chunks of size 12845056 totalling 36.75MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 13053184 totalling 12.45MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 31360000 totalling 29.91MiB
I tensorflow/core/common_runtime/bfc_allocator.cc:692] 1 Chunks of size 1366651136 totalling 1.27GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:696] Sum Total of in-use chunks: 1.35GiB
I tensorflow/core/common_runtime/bfc_allocator.cc:698] Stats: 
Limit:                  1463468032
InUse:                  1450708992
MaxInUse:               1451105792
NumAllocs:                 2147898
MaxAllocSize:           1366651136

W tensorflow/core/common_runtime/bfc_allocator.cc:270] ****************************************************************************xxxxxxxxxxxxxxxxxxxxxxxx
W tensorflow/core/common_runtime/bfc_allocator.cc:271] Ran out of memory trying to allocate 957.03MiB.  See logs for memory state.
W tensorflow/core/framework/op_kernel.cc:968] Resource exhausted: OOM when allocating tensor with shape[10000,32,28,28]
 accuracy 1
step 12100, training accuracy 0.98
step 12200, training accuracy 1
step 12300, training accuracy 1
step 12400, training accuracy 0.98
step 12500, training accuracy 1
step 12600, training accuracy 1
step 12700, training accuracy 1
step 12800, training accuracy 1
step 12900, training accuracy 1
step 13000, training accuracy 0.98
step 13100, training accuracy 0.98
step 13200, training accuracy 1
step 13300, training accuracy 1
step 13400, training accuracy 1
step 13500, training accuracy 1
step 13600, training accuracy 1
step 13700, training accuracy 0.96
step 13800, training accuracy 1
step 13900, training accuracy 0.98
step 14000, training accuracy 1
step 14100, training accuracy 1
step 14200, training accuracy 0.98
step 14300, training accuracy 1
step 14400, training accuracy 1
step 14500, training accuracy 1
step 14600, training accuracy 1
step 14700, training accuracy 1
step 14800, training accuracy 1
step 14900, training accuracy 1
step 15000, training accuracy 1
step 15100, training accuracy 1
step 15200, training accuracy 1
step 15300, training accuracy 1
step 15400, training accuracy 1
step 15500, training accuracy 1
step 15600, training accuracy 1
step 15700, training accuracy 1
step 15800, training accuracy 1
step 15900, training accuracy 1
step 16000, training accuracy 1
step 16100, training accuracy 1
step 16200, training accuracy 1
step 16300, training accuracy 1
step 16400, training accuracy 1
step 16500, training accuracy 1
step 16600, training accuracy 1
step 16700, training accuracy 1
step 16800, training accuracy 1
step 16900, training accuracy 1
step 17000, training accuracy 1
step 17100, training accuracy 1
step 17200, training accuracy 1
step 17300, training accuracy 1
step 17400, training accuracy 1
step 17500, training accuracy 1
step 17600, training accuracy 1
step 17700, training accuracy 1
step 17800, training accuracy 1
step 17900, training accuracy 1
step 18000, training accuracy 1
step 18100, training accuracy 1
step 18200, training accuracy 1
step 18300, training accuracy 1
step 18400, training accuracy 1
step 18500, training accuracy 1
step 18600, training accuracy 1
step 18700, training accuracy 1
step 18800, training accuracy 1
step 18900, training accuracy 1
step 19000, training accuracy 1
step 19100, training accuracy 1
step 19200, training accuracy 1
step 19300, training accuracy 1
step 19400, training accuracy 1
step 19500, training accuracy 1
step 19600, training accuracy 1
step 19700, training accuracy 1
step 19800, training accuracy 1
step 19900, training accuracy 1
Traceback (most recent call last):
  File "sample02.py", line 124, in <module>
    main()
  File "sample02.py", line 121, in main
    x: mnist.test.images, y_: mnist.test.labels, keep_prob: 1.0}))
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 559, in eval
    return _eval_using_default_session(self, feed_dict, self.graph, session)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3761, in _eval_using_default_session
    return session.run(tensors, feed_dict)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 717, in run
    run_metadata_ptr)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 915, in _run
    feed_dict_string, options, run_metadata)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 965, in _do_run
    target_list, options, run_metadata)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 985, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.ResourceExhaustedError: OOM when allocating tensor with shape[10000,32,28,28]
	 [[Node: Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Reshape, Variable/read)]]

Caused by op u'Conv2D', defined at:
  File "sample02.py", line 124, in <module>
    main()
  File "sample02.py", line 61, in main
    h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
  File "sample02.py", line 28, in conv2d
    padding='SAME')
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
    data_format=data_format, name=name)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10000,32,28,28]
	 [[Node: Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Reshape, Variable/read)]]
```

### エラー > 抜粋


```
Caused by op u'Conv2D', defined at:
  File "sample02.py", line 124, in <module>
    main()
  File "sample02.py", line 61, in main
    h_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
  File "sample02.py", line 28, in conv2d
    padding='SAME')
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
    data_format=data_format, name=name)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/yasokada/tensorflow-GPU/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10000,32,28,28]
	 [[Node: Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Reshape, Variable/read)]]
```

### 対応案

以下のいずれか。

- このサンプルは諦める
- 12GBのTitan X (Maxwell)を中古で買う

### 実際にしたこと

GPU使用を諦めた。

コードを以下のように変更してCPU処理にした。

```
#    with tf.device("/gpu:0"):
    with tf.device("/cpu:0"):
```

これにより正常終了するようになった。

```
step 19400, training accuracy 1
step 19500, training accuracy 1
step 19600, training accuracy 1
step 19700, training accuracy 1
step 19800, training accuracy 1
step 19900, training accuracy 1
test accuracy 0.9926
```

GPUで処理できないのは、以下が関係しそう。

```
        x_image = tf.reshape(x, [-1, 28, 28, 1])
```

tensorflow.orgのTutorialによると
https://www.tensorflow.org/versions/master/tutorials/mnist/beginners/index.html

- 55,000 mnist.train
- 10,000 mnist.test
-  5,000 mnist.validation

> ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[10000,32,28,28]

のエラーではmnist.testで10,000枚の画像を扱うことにより、2GBでは足りないのだと思う。


### 補足

> step 19900, training accuracy 1

までGPUで処理できているので、ある段階以降はCPUで処理できるという実装ができれば、メモリ制約問題を緩和できそう。