Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04.5

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary): Issue occurs on both pip binary and compiled from source

TensorFlow version (use command below): 'v1.8.0-8-g23c2187' 1.8.0

Python version: 3.5.2

Bazel version (if compiling from source): 0.16.1

GCC/Compiler version (if compiling from source): gcc (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609

CUDA/cuDNN version: CUDA 9.0, cuDNN 7.2.1.38

GPU model and memory: GeForce GTX 1080, 11178MiB

Exact command to reproduce: Cannot share the exact codebase

Describe the problem

Tensorflow attempts to swap-out a node to the CPU to save memory, even when the node is already on CPU.

This happens in core/grappler/optimizers/memory_optimizers.cc with the op _CopyFromGpuToHost.

But since this swapping is intended to save GPU memory by swapping temporarily nodes from GPU to the the CPU, the kernel for that op in core/grappler/optimizers/gpu_swapping_kernels.cc is only for GPU devices.

A suggested fix will be to simply check if the node to be swapped out is already in CPU before making it eligible for swap_out/swap_in. Note that on reducing the model size this error disappears as more memory becomes free.

Some more information confirming the issue -

The op is correctly registered in OpRegistry as shown by tensorflow::OpRegistry::Global()->DebugString(false).

The OpKernel CopyFromGpuToHostKernel is built according to bazel logs.

The OpKernel is loaded on importing tensorflow in python, verified by adding a print in that file using __attribute__((constructor)) )

Source code / logs

The relevant error is -

tensorflow.python.framework.errors_impl.NotFoundError: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_context_zoom/GatherNd_2_0 = _CopyFromGpuToHostT=DT_INT32, _class=["loc@context_zoom/GatherNd_2_0"], _device="/job:localhost/replica:0/task:0/device:CPU:0"

. Registered: device='GPU'

[[Node: swap_out_context_zoom/GatherNd_2_0 = _CopyFromGpuToHost[T=DT_INT32, _class=["loc@context_zoom/GatherNd_2_0"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_context_sents_len_0_10)

I have attached the full log here -

https://gist.github.com/akhilkedia/c13ff191e08b4462a3870a0b3a091e93

Issue 2:
Please go to Stack Overflow for help and support:

https://stackoverflow.com/questions/tagged/tensorflow

If you open a GitHub issue, here is our policy:

It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).

The form below must be filled out.

It shouldn't be a TensorBoard issue. Those go here.

Here's why we have that policy: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes. Added some custom TF operators.

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary): Binary

TensorFlow version (use command below):1.10.1

Python version:3

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:9.2

GPU model and memory: GeForce GTX 970

Exact command to reproduce:python3 train/train.py --gpu 0 --model frustum_pointnets_v2 --log_dir train/log_v2 --num_point 1024 --max_epoch 201 --batch_size 24 --decay_step 800000 --decay_rate 0.5

You can collect some of this information using our environment capture script:

https://github.com/tensorflow/tensorflow/tree/master/tools/tf_env_collect.sh

You can obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the problem

Getting an error that "No registered '_CopyFromGpuToHost' OpKernel". Not able to find the refernce to this OpKernel anywhere.

Source code / logs

pid: 16638

WARNING:tensorflow:From /home/rishi/sw/frustum-pointnets/models/pointnet_util.py:126: calling reduce_max (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.

Instructions for updating:

keep_dims is deprecated, use keepdims instead

WARNING:tensorflow:From /home/rishi/sw/frustum-pointnets/models/pointnet_util.py:212: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.

Instructions for updating:

keep_dims is deprecated, use keepdims instead

WARNING:tensorflow:From /home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py:519: calling conv1d (from tensorflow.python.ops.nn_ops) with data_format=NHWC is deprecated and will be removed in a future version.

Instructions for updating:

NHWC for data_format is deprecated, use NWC instead

2018-09-06 19:00:38.848283: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA

2018-09-06 19:00:38.920714: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero

2018-09-06 19:00:38.921052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:

name: GeForce GTX 970 major: 5 minor: 2 memoryClockRate(GHz): 1.1775

pciBusID: 0000:01:00.0

totalMemory: 3.94GiB freeMemory: 3.56GiB

2018-09-06 19:00:38.921067: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0

2018-09-06 19:00:39.115930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:

2018-09-06 19:00:39.115958: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0

2018-09-06 19:00:39.115967: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N

2018-09-06 19:00:39.116154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3282 MB memory) -> physical GPU (device: 0, name: GeForce GTX 970, pci bus id: 0000:01:00.0, compute capability: 5.2)

**** EPOCH 000 ****

2018-09-06 19:00:42.022472

2018-09-06 19:00:42.305938: W tensorflow/core/framework/allocator.cc:108] Allocation of 50331648 exceeds 10% of system memory.

2018-09-06 19:00:42.322253: W tensorflow/core/framework/allocator.cc:108] Allocation of 50331648 exceeds 10% of system memory.

2018-09-06 19:00:42.337831: W tensorflow/core/framework/allocator.cc:108] Allocation of 100663296 exceeds 10% of system memory.

2018-09-06 19:00:42.368540: W tensorflow/core/framework/allocator.cc:108] Allocation of 100663296 exceeds 10% of system memory.

2018-09-06 19:00:42.401274: W tensorflow/core/framework/allocator.cc:108] Allocation of 50331648 exceeds 10% of system memory.

2018-09-06 19:00:52.106763: E tensorflow/core/common_runtime/executor.cc:697] Executor failed to create kernel. Not found: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHostT=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"

. Registered: device='GPU'

[[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](fa_layer3/truediv_2/_91)]]

Traceback (most recent call last):

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call

return fn(*args)

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn

options, feed_dict, fetch_list, target_list, run_metadata)

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun

run_metadata)

tensorflow.python.framework.errors_impl.NotFoundError: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHostT=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"

. Registered: device='GPU'

[[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](fa_layer3/truediv_2/_91)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "train/train.py", line 376, in

train()

File "train/train.py", line 201, in train

train_one_epoch(sess, ops, train_writer)

File "train/train.py", line 256, in train_one_epoch

feed_dict=feed_dict)

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run

run_metadata_ptr)

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run

feed_dict_tensor, options, run_metadata)

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run

run_metadata)

File "/home/rishi/.local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call

raise type(e)(node_def, op, message)

tensorflow.python.framework.errors_impl.NotFoundError: No registered '_CopyFromGpuToHost' OpKernel for CPU devices compatible with node swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHostT=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"

. Registered: device='GPU'

[[Node: swap_out_gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2 = _CopyFromGpuToHost[T=DT_FLOAT, _class=["loc@gradients/fa_layer3/ThreeInterpolate_grad/ThreeInterpolateGrad_2"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](fa_layer3/truediv_2/_91)]]