Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): tensorflow/benchmarks:cnn_tf_v1.12_compatible

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04-hwe

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): ('v1.12.0-0-ga6d8ffae09', '1.12.0')

Python version: 2.7.12

Bazel version (if compiling from source): N/A

GCC/Compiler version (if compiling from source): N/A

CUDA/cuDNN version: 9.0/7.2.1.38-1+cuda9.0

GPU model and memory: V100-PCIE-32GB, NVIDIA driver version 384.145, with intel_iommu=on

$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-4.15.0-38-generic root=/dev/mapper/vgroot-lvroot ro intel_iommu=on

$ nvidia-smi topo -m
GPU0	GPU1	GPU2	GPU3	CPU Affinity
GPU0	X	PIX	NODE	NODE	0-9,20-29
GPU1	PIX	X	NODE	NODE	0-9,20-29
GPU2	NODE	NODE	X	PIX	0-9,20-29
GPU3	NODE	NODE	PIX	X	0-9,20-29

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe switches (without traversing the PCIe Host Bridge)
PIX = Connection traversing a single PCIe switch
NV# = Connection traversing a bonded set of # NVLinks

Describe the current behavior

FP-16 multi-GPU training with CPU as local parameter server is converging in single process mode, but diverging loss value (nan) adding another loopback PS process with grpc. Consistent behaviours for both ResNet-50 and VGG16.

Describe the expected behavior

The same training procedure should yield exactly same result using only 1 worker with/without 1 PS. Not sure why adding SendRecvOps causes data corruption. See other info below.

Code to reproduce the issue

ResNet50 local with output:

$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--variable_update=parameter_server \
--local_parameter_device=cpu \
--model=resnet50 \
--num_gpus=4 \
--use_fp16 \
--batch_size=256

VGG16 local with output:

$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--variable_update=parameter_server \
--local_parameter_device=cpu \
--model=vgg16 \
--num_gpus=4 \
--use_fp16 \
--batch_size=256

Distributed using the same PS command with output:

CUDA_VISIBLE_DEVICES= \
python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--ps_hosts=localhost:5000 \
--worker_hosts=localhost:5001 \
--job_name=ps \
--local_parameter_device=cpu \
--task_index=0 \
--server_protocol=grpc

ResNet50 distributed with output:

$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--ps_hosts=localhost:5000 \
--worker_hosts=localhost:5001 \
--job_name=worker \
--task_index=0 \
--server_protocol=grpc \
--variable_update=parameter_server \
--local_parameter_device=cpu \
--model=resnet50 \
--num_gpus=4 \
--use_fp16 \
--batch_size=256

VGG16 distributed with output:

$ python benchmarks/scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--ps_hosts=localhost:5000 \
--worker_hosts=localhost:5001 \
--job_name=worker \
--task_index=0 \
--server_protocol=grpc \
--variable_update=parameter_server \
--local_parameter_device=cpu \
--model=vgg16 \
--num_gpus=4 \
--use_fp16 \
--batch_size=256

Other info / logs

Kernel reports [DMA Write] PTE Write access is not set for one of the GPU.

Detailed log could be found here.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04 LTS (GNU/Linux 4.4.38-rt49-tegra aarch64)

TensorFlow installed from (source or binary): source

TensorFlow version (use command below): 1.7.0-rc0, r1.7, master

Python version: 2.7.12

Bazel version (if compiling from source): 0.11.1

GCC/Compiler version (if compiling from source): 5.4.0-6ubuntu1~16.04.9

CUDA/cuDNN version: 8.0/6

GPU model and memory: name: GP106 major: 6 minor: 1 totalMemory: 3.75GiB (as reported by some previous version of Tensorflow on the other part of the same Drive PX2)

Exact command to reproduce:

git checkout v1.7.0-rc0

./configure

bazel build //tensorflow:libtensorflow_cc.so

Describe the problem

Same problem for all branches listed under TensorFlow version.

Build fails immediately with

Cuda Configuration Error: Cannot find libdevice.10.bc under /usr/local/cuda-8.0

It's strange because I explicitly answered 8.0 versions and libdevice.10.bc exists only in 9.0 AFAIK.

Source code / logs

error.log

configure.log

tf_configure.txt