Issue 1:
I made benchmark tests for distributed setup with loopback network, profiling it and found there is excessive memory copying in the client side of RecvTensor call, which is actually one of the bottleneck.

Here is the code, which mainly stolen from @yaroslavvb here,

with tf.device(device1):
params = tf.get_variable("params", shape=[params_size], dtype=dtype,
initializer=tf.zeros_initializer)
with tf.device(device2):
# constant node gets placed on device1 because of simple_placer
# update = tf.constant(1, shape=[params_size], dtype=dtype)
update = tf.get_variable("update", shape=[params_size], dtype=dtype,
initializer=tf.ones_initializer())
add_op = params.assign(update)

Here is the profiling result (google perftools) with tensor size 100MB (one fact is, the throughput will degrade with the increasing of tensor size):

device1 worker profiling report

device2 worker profiling report

From the result, the sending side (device2) look fine, but the receiving side (device1, the grpc client) consumes too many CPU cycles for the data transfer.

By the way, I made rough stats for this memmove call. For one round of 100MB tensor assignment, there are roughly 2GB data moved (actually, including the copy inside memmove, it should be 4GB copied with a naive memmove), which is 20+ times RAM bandwidth amplification (the result is an average of 100 round run, which may not precise but the scale should be ok).

Issue 2:
NOTE: Only file GitHub issues for bugs and feature requests. All other topics will be closed.

For general support from the community, see StackOverflow.

To make bugs and feature requests more easy to find and organize, we close issues that are deemed

out of scope for GitHub Issues and point people to StackOverflow.

For bugs or installation issues, please provide the following information.

The more information you provide, the more easily we will be able to offer

help and advice.

What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?

tf.contrib.learn.metric_spec.MetricSpec should be changed to MetricSpec in iris_monitors.py in release 1.0.0

No need to import MetricSpec in iris_monitors.py it's already imported on line 24 in commit fa4ba830f437fdb9dc1085b4d68a3bab41a16e20:

tensorflow.contrib.learn.python.learn.metric_spec import MetricSpec

Environment info

Operating System:

tensorflow/tensorflow/examples/tutorials/monitors$ lsb_release -a

No LSB modules are available.

Distributor ID:	Ubuntu

Description:	Ubuntu 16.04.1 LTS

Release:	16.04

Codename:	xenial

tensorflow/tensorflow/examples/tutorials/monitors$ uname -a

Linux panchito 4.4.0-62-generic-tuxonice #83~ppa1-Ubuntu SMP Thu Feb 2 23:17:45 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Installed version of CUDA and cuDNN:

(please attach the output of ls -l /path/to/cuda/lib/libcud*):

If installed from binary pip package, provide:

A link to the pip package you installed:

The output from python -c "import tensorflow; print(tensorflow.__version__)".

repositories/tensorflow/tensorflow/examples/tutorials/monitors$ python -c "import tensorflow; print(tensorflow.__version__)"
1.0.0

If installed from source, provide

The commit hash (git rev-parse HEAD)

The output of bazel version

If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)

repositories/tensorflow/tensorflow/examples/tutorials/monitors$ python iris_monitors.py

Traceback (most recent call last):

File "iris_monitors.py", line 116, in

tf.app.run()

File "/home/chidochipotle/anaconda3/lib/python3.5/site-packages/tensorflow/python/platform/app.py", line 44, in run

_sys.exit(main(_sys.argv[:1] + flags_passthrough))

File "iris_monitors.py", line 42, in main

tf.contrib.learn.metric_spec.MetricSpec(

AttributeError: module 'tensorflow.contrib.learn' has no attribute 'metric_spec'

What other attempted solutions have you tried?

Logs or other output that would be helpful

(If logs are large, please upload as attachment or provide link).