Issue 1:
Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

TensorFlow version (you are using): 2.0

Are you willing to contribute it (Yes/No): Happy to help as much as I can!

Describe the feature and the current behavior/state.

TensorFlow 1.x support specifying GPU devices to use:

# Horovod: pin GPU to be used to process local rank (one GPU per process)
config = tf.ConfigProto()
config.gpu_options.visible_device_list = str(hvd.local_rank())

There's no comparable API in TensorFlow 2.0. The closest option is to use the CUDA_VISIBLE_DEVICES environment variable. Unfortunately, CUDA_VISIBLE_DEVICES prevents processes from doing cudaMemcpy from/to devices not owned by the process. There's a significant performance degradation when NCCL is used with P2P communication disabled.

The ask is to add an API to TensorFlow 2.0 to enable device selection.

Will this change the current api? How?

Yes, will introduce an API to select GPU devices to use.

Who will benefit with this feature?

Users of Horovod.

Any Other info.

cc @azaks2 @alextp @jaingaurav @guptapriya

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):

Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):

Linux Ubuntu 16.04

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary):

My own build from source as well as google binary found at: https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-1.8.0.tar.gz

TensorFlow version (use command below):

1.8.0 - using C APi, not python

Python version:

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory:

Exact command to reproduce:

N/A, using google TF 1.8.0 cpu build, calling libtensorflow.so via C API

Describe the problem

I get valgrind reported memory leak(s), when using TF 1.8 (updating from previously used TF 1.2). Specifically, the leak comes from nsync::nsync_waiter_new_(). The valgrind output looks like this:

==1947== 400 bytes in 1 blocks are possibly lost in loss record 80,541 of 82,598 ==1947== at 0x4C2DBF6: malloc (vg_replace_malloc.c:299) ==1947== by 0xC9A6C34: nsync::nsync_waiter_new_() (in /usr/local/lib/libtensorflow_framework.so) ==1947== by 0xC9A741E: nsync::nsync_cv_wait_with_deadline_generic(nsync::nsync_cv_s_*, void*, void (*)(void*), void (*)(void*), timespec, nsync::nsync_note_s_*) (in /usr/local/lib/libtensorflow_framework.so) ==1947== by 0xC9A7AE4: nsync::nsync_cv_wait_with_deadline(nsync::nsync_cv_s_*, nsync::nsync_mu_s_*, timespec, nsync::nsync_note_s_*) (in /usr/local/lib/libtensorflow_framework.so) ==1947== by 0x73DCA4A: ??? (in /usr/local/lib/libtensorflow.so) ==1947== by 0x73DCA9A: ??? (in /usr/local/lib/libtensorflow.so) ==1947== by 0x73E228F: ??? (in /usr/local/lib/libtensorflow.so) ==1947== by 0x73EBCE4: ??? (in /usr/local/lib/libtensorflow.so) ==1947== by 0x5258D29: ??? (in /usr/local/lib/libtensorflow.so) ==1947== by 0x5259925: TF_SessionRun (in /usr/local/lib/libtensorflow.so)

I think I may have found the source by looking at the nsync code located at: https://github.com/google/nsync/blob/master/internal/common.c

I cannot observe where the memory allocated by malloc() in nsync_waiter_new_() is freed(). Furthermore, I don't see how the memory could safely be freed because the function allows for a caller-supplied allocator, which it conditionally uses, but does not seem to record which allocator was used, or have a mechanism for calling a caller-provided free().

I believe that this is a memory leak bug in Tensorflow, being caused by the nsync dependency.

Because the optimized monolithic build of libtensorflow does not seem to have debug symbols enabled by default, and because this call can occur in so many different contexts; it is difficult to generate an appropriate valgrind suppresions file which does not suppress too much.