Issue 1:
Please make sure that this is a feature request. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:feature_template

System information

TensorFlow version (you are using): 1.12.0

Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.

The STFT Tensorflow function takes into consideration the Hermitian symmetry of the generated output i.e. it only considers the positive frequency elements. Is there a way to recover the entire pre-processed tensor that contains all the calculated elements?

In other words, is there a way to calculate the STFT of an input tensor and generate a spectrogram that has dimensions fft_length * frame_length?

Will this change the current api? How?

This will just add an additional feature to the existing function.

Who will benefit with this feature?

Anyone who is doing audio processing using Tensorflow.

Any Other info.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No, running tf_cnn_benchmarks.py from benchmarks repo

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 16.04.2 LTS

TensorFlow installed from (source or binary): Unmodified source with RDMA Verbs enabled

TensorFlow version (use command below): 1.3.0-rc0

Python version: 2.7.12

Bazel version (if compiling from source): 0.5.1

CUDA/cuDNN version: 8.0/6

GPU model and memory: NVIDIA Tesla P100 PCIe 16GB (8 per node)

Exact command to reproduce:

PS: CUDA_VISIBLE_DEVICES='' python tf_cnn_benchmarks.py --ps_hosts 12.12.12.43:20000 --worker_hosts 12.12.12.44:20000,12.12.12.41:20000 --batch_size=64 --model=inception3 --variable_update=parameter_server --local_parameter_device=cpu --job_name=ps --task_index=0 --server_protocol grpc+verbs

Worker0: CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python tf_cnn_benchmarks.py --ps_hosts 12.12.12.43:20000 --worker_hosts 12.12.12.44:20000,12.12.12.41:20000 --batch_size=64 --model=inception3 --variable_update=parameter_server --local_parameter_device=cpu --job_name=worker --task_index=0 --num_gpus=8 --data_dir=/data/imagenet_data/ --train_dir=/data/imagenet_train/ --server_protocol grpc+verbs

Worker1: CUDA_VISIBLE_DEVICES='0,1,2,3,4,5,6,7' python tf_cnn_benchmarks.py --ps_hosts 12.12.12.43:20000 --worker_hosts 12.12.12.44:20000,12.12.12.41:20000 --batch_size=64 --model=inception3 --variable_update=parameter_server --local_parameter_device=cpu --job_name=worker --task_index=1 --num_gpus=8 --data_dir=/data/imagenet_data/ --train_dir=/data/imagenet_train/ --server_protocol grpc+verbs

RDMA driver version: MLNX_OFED_LINUX-4.1-1.0.2.0

Describe the problem

When running the above commands (Inception V3 synchronized data parallelism training with 2 workers and 1 external ps), the tf_cnn_benchmarks application hangs forever after some iterations (usually in warm up).

It happens only when real data is involved (ImageNet), and with >4 GPUs. (More GPUs, less iterations before it hangs). Doesn't happen with grpc protocol, or when running with "synthetic" data.

The master_service in the workers is stuck here, which I guess means some operations in the computation have not been completed.

The RDMA protocol looks valid and clean, all messages corresponds to the protocol (see below logs).

There some tensors requested by the workers which they don't receive, but they are passed by the RDMA Verbs transport to the BaseRendezvoudMgr with RecvLocalAsync in a valid way, and for some reason the higher level worker service doesn't trigger the Send kernel on those tensors.

Any help is much appreciated!

If there are some debug mechanisms I can use to understand which tensors/operations have not been completed it can greatly help. I was mostly debugging this from the RDMA Verbs layer till now, without much success, and I feel I don't have enough information there to understand what's missing.

Also I feel we don't have enough knowledge on how the step_id acts (diving into this in the code now, but there's some higher level documentation it can greatly help).

My initial guess was an occurrence of a racy condition when loading the data, since it creates a gap in execution time (worker0 starts the first training step 30-60 seconds after worker1, since it does the preprocessing of the data twice for a reason I couldn't understand yet), but after the first iteration (which usually passes successfully) the time is synchronized between workers.

Source code / logs

Those are the logs of the runtime after moving the logging in rdma.cc to VLOG(0) (also adding Tensor name and step id for all cases, in some cases the step_id doesn't mean anything like BUFFER_REQUEST/RESPONSE for example), and also some VLOG in master_session.cc

worker0

worker1

ps

Unfortunately they are fairly large, but it's better then to cut the log files IMO.

Example for analysis I did in the verbs layer, comparing the Sent Tensor requests to the actual received tensors writes in both workers:

worker 0:

/job:ps/replica:0/task:0/cpu:0;f3c10d28b54074c0;/job:worker/replica:0/task:0/gpu:0;edge_116943_group_deps_2/NoOp_1;0:0 80661058974090965

/job:worker/replica:0/task:1/cpu:0;1a50d5c51cd9c5d1;/job:worker/replica:0/task:0/gpu:0;edge_116947_group_deps_3/NoOp_1;0:0 80661058974090965

/job:worker/replica:0/task:1/gpu:2;7f00fadabfe781f5;/job:worker/replica:0/task:0/gpu:0;edge_111078_group_deps_1/NoOp_2;0:0 80661058974090965

/job:worker/replica:0/task:1/gpu:4;b07185dd19f62088;/job:worker/replica:0/task:0/gpu:0;edge_111080_group_deps_1/NoOp_4;0:0 80661058974090965

worker 1:

/job:ps/replica:0/task:0/cpu:0;f3c10d28b54074c0;/job:worker/replica:0/task:1/cpu:0;edge_155113_AssignAdd;0:0 80661058974090965

/job:worker/replica:0/task:0/gpu:0;f3df8abf03739fe8;/job:worker/replica:0/task:1/cpu:0;edge_116948_group_deps_3;0:0 80661058974090965

The tensors requests received well by the other side and passed to RecvLocalAsync, but are not called later.

Thanks a lot.