Issue 1:
System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):WINDOWS 7 64

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): source

TensorFlow version: r1.11

Python version: 3.7

Installed using virtualenv? pip? conda?: PIP

Bazel version (if compiling from source): 0.18.0

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: not using

GPU model and memory:

I cant build tensorflow because happens some error about@png_archive

C:\Users\00\tensorflow>bazel build --config=opt //tensorflow/tools/pip_package: uild_pip_package WARNING: Processed legacy workspace file c:\users\00\tensorflow/tools/bazel.rc. This file will not be processed in the next release of Bazel. Please read https //github.com/bazelbuild/bazel/issues/6319 for further information, including ho to upgrade. WARNING: Option 'experimental_shortened_obj_file_path' is deprecated Loading: Loading: 0 packages loaded INFO: Build options have changed, discarding analysis cache. Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: comando n├úo encontrado and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded) Analyzing: target //tensorflow/tools/pip_package:build_pip_package (0 packages oaded) ERROR: C:/users/00/tensorflow/tensorflow/core/platform/default/build_config/BUI D:188:1: no such package '@png_archive//': Traceback (most recent call last): File "C:/users/00/tensorflow/third_party/repo.bzl", line 106 _apply_patch(ctx, ctx.attr.patch_file) File "C:/users/00/tensorflow/third_party/repo.bzl", line 73, in _apply_ atch _execute_and_check_ret_code(ctx, cmd) File "C:/users/00/tensorflow/third_party/repo.bzl", line 52, in _execut _and_check_ret_code fail("Non-zero return code({1}) when ...)) Non-zero return code(127) when executing 'E:\MSYS2\usr\bin\bash.exe -l -c patch -p1 -d C:/users/00/_bazel_00/xjibpket/external/png_archive -i C:/users/00/tenso flow/third_party/png_fix_rpi.patch': Stdout: Stderr: /usr/bin/bash: patch: command not found and referenced by '//tensorflow/core/platform/default/build_config:png' ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' fa led; build aborted: Analysis failed INFO: Elapsed time: 9,864s INFO: 0 processes. FAILED: Build did NOT complete successfully (0 packages loaded)

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04.5 LTS

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy): N/A

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): v1.10.1-0-g4dcfddc5d1 1.10.1

Python version: 3.6.0

Bazel version (if compiling from source): N/A

CUDA/cuDNN version:

CUDA Version 9.0.176

cudnn: 7.1.4

GPU model and memory:

NVIDIA GeForce GTX 1080 Ti

11GB

Exact command to reproduce:

python test.py

Describe the problem

Description

If 2 instances of tensorflow.contrib.cudnn_rnn.CudnnLSTM are built via build() method they can not be saved together in a checkpoint.

Bug borders

If instances are built implicitly in __call__(), no exceptions are thrown. Next code executes OK.

import os
import tensorflow as tf
from tensorflow.contrib.cudnn_rnn import CudnnLSTM as CudnnLSTM
inp = tf.zeros([10, 32, 100])
lstm1 = CudnnLSTM(1, 128)
lstm2 = CudnnLSTM(2, 256)
lstm1(inp)
lstm2(inp)
saver = tf.train.Saver()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
save_path = os.path.join('test_cudnn_lstm_save', '1')
if not os.path.exists(save_path):
os.makedirs(os.path.join(save_path))
saver.save(sess, save_path)

Also if only 1 instance is built using build() or if only 1 instance is passed to saver in var_list (see exсerpt below), save op is executed without exceptions.

saver = tf.train.Saver([lstm1.saveable])

Relevance

I ran into this problem when tried to train my net on several GPUs. Shared CudnnLSTM params are built on CPU, while computations are made on GPUs.

Thank you in advance for any help!

Source code / logs

Code to reproduce the bug

import os
import tensorflow as tf
from tensorflow.contrib.cudnn_rnn import CudnnLSTM as CudnnLSTM
inp = tf.zeros([10, 32, 100])
lstm1 = CudnnLSTM(1, 128)
lstm2 = CudnnLSTM(2, 256)
lstm1.build(inp.shape)
lstm2.build(inp.shape)
saver = tf.train.Saver()
with tf.Session() as sess:
sess.run(tf.global_variables_initializer())
save_path = os.path.join('test_cudnn_lstm_save', '1')
if not os.path.exists(save_path):
os.makedirs(os.path.join(save_path))
saver.save(sess, save_path)

Here is a traceback

2018-10-04 12:59:46.556727: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-10-04 12:59:46.635329: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-10-04 12:59:46.635669: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1405] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.582
pciBusID: 0000:01:00.0
totalMemory: 10.92GiB freeMemory: 10.01GiB
2018-10-04 12:59:46.635682: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1484] Adding visible gpu devices: 0
2018-10-04 12:59:46.841898: I tensorflow/core/common_runtime/gpu/gpu_device.cc:965] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-10-04 12:59:46.841930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0
2018-10-04 12:59:46.841937: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] 0: N
2018-10-04 12:59:46.842099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9673 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2018-10-04 12:59:47.465382: W tensorflow/core/framework/op_kernel.cc:1275] OP_REQUIRES failed at save_restore_v2_ops.cc:134 : Invalid argument: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
Traceback (most recent call last):
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1278, in _do_call
return fn(*args)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
[[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "test.py", line 17, in <module>
saver.save(sess, save_path)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1620, in save
{self.saver_def.filename_tensor_name: checkpoint_file})
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 877, in run
run_metadata_ptr)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1272, in _do_run
run_metadata)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
[[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]

Caused by op 'save/SaveV2', defined at:
File "test.py", line 11, in <module>
saver = tf.train.Saver()
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1281, in __init__
self.build()
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1293, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1330, in _build
build_save=build_save, build_restore=build_restore)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 775, in _build_internal
save_tensor = self._AddSaveOps(filename_tensor, saveables)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 275, in _AddSaveOps
save = self.save_op(filename_tensor, saveables)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 193, in save_op
tensors)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1687, in save_v2
shape_and_slices=shape_and_slices, tensors=tensors, name=name)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3155, in create_op
op_def=op_def)
File "/home/anton/dpenv/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1717, in __init__
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Adding duplicate key: rnn/multi_rnn_cell/cell_0/cudnn_compatible_lstm_cell/kernel
[[Node: save/SaveV2 = SaveV2[dtypes=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/SaveV2/tensor_names, save/SaveV2/shape_and_slices, cudnn_lstm/opaque_kernel/_1, transpose/_3, concat_5/_5, cudnn_lstm_1/opaque_kernel/_7, transpose_1/_9, transpose_2/_11, concat_11/_13, concat_17/_15)]]