Issue 1:
OS Platform and Distribution: Ubuntu 16.04

TensorFlow installed from (source or binary): binary

TensorFlow version: 1.4.1

Python version: 3.5.2

CUDA/cuDNN version: CUDA 8.0, CUDNN 7.0.5

GPU model and memory: GTX1080 (8G)

Exact command to reproduce: python3 test.py

Have I written custom code: True

Bazel version: N/A

GCC version: N/A

Repoduce code:

import tensorflow as tf

_BATCH_NORM_DECAY = 0.997
_BATCH_NORM_EPSILON = 1e-5

def two_batchnorm(inputs):
with tf.variable_scope('two_batchnorm'):
inputs = tf.layers.batch_normalization(
inputs=inputs,
axis=3,
momentum=_BATCH_NORM_DECAY,
epsilon=_BATCH_NORM_EPSILON,
center=True,
scale=True,
training=True,
fused=True)
inputs = tf.layers.batch_normalization(
inputs=inputs,
axis=3,
momentum=_BATCH_NORM_DECAY,
epsilon=_BATCH_NORM_EPSILON,
center=True,
scale=True,
training=True,
fused=True)
return inputs

inputs = tf.placeholder(tf.float32, [1, 5, 5, 3])
x = inputs
x = two_batchnorm(x)
x = two_batchnorm(x)

It'll trigger an unexpected ValueError as following:

ValueError: Variable two_batchnorm/batch_normalization/gamma already exists, disallowed.

Removing the variable scope with tf.variable_scope('two_batchnorm') in two_batchnorm will work as expected.

All variables defined in the graph should be (in creation sequense):

two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization_2/gamma:0
two_batchnorm/batch_normalization_2/beta:0
two_batchnorm/batch_normalization_2/moving_mean:0
two_batchnorm/batch_normalization_2/moving_variance:0
two_batchnorm/batch_normalization_3/gamma:0
two_batchnorm/batch_normalization_3/beta:0
two_batchnorm/batch_normalization_3/moving_mean:0
two_batchnorm/batch_normalization_3/moving_variance:0

However, with tf.layers.Layer's add_variable logics, it'll result in an unexpected value name as following (in creation sequence):

two_batchnorm/batch_normalization/gamma:0
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0
two_batchnorm/batch_normalization/gamma:0 <-- ValueError raised here.
two_batchnorm/batch_normalization/beta:0
two_batchnorm/batch_normalization/moving_mean:0
two_batchnorm/batch_normalization/moving_variance:0
two_batchnorm/batch_normalization_1/gamma:0
two_batchnorm/batch_normalization_1/beta:0
two_batchnorm/batch_normalization_1/moving_mean:0
two_batchnorm/batch_normalization_1/moving_variance:0

One solution might be using self._name to setup Layer's scope, not self._base_name.

Issue 2:
I am using tf.TFRecordReader and tf.train.batch to evaluate one epoch of my dataset (it has only one file). In order to get a better performance I tried to use enqueue_many passing a list of reader calls, because supposedly it launches several reader threads. However its behaviour has not been as I expected. It returns each item duplicated, although I thought it would return the same list of items, but faster. I took the code from @yaroslavvb github: https://github.com/yaroslavvb/stuff/blob/master/ericyue-slowreader/benchmark.py

Is this behaviour right? It seems really strange to me and I don't know if it is a bug or a feature.

Here you can see a snippet of the code:

reader = tf.TFRecordReader()
queue_batch = []
for i in range(enqueue_many_size):
_, serialized_example = reader.read(filename_queue)
queue_batch.append(serialized_example)
batch_serialized_example = tf.train.batch(
[queue_batch],
batch_size=batch_size,
num_threads=thread_number,
capacity=capacity,
enqueue_many=True)

Environment info

Operating System: Ubuntu 16.04

Nvidia Geforce 1080

Python 3.5

Tensorflow version: 1.0.0

Installed version of CUDA and cuDNN:

/usr/local/cuda-8.0/lib64/libcudadevrt.a /usr/local/cuda-8.0/lib64/libcudnn.so

/usr/local/cuda-8.0/lib64/libcudart.so /usr/local/cuda-8.0/lib64/libcudnn.so.5

/usr/local/cuda-8.0/lib64/libcudart.so.8.0 /usr/local/cuda-8.0/lib64/libcudnn.so.5.0.5

/usr/local/cuda-8.0/lib64/libcudart.so.8.0.44 /usr/local/cuda-8.0/lib64/libcudnn.so.5.1.5

/usr/local/cuda-8.0/lib64/libcudart_static.a /usr/local/cuda-8.0/lib64/libcudnn_static.a

The output for batch size of 4 records, enqueue_many_size=2, and record size of 16 floats, is:

[array([[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.,
11., 12., 13., 14., 15.],
[ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9., 10.,
11., 12., 13., 14., 15.],
[ 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
27., 28., 29., 30., 31.],
[ 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.,
27., 28., 29., 30., 31.]], dtype=float32)]