Issue 1:
#1300

I have got a nvidia p100 GPU which is support fp16, and I run the TF case 'cifar10_train.py'. Without option '--use_fp16', the performance is also 1600 examples/sec, and with the option '--use_fp16', the performance down to 500 examples/sec. Any ideas about this issue?

userid@ubuntu-WK-4xP100:/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ vi cifar10_train.py

userid@ubuntu-WK-4xP100:/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ python cifar10_train.py

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally

Downloading cifar-10-binary.tar.gz 100.0%

...

2016-11-14 00:07:57.143739: step 0, loss = 4.67 (12.1 examples/sec; 10.549 sec/batch)

2016-11-14 00:07:58.395209: step 10, loss = 4.57 (1693.6 examples/sec; 0.076 sec/batch)

2016-11-14 00:07:59.177525: step 20, loss = 4.97 (1668.9 examples/sec; 0.077 sec/batch)

2016-11-14 00:07:59.957789: step 30, loss = 4.43 (1588.3 examples/sec; 0.081 sec/batch)

2016-11-14 00:08:00.738431: step 40, loss = 4.52 (1690.5 examples/sec; 0.076 sec/batch)

2016-11-14 00:08:01.501940: step 50, loss = 4.33 (1680.4 examples/sec; 0.076 sec/batch)

2016-11-14 00:08:02.241604: step 60, loss = 4.20 (1733.4 examples/sec; 0.074 sec/batch)

2016-11-14 00:08:03.001845: step 70, loss = 4.27 (1706.0 examples/sec; 0.075 sec/batch)

2016-11-14 00:08:03.765522: step 80, loss = 4.18 (1601.3 examples/sec; 0.080 sec/batch)

2016-11-14 00:08:04.516780: step 90, loss = 4.25 (1646.3 examples/sec; 0.078 sec/batch)

userid@ubuntu-WK-4xP100:~/weike/tensorflow-r0.11/tensorflow/models/image/cifar10$ python cifar10_train.py --use_fp16

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so.8.0 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so.5 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so.8.0 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally

I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so.8.0 locally

Filling queue with 20000 CIFAR images before starting to train. This will take a few minutes.

..

2016-11-14 00:09:09.382854: step 0, loss = 4.67 (12.3 examples/sec; 10.411 sec/batch)

2016-11-14 00:09:11.923842: step 10, loss = 4.58 (659.6 examples/sec; 0.194 sec/batch)

2016-11-14 00:09:13.918448: step 20, loss = 6.62 (460.6 examples/sec; 0.278 sec/batch)

2016-11-14 00:09:16.211809: step 30, loss = 4.37 (583.7 examples/sec; 0.219 sec/batch)

2016-11-14 00:09:18.327690: step 40, loss = 4.30 (618.3 examples/sec; 0.207 sec/batch)

2016-11-14 00:09:20.395409: step 50, loss = 4.37 (643.8 examples/sec; 0.199 sec/batch)

2016-11-14 00:09:22.466230: step 60, loss = 4.32 (574.0 examples/sec; 0.223 sec/batch)

2016-11-14 00:09:24.533225: step 70, loss = 4.17 (646.2 examples/sec; 0.198 sec/batch)

2016-11-14 00:09:26.609277: step 80, loss = 2.59 (601.5 examples/sec; 0.213 sec/batch)

2016-11-14 00:09:28.703648: step 90, loss = 4.15 (625.5 examples/sec; 0.205 sec/batch)

Issue 2:
Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS High Sierra

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): 1.13.1

Python version: 2.7.14

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version: None

GPU model and memory: None

You can collect some of this information using our environment capture script

You can also obtain the TensorFlow version with

python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Describe the current behavior

I'm currently using MonitoredTrainingSession and passing it the CheckpointSaverHook. It works fine for new models. When I want to restore a model, I give a path to the checkpoint_dir arg. However, this also inits the default CheckpointSaver, and it will double save checkpoints in the directory. This is evident if I change checkpoint_basename as I get duplicate checkpoints at each global step.

Describe the expected behavior

I would not expect the default CheckpointSaver to init, and instead use the params given by the CheckpointSaverHook.

Code to reproduce the issue

...
initializer_hook = _DatasetInitializerHook(init_train, self.train_tfrecords_glob,
queue_name=self.tensorq_name)
summary_hook = tf.train.SummarySaverHook(save_steps=50, save_secs=None, summary_op=summary_op)
logging_hook = tf.train.LoggingTensorHook(tensors={'step': global_step, 'loss': loss,
'accuracy': acc_op},
every_n_iter=logging_step)
saver = tf.train.Saver(max_to_keep=2)
saver_hook = tf.train.CheckpointSaverHook(checkpoint_dir=self.ckpt_dir,
saver=saver, save_steps=save_step,
checkpoint_basename='latest_model')
step_hook = tf.train.StepCounterHook(every_n_steps=logging_step)

hooks = [summary_hook, logging_hook]

# Checkpoint Saver - Check for new model
if self.new_model:
logging.info('Starting New Model...existing ckpts will be deleted')
hooks.extend([saver_hook, step_hook])
fl = [f for f in os.listdir(self.ckpt_dir)]
for f in fl:
os.remove(os.path.join(self.ckpt_dir, f))
checkpoint_dir = None

else:
checkpoint_dir = self.ckpt_dir
hooks.append(saver_hook)

# Start Training
with tf.train.MonitoredTrainingSession(
checkpoint_dir=checkpoint_dir,
hooks=[initializer_hook],
chief_only_hooks=hooks,
log_step_count_steps=logging_step,
config=tf.ConfigProto(allow_soft_placement=True)) as mon_sess:
while not mon_sess.should_stop():
mon_sess.run(training_op)

logging.info("Training Completed")

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.