Issue 1:
With this issue I would like to track the efforts in integrating the

cudnn library within tensorflow.

As of June 17th 2016 doing a manual grep on the repository gives these

functions as being mapped from cudnn to the stream executor,

From chapter 4 of the cudnn User Guide version 5.0 (April 2016):

cudnnGetVersion

cudnnGetErrorString

cudnnCreate

cudnnDestroy

cudnnSetStream

cudnnGetStream

cudnnCreateTensorDescriptor

cudnnSetTensor4dDescriptor

cudnnSetTensor4dDescriptorEx

cudnnGetTensor4dDescriptor

cudnnSetTensorNdDescriptor

cudnnDestroyTensorDescriptor

cudnnTransformTensor

cudnnAddTensor

cudnnOpTensor

cudnnSetTensor

cudnnScaleTensor

cudnnCreateFilterDescriptor

cudnnSetFilter4dDescriptor

cudnnGetFilter4dDescriptor

cudnnSetFilter4dDescriptor_v3 (versioned)

cudnnGetFilter4dDescriptor_v3 (versioned)

cudnnSetFilter4dDescriptor_v4 (versioned)

cudnnGetFilter4dDescriptor_v4 (versioned)

cudnnSetFilterNdDescriptor

cudnnGetFilterNdDescriptor

cudnnGetFilterNdDescriptor_v3 (versioned)

cudnnGetFilterNdDescriptor_v3 (versioned)

cudnnGetFilterNdDescriptor_v4 (versioned)

cudnnGetFilterNdDescriptor_v4 (versioned)

cudnnDestroyFilterDescriptor

cudnnCreateConvolutionDescriptor

cudnnSetConvolution2dDescriptor

cudnnGetConvolution2dDescriptor

cudnnGetConvolution2dForwardOutputDim

cudnnSetConvolutionNdDescriptor

cudnnGetConvolutionNdDescriptor

cudnnGetConvolutionNdForwardOutputDim

cudnnDestroyConvolutionDescriptor

cudnnFindConvolutionForwardAlgorithm

cudnnFindConvolutionForwardAlgorithmEx

cudnnGetConvolutionForwardAlgorithm

cudnnGetConvolutionForwardWorkspaceSize

cudnnConvolutionForward

cudnnConvolutionBackwardBias

cudnnFindConvolutionBackwardFilterAlgorithm

cudnnFindConvolutionBackwardFilterAlgorithmEx

cudnnGetConvolutionBackwardFilterAlgorithm

cudnnGetConvolutionBackwardFilterWorkspaceSize

cudnnConvolutionBackwardFilter

cudnnFindConvolutionBackwardDataAlgorithm

cudnnFindConvolutionBackwardDataAlgorithmEx

cudnnGetConvolutionBackwardDataAlgorithm

cudnnGetConvolutionBackwardDataWorkspaceSize

cudnnConvolutionBackwardData

cudnnSoftmaxForward

cudnnSoftmaxBackward

cudnnCreatePoolingDescriptor

cudnnSetPooling2dDescriptor

cudnnGetPooling2dDescriptor

cudnnSetPoolingNdDescriptor

cudnnGetPoolingNdDescriptor

cudnnSetPooling2dDescriptor_v3 (versioned)

cudnnGetPooling2dDescriptor_v3 (versioned)

cudnnSetPoolingNdDescriptor_v3 (versioned)

cudnnGetPoolingNdDescriptor_v3 (versioned)

cudnnSetPooling2dDescriptor_v4 (versioned)

cudnnGetPooling2dDescriptor_v4 (versioned)

cudnnSetPoolingNdDescriptor_v4 (versioned)

cudnnGetPoolingNdDescriptor_v4 (versioned)

cudnnDestroyPoolingDescriptor

cudnnGetPooling2dForwardOutputDim

cudnnGetPoolingNdForwardOutputDim

cudnnPoolingForward

cudnnPoolingBackward

cudnnActivationForward

cudnnActivationBackward

cudnnCreateActivationDescriptor

cudnnSetActivationDescriptor

cudnnGetActivationDescriptor

cudnnDestroyActivationDescriptor

cudnnActivationForward_v3 (versioned)

cudnnActivationBackward_v3 (versioned)

cudnnActivationForward_v4 (versioned)

cudnnActivationBackward_v4 (versioned)

cudnnCreateLRNDescriptor

cudnnSetLRNDescriptor

cudnnGetLRNDescriptor

cudnnDestroyLRNDescriptor

cudnnLRNCrossChannelForward

cudnnLRNCrossChannelBackward

cudnnDivisiveNormalizationForward

cudnnDivisiveNormalizationBackward

cudnnBatchNormalizationForwardInference

cudnnBatchNormalizationForwardTraining

cudnnBatchNormalizationBackward

cudnnDeriveBNTensorDescriptor

cudnnCreateRNNDescriptor

cudnnDestroyRNNDescriptor

cudnnSetRNNDescriptor

cudnnGetRNNWorkspaceSize

cudnnGetRNNTrainingReserveSize

cudnnGetRNNParamsSize

cudnnGetRNNLinLayerMatrixParams

cudnnGetRNNLinLayerBiasParams

cudnnRNNForwardInference

cudnnRNNForwardTraining

cudnnRNNBackwardData

cudnnRNNBackwardWeights

cudnnCreateDropoutDescriptor

cudnnDestroyDropoutDescriptor

cudnnDropoutGetStatesSize

cudnnDropoutGetReserveSpaceSize

cudnnSetDropoutDescriptor

cudnnDropoutForward

cudnnDropoutBackward

cudnnCreateSpatialTransformerDescriptor

cudnnDestroySpatialTransformerDescriptor

cudnnSetSpatialTransformerNdDescriptor

cudnnSpatialTfGridGeneratorForward

cudnnSpatialTfGridGeneratorBackward

cudnnSpatialTfSamplerForward

cudnnSpatialTfSamplerBackward

Batch Normalization

Seems @lukemetz was working on it but has stalled for a bit #1759

What is the plan for the RNN ?

I know @wchan was working on a cpu version. I was trying to get a stub at using the cudnn version but from the comments in that thread seemed like @zheng-xq was working on it internally.

Can anyone comment on the status of these ops?

Other Questions

Any reasons why the Softmax functions are not being used ?

Would make sense to split the above list in chunks and create issues with contribution welcome, so external contributors can tackle them without duplicating internal work, or mark what google will be working on internally ?

I hope this helps in organizing the work around the cudnn and inspire the community to contribute. I will try to keep this issue up to date.

Issue 2:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mac OS

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

TensorFlow installed from (source or binary): Yes

TensorFlow version (use command below):

Python version: 3.5

Bazel version (if compiling from source):

GCC/Compiler version (if compiling from source):

CUDA/cuDNN version:

GPU model and memory:

Describe the current behavior

when using a keras model and transforming it with model_to_estimator accuracy and loss are not stored inevents.out.tfevents.xxxxxxxxxx (used by TensorBoard for visualiztion). I am using tf.estimator.train() or tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec) and in both cases I only get:

dict_keys(['global_step/sec', 'loss_1'])

using tf.estimator.train() I am expecting loss and accuray in events.out.tfevents.xxxxxxxxxx for the training dataset

using tf.estimator.train_and_evaluate(estimator, train_spec, eval_spec) I am expecting loss and accuray in events.out.tfevents.xxxxxxxxxx for the training dataset abd testing datset.

The info are computed and display in the logfiles:

INFO:tensorflow:Done running local_init_op.
INFO:tensorflow:Evaluation [10/100]
INFO:tensorflow:Evaluation [20/100]
INFO:tensorflow:Evaluation [30/100]
INFO:tensorflow:Evaluation [40/100]
INFO:tensorflow:Evaluation [50/100]
INFO:tensorflow:Evaluation [60/100]
INFO:tensorflow:Evaluation [70/100]
INFO:tensorflow:Finished evaluation at 2019-03-16-12:28:05
INFO:tensorflow:Saving dict for global step 1000: accuracy = 0.9766614, global_step = 1000, loss = 0.074898034
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 1000: results/Models/Mnist/tf_1_12/estimator/ckpt/model.ckpt-1000

Is this expected ? It is documented somehwere ? without such info in TensorBoard then it is impossible to see if the model is overfitting or not. I am not even sure which info is displayed in the logfile, right ? or is there some other magic way to get such info ?

why tf.summary information are not propagated from keras to estimator model ? We don't have much control over the estimator model when it is converted from Keras

One option is to use a estimator model using keras layer so we can control everything but since Keras is now the official high level API from TensorFlow, I was thinking I could just transform it to an estimator model. I will be happy to test TF 2.0 if this will fix my issue.

Describe the expected behavior

when I am using the same keras model and running the fit method, I got in the events.out.tfevents.xxxxxxxxxx file:

dict_keys(['batch_acc', 'batch_loss', 'epoch_acc', 'epoch_loss', 'epoch_val_acc', 'epoch_val_loss'])

I got 6 variables inluding accuracy and loss for the training and test datset. This is what Iam expected.

Code to reproduce the issue

code to inspect events.out.tfevents.xxxxxxxxxx files:

from tensorboard.backend.event_processing import event_accumulator
import numpy as np

def load_data_tensorboard(path):
event_acc = event_accumulator.EventAccumulator(path)
event_acc.Reload()
data = {}

for tag in sorted(event_acc.Tags()["scalars"]):
x, y = [], []
for scalar_event in event_acc.Scalars(tag):
x.append(scalar_event.step)
y.append(scalar_event.value)
data[tag] = (np.asarray(x), np.asarray(y))
return data

one notebook contain the code but this is "work in progress":

https://github.com/tarrade/proj_DL_models_and_pipelines_with_GCP/blob/master/notebook/08-Mnist_keras_estimator.ipynb