Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04

TensorFlow installed from (source or binary): binary

TensorFlow version (use command below): 1.14 and 2

Python version: 3.6

CUDA/cuDNN version: 10/7

GPU model and memory: NA

Describe the current behavior

Imagine you extend the tf.keras.layers.Layer with the class MyLayer, which is a wrapper for the tf.keras.layers.Dense layer class (this class extension could also be a Model, but that's not important).

Now imagine you instantiate a Dense layer dense (w/ no bias) and initialize two independent MyLayer instances dense1 and dense2 using dense...

Now, we extend the Model class to take two MyLayer instances on init. We do this because we want to use dense1 and dense2 to perform different tasks but we want their weights to be shared (in a realistic example, MyLayer would be split into MyLayer1 and MyLayer2, which would perform different operations on call but would both depend on dense). We run data through our model pipeline and compute a loss, gathering gradients by using either the GradientTape or keras optimizers. However, when we gather and apply gradients with respect to Model.trainable_variables, even though our model has one unique weight, the kernel for dense, Model.trainable_variables will return two variables, and the gradients will be calculated and thus applied twice, which I would say is a bug, insofar as it's unexpected.

Describe the expected behavior

Models whose layers have shared/tied weights should not return duplicate weights when accessing the trainable_variables property. A super simple work-around is to call list(set(model.trainable_variables)) but the real issue is the unexpected behavior: "Why would I ever think that model.trainable_variables would return duplicates of the same variable ?!"

The two calls to test in the code below should have the same output.

Code to reproduce the issue

`

code

import numpy as np
import tensorflow as tf

tf.enable_eager_execution()

class MyDense(tf.keras.layers.Layer):
def __init__(self, dense, **kwargs):
super().__init__(**kwargs)
self.dense = dense
def call(self, inputs):
return self.dense(inputs)

class MyModel(tf.keras.Model):
def __init__(self, dense1, dense2, **kwargs):
super().__init__(**kwargs)
self.dense1 = dense1
self.dense2 = dense2
def call(self, inputs):
return self.dense1(inputs) + self.dense2(inputs)

def test(unique):
x = tf.ones(shape=(10, 5))
y = tf.ones(shape=(10, 1)) + 2

dense = tf.keras.layers.Dense(1, kernel_initializer=tf.keras.initializers.constant(0.2), use_bias=False)
dense1 = MyDense(dense)
dense2 = MyDense(dense)
model = MyModel(dense1, dense2)

adam = tf.keras.optimizers.Adam(0.001)

with tf.GradientTape() as tape:
y_hat = model(x)
loss = tf.keras.losses.mse(y, y_hat)
variables = model.trainable_variables if unique else list(set(model.trainable_variables))
print(f"# trainable variables: {len(variables)}")

pre_weight = variables[0][0]

grads = tape.gradient(loss, variables)
adam.apply_gradients(list(zip(grads, variables)))

post_weight = variables[0][0]

return post_weight - pre_weight

ex1 = test(True)
print(ex1)
print("\n+=+=+=+=\n")
ex2 = test(False)
print(ex2)

`

Issue 2:
This is a little difficult to explain, and I'm guessing someone has to have run into it before, but it would be nice to at least get an explanation for what's going on here.

The issue I'm finding is that when you specify a model_dir argument for any tflearn model, the first time you fit that model, the tensorboard warning like the following will always appear:

WARNING:tensorflow:Found more than one graph event per run. Overwriting the graph with the newest event.

If you clear the contents of that directory and rerun the same model, the warning no longer appears.

For example, here is a piece of code to reproduce:

import tensorflow as tf
import random

from tensorflow.contrib.learn.python import learn
from tensorflow.contrib.learn.python.learn import datasets
from tensorflow.contrib.learn.python.learn.estimators._sklearn import accuracy_score
from tensorflow.contrib.learn.python.learn.estimators._sklearn import train_test_split
random.seed(42)

iris = datasets.load_iris()
x_train, x_test, y_train, y_test = train_test_split(iris.data,
iris.target,
test_size=0.2,
random_state=42)

def custom_optimizer(learning_rate):
return tf.train.MomentumOptimizer(learning_rate, 0.9)

classifier = learn.TensorFlowDNNClassifier(
hidden_units=[10, 20, 10],
n_classes=3,
steps=400,
learning_rate=0.01,
optimizer=custom_optimizer,
model_dir='/tmp/tf/bug_test_1')
classifier.fit(x_train, y_train)

If you run this in a jupyter notebook, wait until it is done, cd into /tmp/tf/bug_test_1, and run tensorboard --logdir=., the warning will appear. If you clear the contents and run again, it will no longer appear. If you, in the same notebook within the same kernel, change the directory to something else (e.g. /tmp/tf/bug_test_2) and run again, the warning will again appear. And the warning will again go away if you clear the contents and run again with the same directory.

I also tried pre-creating the directory but that appears to make no difference.

I wouldn't imagine this indicates a major problem but I try to take those warnings seriously since they have indicated more problematic issues in the past with running multiple models in the same kernel, so it would be great to know what this means.

Environment info

Operating System: OS X Yosemite (10.10.5)

Tensorflow Version: 0.10.0