Issue 1:
System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow):Yes

OS Platform and Distribution (e.g., Linux Ubuntu 16.04):MS Windows10 X64 1809 build17763

Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: N/A

TensorFlow installed from (source or binary):binary(use pip)

TensorFlow version (use command below):v1.13.1-0-g6612da8951

Python version:3.6.7

Bazel version (if compiling from source):N/A

GCC/Compiler version (if compiling from source):N/A

CUDA/cuDNN version:10.0/7.5.1

GPU model and memory:NVIDIA Geforce RTX2080TI 11GB x2

Describe the current behavior

def _model_fn(...):
logits, _ = _models.build_model(
features,
model_name=FLAGS.model_name,
training=is_training,
override_params=override_params)
......
global_step = tf.train.get_global_step()
if has_moving_average_decay:
ema = tf.train.ExponentialMovingAverage(
decay=FLAGS.moving_average_decay, num_updates=global_step)
ema_vars = tf.trainable_variables() + tf.get_collection('moving_vars')
for v in tf.global_variables():
if 'moving_mean' in v.name or 'moving_variance' in v.name:
ema_vars.append(v)
ema_vars = list(set(ema_vars))
......
if has_moving_average_decay:
with tf.control_dependencies([train_op]):
train_op = ema.apply(ema_vars)
......
def main():
......
if FLAGS.num_gpus <= 1:
distribution_strategy = None
else:
distribution_strategy = tf.contrib.distribute.MirroredStrategy(num_gpus=FLAGS.num_gpus, cross_device_ops=tf.contrib.distribute.AllReduceCrossDeviceOps('hierarchical_copy', num_packs=2))
config = tf.estimator.RunConfig(
model_dir=FLAGS.model_dir,
train_distribute=distribution_strategy,
save_checkpoints_steps=save_checkpoints_steps,
log_step_count_steps=FLAGS.log_step_count_steps,
session_config=tf.ConfigProto(
allow_soft_placement=True,
graph_options=tf.GraphOptions(
rewrite_options=rewriter_config_pb2.RewriterConfig(
disable_meta_optimizer=True))),)
model_est = tf.estimator.Estimator(
model_fn=_model_fn,
config=config,
params=params
)
......

I have find a bug when use ExponentialMovingAverage with distribute.Strategy.These codes work fine without using a distribution strategy.But when using a distribution strategy (such as MirroredStrategy), an error is reported:

Traceback (most recent call last):
File "C:\Users\admin\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\distribute\shared_variable_creator.py", line 90, in reuse_variable
v = shared_variable_store[canonical_name][variable_index]
KeyError: 'mnasnet-a1/mnasnet/mnas_stem/conv2d/kernel/replica/ExponentialMovingAverage/'

By debugging I can see that it works fine when ema.apply() is executed on the first GPU.But when executed on the second GPU, it will report an error.This is because MirroredVariable/ReplicaLocalVariable has been created when building the model in _model_fn().So on the second GPU, TF tries to create the variable 'conv2d/kernel/replica/ExponentialMovingAverage/'.But it doesn't exist at all, 'replica' should appear at the end of the variable name.

I found a temporary solution by modifying the shared_variable_creator:

def reuse_variable(next_creator, *args, **kwargs):
"""Re-use existing variable from store with same name (in order)."""
del next_creator, args
name = kwargs.get("name")
canonical_name = _canonicalize_variable_name(name)
replica_index = canonical_name.find('replica/')
if replica_index != -1:
canonical_name = canonical_name[0:replica_index] + canonical_name[replica_index+8:]
try:
variable_index = variable_scope_access_index.get(canonical_name, 0)
v = shared_variable_store[canonical_name][variable_index]
# TODO(priyag): Make this variable re-use more robust by adding checks
# that the requested shape and dtype match the existing variable.
variable_scope_access_index[canonical_name] = variable_index + 1
return v
except (KeyError, IndexError):
raise RuntimeError(
"Tried to create variable {} with mismatching name on device {}".
format(name, device_id))

if device_id == 0:
return create_new_variable
else:
return reuse_variable

But this looks more like a patch than a solution.I hope someone can review this bug and fix it.

Describe the expected behavior

Code to reproduce the issue

Provide a reproducible test case that is the bare minimum necessary to generate the problem.

Other info / logs

Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

Issue 2:
I have Ubuntu 14.04 system I installed tensor flow using

pip install https://storage.googleapis.com/tensorflow/linux/gpu/tensorflow-0.5.0-cp27-none-linux_x86_64.whl

then when I try to import tensorflow in my python window, I get the error:

init() got an unexpected keyword argument 'syntax'

Is it a protobuf version problem ?

I have $ protoc --version

libprotoc 2.5.0