Repository: tensorflow/tensor2tensor

Issue:
distributed training on multiple machine fails
### Description
I am trying to do distributed training on multiple machines with 1 GPU each. It is failing on the workers. Please look into this!
...

### Environment information
```
OS: Linux - 18

On master i run -
t2t-trainer --master=grpc://10.10.1.2:2219 --ps_replicas=3 --worker_replicas=1 --worker_gpu=0 --worker_id=0 --ps_gpu=1 --sync --schedule=train --worker_job='/job:master' --model=transformer --hparams_set=transformer_base --problem=translate_ende_wmt32k --data_dir=/users/kshiteej/varunimagenet/tensor2tensor/t2t_data/ --output_dir=/users/kshiteej/

On PS-
1. t2t-trainer --schedule=run_std_server 
2. t2t-trainer --schedule=run_std_server 
3. t2t-trainer --schedule=run_std_server 

OUTPUT of Master - 
..
.
.
.
13] Done calling model_fn.
INFO:tensorflow:Create CheckpointSaverHook.
I0331 22:40:02.157696 139967148951360 basic_session_run_hooks.py:527] Create CheckpointSaverHook.
INFO:tensorflow:Graph was finalized.

OUTPUT of Worker - 
Traceback (most recent call last):
  File "/usr/local/bin/t2t-trainer", line 33, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "/usr/local/bin/t2t-trainer", line 28, in main
    t2t_trainer.main(argv)
  File "/usr/local/lib/python3.6/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 413, in main
    hparams = create_hparams()
  File "/usr/local/lib/python3.6/dist-packages/tensor2tensor/bin/t2t_trainer.py", line 176, in create_hparams
    return trainer_lib.create_hparams(FLAGS.hparams_set, FLAGS.hparams,hparams_path=hparams_path)
  File "/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/hparams_lib.py", line 48, in create_hparams
    hparams = registry.hparams(hparams_set)
  File "/usr/local/lib/python3.6/dist-packages/tensor2tensor/utils/registry.py", line 254, in __getitem__
    (key, self.name, display_list_by_prefix(sorted(self), 4)))
KeyError: 'None never registered with registry hparams. Available:\n     adaptive:\n      * adaptive_universal_transformer_base\n      * adaptive_universal_tr...
..
..
..



$ pip freeze | grep tensor
# your output here
mesh-tensorflow==0.0.5
tensor2tensor==1.13.1
tensorboard==1.13.0
tensorflow-datasets==1.0.1
tensorflow-estimator==1.13.0
tensorflow-gpu==1.13.1
tensorflow-metadata==0.13.0
tensorflow-probability==0.6.0
tensorflow-tensorboard==0.4.0

$ python -V
# your output here
```
Python 2.7.15rc1

### For bugs: reproduction and error logs

```
# Steps to reproduce:
...
```

```
# Error logs:
...
```



Discussion:
You need to pass ``hparams_set`` to ps as well.

Resolving patch:
diff --git a/tensor2tensor/bin/t2t_trainer.py b/tensor2tensor/bin/t2t_trainer.py
--- a/tensor2tensor/bin/t2t_trainer.py
+++ b/tensor2tensor/bin/t2t_trainer.py
@@ -372,7 +372,8 @@ def main(argv):
   # Create HParams.
   if argv:
     set_hparams_from_args(argv[1:])
-  hparams = create_hparams()
+  if FLAGS.schedule != "run_std_server":
+    hparams = create_hparams()
 
   if FLAGS.schedule == "train" or FLAGS.schedule == "train_eval_and_decode":
     mlperf_log.transformer_print(key=mlperf_log.RUN_START, hparams=hparams)


Test patch: