opened
jlewi
username_0
After running synchronous training for a while the ps (really the workers) started crashing with.

```
2018-11-07T23:51:00Z  tensorflow        "code": 429,
2018-11-07T23:51:00Z  tensorflow        "message": "The total number of changes to the object code-search-demo/models/20181107-dist-sync-gpu/model.ckpt-8000_temp_aa3d8a'
2018-11-07T23:51:00Z  tensorflow      	 when uploading gs://code-search-demo/models/20181107-dist-sync-gpu/model.ckpt-8000_temp_aa3d8a9093914456acccf5aea627e82e/
2018-11-07T23:51:01Z  tensorflow      2018-11-07 23:51:01.577393: I tensorflow/core/platform/cloud/retrying_utils.cc:77] The operation failed and will be automatically retried in 1.49778 seconds (attempt 1 out of 10), caused by: Unavailable: Upload to gs://code-search-dem
o/models/20181107-dist-sync-gpu/model.ckpt-8000_temp_aa3d8a9093914456acccf5aea627e82e/ failed, caused by: Not found: Error executing an HTTP request: HTTP response code 410 with body '{
2018-11-07T23:51:01Z  tensorflow       "error": {
2018-11-07T23:51:01Z  tensorflow        "errors": [
2018-11-07T23:51:01Z  tensorflow         {
2018-11-07T23:51:01Z  tensorflow          "domain": "usageLimits",
2018-11-07T23:51:01Z  tensorflow          "reason": "rateLimitExceeded",
2018-11-07T23:51:01Z  tensorflow          "message": "The total number of changes to the object code-search-demo/models/20181107-dist-sync-gpu/model.ckpt-8000_temp_aa3d8a9093914456acccf5aea627e82e/ exceeds the rate limit. Please reduce the rate of create, update, and dele
te requests."
2018-11-07T23:51:01Z  tensorflow         }
2018-11-07T23:51:01Z  tensorflow        ],
```

The chief then exited with

```
2018-11-08T17:22:51Z  tensorflow         INFO:tensorflow:An error was raised. This may be due to a preemption in a connected worker or parameter server. The current session will be closed and a new session will be created. Error: Socket closed
```

Here are the logs

[demo-trainer-11-07-dist-sync-gpu-chief-0.log.txt](https://github.com/kubeflow/examples/files/2564182/demo-trainer-11-07-dist-sync-gpu-chief-0.log.txt)
[demo-trainer-11-07-dist-sync-gpu-ps-0.log.txt](https://github.com/kubeflow/examples/files/2564184/demo-trainer-11-07-dist-sync-gpu-ps-0.log.txt)
[demo-trainer-11-07-dist-sync-gpu-ps-1.log.txt](https://github.com/kubeflow/examples/files/2564185/demo-trainer-11-07-dist-sync-gpu-ps-1.log.txt)

A couple questions come to mind 

1. Why are the ps trying to write the checkpoint files? Shouldn't it be the chief that writes the check point files? Is something misconfigured with distributed training?

1. Why did the TFJob exit rather than restarting? Was the chief exit code set correctly?
[code_search] synchronous training crashes; "The total number of changes to the object exceeds the rate limit"
issue
created
jlewi
username_0
Here are the events for the TFJob.

[events.txt](https://github.com/kubeflow/examples/files/2564205/events.txt)

The events indicated the PS exited and then it looks like the job failed. This looks like a bug. At a minimum why didn't we emit an event about the chief pod?
comment
created
jlewi
username_0
[demo-trainer-11-07-dist-sync-gpu-chief-0.events.txt](https://github.com/kubeflow/examples/files/2564208/demo-trainer-11-07-dist-sync-gpu-chief-0.events.txt)
comment
created
jlewi
username_0
@username_1 Do you know off hand whether its the chief that writes the checkpoints in the case of synchronous training?
comment
created
jlewi
username_0
It looks like if "Chief" is specified in the job spec; T2T thinks its an async job and changes the RunConfig
https://github.com/tensorflow/tensor2tensor/blob/091373cd5eade2f108ad0fd8a209df3a5cf76598/tensor2tensor/utils/trainer_lib.py#L132

Here's the job spec
[demo-trainer-11-07-dist-sync-gpu.yaml.txt](https://github.com/kubeflow/examples/files/2566169/demo-trainer-11-07-dist-sync-gpu.yaml.txt)
comment
created
cwbeitel
username_1
@username_0 I tried repeating what worked for me previously and I got the same master error you are seeing about a gRPC failure, here are my logs and configs:

Master config: https://gist.github.com/username_1/7a6404ad1639af57518cfbf0432def8f
Master logs: https://gist.github.com/username_1/0b65d9234e66198c11e99ea84bf65779

PS/worker conifg: https://gist.github.com/username_1/234036082f70fb1254a390535f6a510a
PS/worker logs: https://gist.github.com/username_1/67669a314bcd0761d97a2665958b0e62

This is interesting because (1) it failed much sooner than yours, and (2) it failed when it hadn't been before with the only change I know of being from transformer_tiny to transformer_base hparams. You could verify something similar happens for you by running a job with 2 workers and still with tarnsformer_base as well as a job with transformer_tiny. I'm guessing the issue relates in part to model size which should in turn mean more data transmitted (and perhaps more write operations) at some point in the chain.

But I don't know why the PS are attempting to write anything. Perhaps only gradients are communicated over gRPC and model parameters are written to disk by workers and aggregated by master at checkpoint writing time?

Also what tf logging verbosity are you using? Some of your extra log info is probably coming from using more recent versions of tf and t2t.
comment
created
cwbeitel
username_1
@username_0 In the worst case we could just have training nodes write checkpoints to NFS and use a separate process for syncing checkpoints to GCS, perhaps only those that have finished being written (i.e. not those with _temp... or whatever in the checkpoint name).
comment