Add logging and backoff and simple retry logic

Fixes #1243 as described with a few more obvious cases as well.

/cc @philips @unihorn @xiangli-cmu
Yeah. That and shadowing client.