Add logging and backoff and simple retry logic

Fixes #1243 as described with a few more obvious cases as well.

/cc @philips @unihorn @xiangli-cmu
can we make this a const?