Add logging and backoff and simple retry logic

Fixes #1243 as described with a few more obvious cases as well.

/cc @philips @unihorn @xiangli-cmu
it is confusing to use nesting here imo.
could we use `for` loop instead?