Add logging and backoff and simple retry logic

Fixes #1243 as described with a few more obvious cases as well.

/cc @philips @unihorn @xiangli-cmu
thanks. i am a fan of using short name in function scope when it makes sense.