Add some retries

Getting a lot of connection errors on the CI and it's annoying.
Swallow errors in stop