Fix web config reloading logic error

We should stop retrying regardless of whether it's a reload or not. We
also want to keep track of reload retries.