[Reviewer: Andy] Give up on 4xx or non-503 response

Andy,

Please can you review this fix to make a 4xx or non-503 response give up immediately?  Unfortunately, UT isn't easy here due to infrastructure issues, but I've live-tested.

Thanks,

Matt
We can't hit the `!= 504` branch here... bit weird.  Looks like it works fine regardless.