Swallow error if Accept() fails on a stopped server.