Avoid errors during RPC cancel

Cancelling the RPC acceptors can sometimes result in an error about a bad file descriptor.

As this is the shutdown sequence we need to continue nevertheless, ignore these errors, log a warning and proceed.

As suggested by @theuni.

Fixes #4352.