Ignore some errors that may occur during handshake

These errors just pollute the logs when garbage is sent to the
socket. Exit the process normally to avoid unwanted logs.