Now I found out why it's silently returning...

When closing the socket due to a shutdown, the recv() call returns an error that seems to be net.errClosing. I don't know how to filter that away since it's unexported.