Also drain the inv channel as well as the message channel on peer quit.

Shold prevent a deadlock where we exit but server/blockmaanger is
waiting on our inv channel.

Closes #62