Close down channels that we open in RpcServer

We need to close the Channel that we open in RpcServer, or else we hold the socket open