Separate BufferedStream and remote_addr

Also keep memstream for tests