Better memory management of stream_peers.