Use the channel's remote address instead of holding a changeable reference