reader: use the addr we connected with

this fixes a bug where, if the connections were using
DNS addresses we weren't properly deleting from the map