Do not track IPs which have no connections.

This prevents needless growth of memory footprint over time.