portal may become unselected

This simplifies the logic a bit, hopefully making it easier to debug the
issue See #404. It seems to be related to when the portal changed because
I could trigger it easily on zooming in/out when there was a cache bug
that gave me an R portal in one zoom level and an E one in the other.