Check port when removing self referenced routes

* Cleanup how self referenced routes were determined
* If the ports are different, don't bother checking the host