Fix matching on hosts with custom ports or users set

This is an update of PR #1072 with passing tests
Thanks @cheald

Fixes #1071
Closes #1072