Fix matching on hosts with custom ports or users set

This is an update of PR #1072 with passing tests
Thanks @cheald

Fixes #1071
Closes #1072
Merge pull request #1157 from shaneog/issue-1071

Fix matching on hosts with custom ports or users set