Fix matching on hosts with custom ports or users set. Fixes #1071