Remove port unless specified.