Determine port based on scheme
Allow explict ports through