Fix case where port is missing

Please review and test @karlitschek @LukasReschke @schiesbn