remove duplicate check, added urlparse helper