added URL validation and striking out URLS we have already submitted too