refactored URL visiting logic