use generators to improve url retrieval performance