first pull directory urls. rate limited to 1sec