Recurse through web directory starting from the starting URL.