Add crawler: fetchUrl and robots
Conserve all changes