Crawler does some breadth-first search. Most of the url handling has been implemented. Still some issues with handling relative http paths.