Improvements to the crawler. Still some issues, notably does not deal
with non "http:" protocols.