pretty good crawler for primitive web. for modern web using redirects and everything this is not very great.