Added javadocs, tests. Changed toString mathods, now they use commons lang.
Added fetching of crawling information.