start work to build cache into webcrawler directly. Handle case where url ends in '/' by rewriting links to absolute urls
experimental zodb cache to try to reduce memory of html