Rewrite link crawler code

I shortened the linkcrawler code and removed some unused code.