Crawler functionality added. Needs to be tested - high possibility for error.