add robots parser, changed logger and other optimizations