Add default (global) crawler. Add some crawler optimizations.