Added the ability to ignore crawlers using the '--ignore-crawlers' option.