Crawler user agents are configurable.
making extensionsToIgnore configurable
incremented version