made extensions_to_ignore and crawler_user_agents more easily configurable