continue indexation even when a job repeatedly crashes
better control check of wrong urls collected in crawler
better script controllers