Moving index writer handling to main, web crawler, thread.