Temporary fix to visited url cache synchornisation problem
Fixed concurrency issues...
Index saves files to disk and only interesting extensions are now added to the frontier by the document processor