file generation is done by crawler