web crawler with very customisable and furthermost constant memory footprint