C-based multi-threadded web crawler with perl coordination scripts and BerkeleyDB storage backend