Optimized and debugged crawler program.