A distributed crawling framework using rabbitmq and mongodb