Added deep crawling. Need to figure out a way to handle network resets
Merge branch 'mining'