added testing and finished routes for crawler, all green and ready to merge with master