Better integrate the crawler and monitor when checking for executors

1) Implement the shutdown() method in the crawler to shutdown the executor pool
2) Add a polling interval to the monitor so you can control how long to sleep between active executor checks