BBC-News/wraith — pull request #51 by boldfacedesign (2013-10-28 11:45:04+00:00)
Title: Auto spidering function added

I've added an auto spidering functionality.
To utilize this simply leave the paths array in the config blank.
This will initiate a full spider of the base domain in the config file. File extension pathways such as images and scripts files will be ignored.
A .txt file will be created and kept for the number of days as defined in the config under 'spider_days'. If the existing site spider.txt file is older than stated in the config the base domain will be re-spidered and a new file created.