URI's are now continously added to posts[] list to be re-spidered. After some house keeping, I'll add the spidering function call itself to spider the new links.
It may be more beneifical to have a separate links for the new links to be crawled.