DJR:Added spidering from web
DJR:Improved the profile spidering and redid the FSM interface