added url props
Major additions for crawler
Testing resources and configuration documentaton.