Merge pull request #6 from sunnyLA4/master

Few Documentation
Robots.txt rule ammended for target url in scaper
Merge branch 'test'