add robotstxt check, and simple nmap scan