added logic for tech subdomain and robot disallow