added logic for tech subdomain and robot disallow
Merge branch 'dev'