GitHub repository: simonw/robots-txt-demos
Primary language: HTML
Created: 2018-07-26T17:15:54Z
Description: A tiny site demonstrating different ways to block search engine crawlers