Your robots.txt file is a document that tells search engine bots which pages should or should not be crawled. Every website should have a basic robots.txt file, even if there are not specific resources to exclude.