Package: datasette-block-robots
datasette-block-robotsDatasette plugin that blocks robots and crawlers using robots.txtInstallationInstall this plugin in the same environment as Datasette.$ pip install datasette-block-robotsUsageHaving installed the plugin,/robots.txton your Datasette instance will return the following:User-agent: *
Disallow: /This will request all robots and crawlers not to visit any of the pages on your site.Here's a demo of the plugin in action:https://sqlite-generate-demo.datasette.io/robots.txtConfigurationBy default the plugin will block all access to the site, usingDisallow: /.If you want the index page to be indexed by search engines without crawling the database, table or row pages themselves, you can use the following:{"plugins":{"datasette-block-robots":{"allow_only_index":true}}}This will return a/robots.txtlike so:User-agent: *
Disallow: /db1
Disallow: /db2With aDisallowline for every attached database.To block access to specific areas of the site using custom paths, add this to yourmetadata.jsonconfiguration file:{"plugins":{"datasette-block-robots":{"disallow":["/mydatabase/mytable"]}}}This will result in a/robots.txtthat looks like this:User-agent: *
Disallow: /mydatabase/mytableAlternatively you can set the full contents of therobots.txtfile using theliteralconfiguration option. Here's how to do that if you are using YAML rather than JSON and have ametadata.ymlfile:plugins:datasette-block-robots:literal:|-User-agent: *Disallow: /User-agent: BingbotUser-agent: GooglebotDisallow:This example would block all crawlers with the exception of Googlebot and Bingbot, which are allowed to crawl the entire site.Extending this with other pluginsThis plugin adds a newplugin hookto Datasete calledblock_robots_extra_lines()which can be used by other plugins to add their own additional lines to therobots.txtfile.The hook can optionally accept these parameters:datasette: The currentDatasette instance. You can use this to execute SQL queries or read plugin configuration settings.request: TheRequest objectrepresenting the incoming request to/robots.txt.The hook should return a list of strings, each representing a line to be added to therobots.txtfile.It can also return anasync deffunction, which will be awaited and used to generate a list of lines. Use this option if you need to makeawaitcalls inside you hook implementation.This example uses the hook to add aSitemap: http://example.com/sitemap.xmlline to therobots.txtfile:fromdatasetteimporthookimpl@hookimpldefblock_robots_extra_lines(datasette,request):return["Sitemap:{}".format(datasette.absolute_url(request,"/sitemap.xml")),]This example blocks access to paths based on a database query:@hookimpldefblock_robots_extra_lines(datasette):asyncdefinner():db=datasette.get_database()result=awaitdb.execute("select path from mytable")return["Disallow: /{}".format(row["path"])forrowinresult]returninnerdatasette-sitemapis an example of a plugin that uses this hook.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cd datasette-block-robots
python3 -mvenv venv
source venv/bin/activateOr if you are usingpipenv:pipenv shellNow install the dependencies and tests:pip install -e '.[test]'To run the tests:pytest