Package: robotspy
Robots Exclusion Standard Parser for PythonTherobotspyPython module implements a parser forrobots.txtfiles. The recommended class to use isrobots.RobotsParser.A thin facaderobots.RobotFileParsercan also be used as
a substitute forurllib.robotparser.RobotFileParser,
available in the Python standard library. The classrobots.RobotFileParserexposes an API that is
mostly compatible withurllib.robotparser.RobotFileParser.The main reasons for this rewrite are the following:It was initially intended to experiment with parsingrobots.txtfiles for a link checker project
(not implemented yet).It is attempting to follow the latest internet draftRobots Exclusion Protocol.It does not try to be compliant with commonly accepted directives that are not in the currentspecssuch asrequest-rateandcrawl-delay,
but it currently supportssitemaps.It satisfies the same tests as theGoogle Robots.txt Parser,
except for some custom behaviors specific to Google Robots.To use therobotscommand line tool (CLI) in a Docker container, read the following sectionDocker Image.To installrobotspyglobally as a tool on your system withpipxskip to theGlobal Installationsection.If you are interested in usingrobotspyin a local Python environment or as a library, skip to sectionModule Installation.Docker ImageThe Robotspy CLI,robots, is available as aDockerautomated built image athttps://hub.docker.com/r/andreburgaud/robotspy.If you already haveDockerinstalled on your machine, first pull the image from Docker Hub:$ docker pull andreburgaud/robotspyThen, you can exercise the tool against the following remote Pythonrobots.txttest file located athttp://www.pythontest.net/elsewhere/robots.txt:# Used by NetworkTestCase in Lib/test/test_robotparser.py

User-agent: Nutch
Disallow: /
Allow: /brian/

User-agent: *
Disallow: /webstats/The following examples demonstrate how to use therobotscommand line with the Docker container:$ # Example 1: User agent "Johnny" is allowed to access path "/"
$ docker run --rm andreburgaud/robotspy http://www.pythontest.net/elsewhere/robots.txt Johnny /
user-agent 'Johnny' with path '/': ALLOWED$ # Example 2:  User agent "Nutch" is not allowed to access path "/brian"
$ docker run --rm andreburgaud/robotspy http://www.pythontest.net/elsewhere/robots.txt Nutch /brian
user-agent 'Nutch' with path '/brian': DISALLOWED$ # Example 3: User agent "Johnny" is not allowed to access path "/webstats/"
docker run --rm andreburgaud/robotspy http://www.pythontest.net/elsewhere/robots.txt Johnny /webstats/
user-agent 'Johnny' with path '/webstats/': DISALLOWEDThe arguments are the following:Location of the robots.txt file (http://www.pythontest.net/elsewhere/robots.txt)User agent name (Johnny)Path or URL (/)Without any argument,robotsdisplays the help:docker run --rm andreburgaud/robotspy
usage: robots <robotstxt> <useragent> <path>

Shows whether the given user agent and path combination are allowed or disallowed by the given robots.txt file.

positional arguments:
  robotstxt      robots.txt file path or URL
  useragent      User agent name
  path           Path or URI

optional arguments:
  -h, --help     show this help message and exit
  -v, --version  show program's version number and exitTo use the CLIrobotsas a global tools, continue to the following section. If you want to userobotspyas a Python module, skip toModule Installation.Global Installation with pipxIf you only want to use the command line toolrobots, you may want to usepipxto install it as a global tool on your system.To installrobotspyusingpipxexecute the following command:$pipxinstallrobotspyWhenrobotspyis installed globally on your system, you can invoke it from any folder locations. For example, you can execute:$robots--version
robots0.6.0You can see more detailed usages in sectionUsage.Module InstallationNote: Python 3.8.x or 3.9.x requiredYou preferably want to install therobotspypackage after creating a Python virtual environment,
in a newly created directory, as follows:$ mkdir project && cd project
$ python -m venv .venv
$ . .venv/bin/activate
(.venv) $ python -m pip install --upgrade pip
(.venv) $ python -m pip install --upgrade setuptools
(.venv) $ python -m pip install robotspy
(.venv) $ python -m robots --help
...On Windows:C:/> mkdir project && cd project
C:/> python -m venv .venv
C:/> .venv\scripts\activate
(.venv) c:\> python -m pip install --upgrade pip
(.venv) c:\> python -m pip install --upgrade setuptools
(.venv) c:\> python -m pip install robotspy
(.venv) c:\> python -m robots --help
...UsageTherobotspypackage can be imported as a module and also exposes an executable,robots, invocable withpython -m. If installed globally withpipx, the commandrobotscan be invoked from any folders. The usage examples in the following section use the commandrobots, but you can also substitute it withpython -m robotsin a virtual environment.Execute the ToolAfter installingrobotspy, you can validate the installation by running the following command:$ robots --help
usage: robots <robotstxt> <useragent> <path>

Shows whether the given user agent and path combination are allowed or disallowed by the given robots.txt file.

positional arguments:
  robotstxt      robots.txt file path or URL
  useragent      User agent name
  path           Path or URI

optional arguments:
  -h, --help     show this help message and exit
  -v, --version  show program's version number and exitExamplesThe content ofhttp://www.pythontest.net/elsewhere/robots.txtis the following:# Used by NetworkTestCase in Lib/test/test_robotparser.py

User-agent: Nutch
Disallow: /
Allow: /brian/

User-agent: *
Disallow: /webstats/To check if the user agentNutchcan fetch the path/brian/you can execute:$ robots http://www.pythontest.net/elsewhere/robots.txt Nutch /brian/
user-agent 'Nutch' with path '/brian/': ALLOWEDOr, you can also pass the full URL,http://www.pythontest.net/brian/:$ robots http://www.pythontest.net/elsewhere/robots.txt Nutch /brian/
user-agent 'Nutch' with url 'http://www.pythontest.net/brian/': ALLOWEDCan user agentNutchfetch the path/brian?$ robots http://www.pythontest.net/elsewhere/robots.txt Nutch /brian
user-agent 'Nutch' with path '/brian': DISALLOWEDOr,/?$ robots http://www.pythontest.net/elsewhere/robots.txt Nutch /
user-agent 'Nutch' with path '/': DISALLOWEDHow about user agentJohnny?$ robots http://www.pythontest.net/elsewhere/robots.txt Johnny /
user-agent 'Johnny' with path '/': ALLOWEDUse the Module in a ProjectIf you have a virtual environment with therobotspypackage installed, you can use therobotsmodule from the Python shell:(.venv) $ python
>>> import robots
>>> parser = robots.RobotsParser.from_uri('http://www.pythontest.net/elsewhere/robots.txt')
>>> useragent = 'Nutch'
>>> path = '/brian/'
>>> result = parser.can_fetch(useragent, path)
>>> print(f'Can {useragent} fetch {path}? {result}')
Can Nutch fetch /brian/? True
>>>Bug in the Python standard libraryThere is a bug inurllib.robotparserfrom the Python standard library that causes the following test to differ from the example above withrobotspy.The example withurllib.robotparseris the following:$ python
>>> import urllib.robotparser
>>> rp = urllib.robotparser.RobotFileParser()
>>> rp.set_url('http://www.pythontest.net/elsewhere/robots.txt')
>>> rp.read()
>>> rp.can_fetch('Nutch', '/brian/')
FalseNotice that the result isFalsewhereasrobotspyreturnsTrue.Bugbpo-39187was open to raise awareness on this issue and PRhttps://github.com/python/cpython/pull/17794was submitted as a possible fix.robotspydoes not
exhibit this problem.DevelopmentThe main development dependency ispytestfor executing the tests. It is automatically
installed if you perform the following steps:$ git clone https://github.com/andreburgaud/robotspy
$ cd robotspy
$ python -m venv .venv --prompt robots
$ . .venv/bin/activate
(robots) $ python -m pip install -r requirements.txt
(robots) $ python -m pip install -e .
(robots) $ make test
(robots) $ deactivate
$On Windows:C:/> git clone https://github.com/andreburgaud/robotspy
C:/> cd robotspy
C:/> python -m venv .venv --prompt robotspy
C:/> .venv\scripts\activate
(robots) c:\> python -m pip install -r requirements.txt
(robots) c:\> python -m pip install -e .
(robots) c:\> make test
(robots) c:\> deactivateGlobal ToolsThe following tools were used during the development ofrobotspy:BlackMypyPylinttwineSee the build file,Makefileormake.baton Windows, for the commands and parameters.Release History0.7.0:Fixed bug with the argument path when using the CLIPrint 'url' when the argument is a URL, 'path' otherwise0.6.0:Simplified dependencies by keeping onlypytestinrequirements.txt0.5.0:Updated all libraries. Tested with Python 3.9.0.4.0:Fixed issue with robots text pointed by relative pathsIntegration ofMypy,BlackandPylintas depencencies to ease cross-platform developmentLimitedmake.batbuild file for WindowsGit ignore vscode files,tmpdirectory, multiple virtual env (.venv*)Fixed case insensitive issues on WindowsTests successful on WindowsAdded an ATRIBUTIONS files and build task to generate itUpgradedpyparsingandcertifi0.3.3:Upgradedtqdm, andcryptographypackages0.3.2:Upgradedbleach,tqdm, andsetuptoolspackages0.3.1:UpdatedidnaandwcwidthpackagesAddedpipdeptreepackage to provide visibility on dependenciesFixedmypyerrorsExplicitly ignoredpylinterrors related to commonly used names likef,m, orT0.3.0: Updatedbleachpackage to address CVE-2020-68020.2.0: Updated the documentation0.1.0: Initial releaseLicenseMIT License