Package: sitemaps
sitemapsSitemaps is a Python command line tool and library to generate sitemap files by crawling web servers or ASGI apps. Sitemaps is powered byHTTPXandanyio.Note: This is alpha software. Be sure to pin your dependencies to the latest minor release.QuickstartLive serverpython-msitemapshttps://example.orgExample output:$catsitemap.xml<?xml version="1.0" encoding="utf-8"?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.sitemaps.org/schemas/sitemap/0.9 http://www.sitemaps.org/schemas/sitemap/0.9/sitemap.xsd"><url><loc>https://example.org/</loc><changefreq>daily</changefreq></url></urlset>ASGI appHTTP requests are issued to the ASGI app directly. The target URL is only used as a base URL for building sitemap entries.python-msitemaps--asgi'<module>:<attribute>'http://testserverCheck modeUseful to verify that the sitemap is in sync (e.g. as part of CI checks):python-msitemaps--check[...]FeaturesSupport for crawling any live web server.Support for crawling an ASGI app directly (i.e. without having to spin up a server).--checkmode.Invoke from the command line, or use the programmatic async API (supports asyncio and trio).Fully type annotated.100% test coverage.InstallationInstall with pip:$pipinstallsitemapsSitemaps requires Python 3.7+.Command line reference$python-msitemaps--helpusage: __main__.py [-h] [-o OUTPUT] [-I IGNORE_PATH_PREFIX] [--asgi ASGI][--max-concurrency MAX_CONCURRENCY] [--check]targetpositional arguments:target                The base URL used to crawl the website and buildsitemap URL tags.optional arguments:-h, --help            show this help message and exit-o OUTPUT, --output OUTPUTOutput file path.-I IGNORE_PATH_PREFIX, --ignore-path-prefix IGNORE_PATH_PREFIXIgnore URLs for this path prefix. Can be used multipletimes.--asgi ASGI           Path to an ASGI app, formatted as'<module>:<attribute>'.--max-concurrency MAX_CONCURRENCYMaximum number of URLs to process concurrently.--check               Compare existing output and fail if computed XMLdiffers.Programmatic APILive serverimportsitemapsasyncdefmain():urls=awaitsitemaps.crawl("https://example.org")withopen("sitemap.xml","w")asf:f.write(sitemaps.make_xml(urls))ASGI appimporthttpximportsitemapsfrom.appimportappasyncdefmain():asyncwithhttpx.AsyncClient(app=app)asclient:urls=awaitsitemaps.crawl("http://testserver",client=client)withopen("sitemap.xml","w")asf:f.write(sitemaps.make_xml(urls))Customizing URL tagsBy default,.make_xml()generates<url>tags with adailychange frequency. You can customize the generation of URL tags by passing a customurltagcallable:fromurllib.parseimporturlsplitdefurltag(url):path=urlsplit(url).pathchangefreq="monthly"ifpath.startswith("/reports")else"daily"returnf"<url><loc>{url}</loc><changefreq>{changefreq}</changefreq></url>"asyncdefmain():urls=awaitsitemaps.crawl(...)withopen("sitemap.xml","w")asf:f.write(sitemaps.make_xml(urls,urltag=urltag))LicenseMITChangelogAll notable changes to this project will be documented in this file.The format is based onKeep a Changelog.0.1.0 - 2020-05-31AddedInitial implementation: CLI and programmatic async API.