Package: consentcrawl
ConsentCrawlAutomatically check for GDPR/CCPA consent by running a Playwright headless browser to check for marketing and analytics scripts firing before and after consent.Detect 25+ consent managersDetect unconsented third-party domains and cookiesClassify tracking domains based on 7 commonly used ad blocking listsKeep screenshots before and after consentCapture JSON-LD and meta tags for convenienceRun multiple URLs in batchAdd custom blocklists and consent manager listsCLI Argumentsusage:consentcrawl[-h][--debug][--headless[HEADLESS]][--screenshot][--bootstrap][--batch_sizeBATCH_SIZE][--show_output][--db_fileDB_FILE][--blocklistsBLOCKLISTS]urlArgumentDescriptionurl(required) URL or file with URLs to test--debugEnable debug logging--headlessRun browser in headless mode (true/false)--screenshotTake screenshots of each page before and after consent is given (ifconsent manager is detected)--bootstrapForce bootstrap (refresh) of blocklists--batch_size, -bNumber of URLs (and browser windows) to run in each batch. Default: 15, increase or decrease depending on your system capacity.--show_output, -oShow output of the last results in terminal (max 25 results)--db_file, -dbPath to crawl results and blocklist database--blocklists, -bfPath to custom blocklists file (YAML)In actionDownload and install with:pip install consentcrawlThePlaywright (headless) browsersare not automatically installed so runplaywright installto install all or specify e.g.playwright install chromiumWhen runningconsentcrawlYou can provide either a single URL, comma separated list or a file (.txt) with one URL per line:consentcrawl google.com,google.nl,google.de --headless=false -oIf you havejqinstalled you can pipe the output to jq to directly get, for example, all tracking domains without consent:consentcrawl leboncoin.fr,marktplaats.nl,ebay.com -o | jq '.[] | .tracking_domains_no_consent'Returns:["tiqcdn.com","criteo.net","googlesyndication.com","doubleclick.net"]["google-analytics.com","scorecardresearch.com","bing.com","doubleclick.net","googleadservices.com","criteo.net","googletagmanager.com","spotxchange.com"]["doubleclick.net"]By default the results of your queries will be stored in a SQLite database calledcrawl_results.db.Or if you want to import into an existing Python script:importasynciofromconsentcrawlimportcrawlresults=asyncio.run(crawl.crawl_single("dumky.net"))The playwright browser runs asynchronously which is great for running multiple
URLs in parallel, but for running a single URL you'll need to use asyncio.run()
to run the asynchronous function.How it worksPlaywright allows you to automate browser windows. This script takes a list of URLs, runs a Playwright browser instance and fetches data about cookies and requested domains for each URL. The URLs are fetched asynchronously and in batches to speed up the process. After the URL is fetched, the script tries to identify the consent manager and click 'accept' to determine if and what marketing and analytics tags are fired before and after consent. It uses a 'blocklist' to determine whether a domain is a tracking (marketing/analytics) domain.Available Consent Managers:OneTrustOptanonCookieLaw (CookiePro)Drupal EU Cookie ComplianceJoomlaShaper SP Cookie Consent ExtensionFastCMPGoogle Funding ChoicesKlaroEnsightenGX SoftwareEZ CookieCookieBotCookieHubTYPO3 Wacon Cookie Management ExtensionTYPO3 Cookie Consent ExtensionCommanders Act - Trust CommanderCookieFirstOsanoOrejimeAxceptioCivic UK Cookie ControlUserCentricsCookieYesSecure PrivacyQuantcastDidomiMediaVine CMPCookieLawConsentManager.netHubSpot Cookie BannerLiveRamp PrivacyManager.ioTrustArc TrusteSFBX AppConsentPiwik PRO GDPR Consent ManagerFinsweet Cookie Consent for WebflowNon-specific / Custom (looks for general CSS selectors like "#acceptCookies" or ".cookie-accept")Are you missing a consent manager? Have a look atthe full listand feel free to open an issue or pull request!ExamplesThe examples folder shows examples to run ConsentCrawl:as a Github Actionon Google Cloud Run with a simple FastAPI server that responds with the ConsentCrawl results on a POST request to a/consentcrawlendpoint.To DoFollow redirects on URLsDetect consent managers with cookies instead of just CSS selectorsShow progress when using CLI