Package: wayback-google-analytics
Wayback Google AnalyticsA lightweight tool to gather current and historic Google analytics codes for OSINT investigations.·Report Bug·Request FeatureTable of ContentsAbout The ProjectWhy do I need GA codes?How does this tool help me?Further readingBuilt WithInstallationInstall with pipDownload from sourceUsageGetting startedOutput files and spreadsheetsExample spreadsheetContributingBugs and feature requestsDevelopmentTestingUsing Poetry for developmentLicenseContactAcknowledgmentsAbout The ProjectWayback Google Analytics is a lightweight tool that gathers current and historic
Google analytics data (UA, GA and GTM codes) from a collection of website urls.Read Bellingcat's article about using this tool to uncover disinformation networks onlinehere.Why do I need GA codes?Google Analytics codes are a useful data point when examining relationships between websites. If two seemingly disparate websites share the same UA, GA or GTM code then there is a good chance that they are managed by the same individual or group. This useful breadcrumb has been used by researchers and journalists in OSINT investigations regularly over the last decade, but a recent change in how Google handles its analytics codes threatens to limit its effectiveness. Google began phasing out UA codes as part of its Google Analytics 4 upgrade in July 2023, making it significantly more challenging to use this breadcrumb during investigations.How does this tool help me?Luckily, the Internet Archive'sWayback Machinecontains useful snapshots of websites containing their historic GA IDs. While you could feasibly check each snapshot manually, this tool automates that process with the Wayback Machines CDX API to simplify and speed up the process. Enter a list of urls and a time frame (along with extra, optional parameters) to collect current and historic GA, UA and GTM codes and return them in a format you choose (json, txt, xlsx or csv).The raw json output for each provided url looks something like this:"someurl.com":{"current_UA_code":"UA-12345678-1","current_GA_code":"G-1234567890","current_GTM_code":"GTM-12345678","archived_UA_codes":{"UA-12345678-1":{"first_seen":"01/01/2019(12:30)","last_seen":"03/10/2020(00:00)",},},"archived_GA_codes":{"G-1234567890":{"first_seen":"01/01/2019(12:30)","last_seen":"01/01/2019(12:30)",}},"archived_GTM_codes":{"GTM-12345678":{"first_seen":"01/01/2019(12:30)","last_seen":"01/01/2019(12:30)",},},}Further readingFor more info about analytics codes and what the GA-4 rollout means for OSINT:https://digitalinvestigations.substack.com/p/what-the-rollout-of-google-analyticsFor an example investigation usings analytics codes:https://www.bellingcat.com/resources/how-tos/2015/07/23/unveiling-hidden-connections-with-google-analytics-ids/(back to top)Built WithAdditional libraries/tools:BeautifulSoup4,Asyncio,Aiohttp(back to top)InstallationInstall frompypi(with pip)The easiest way to to install Wayback Google Analytics is from the command line with pip.Open a terminal window and navigate to your chosen directory.Create a virtual environment and activate it (optional, but recommended; if you usePoetryorpipenvthose package managers do it for you)python3 -m venv venv
source venv/bin/activateInstall the project with pippip install wayback-google-analyticsGet a high-level overviewwayback-google-analytics -hDownload from sourceYou can also clone and download the repo from github and use the tool locally.Clone repo:git clone git@github.com:bellingcat/wayback-google-analytics.gitNavigate to root, create a venv and install requirements.txt:cd wayback-google-analytics
python -m venv venv
source venv/bin/activate
pip install -r requirements.txtGet a high-level overview:python -m wayback_google_analytics.main -h(back to top)UsageGetting startedEnter a list of urls manually through the command line using--urls(-u) or from a given file using--input_file(-i).Specify your output format (.csv, .txt, .xlsx or .csv) using--output(-o).Add any of the following options:Options list (runwayback-google-analytics -hto see in terminal):options:
  -h, --help            show this help message and exit
  -i INPUT_FILE, --input_file INPUT_FILE
                        Enter a file path to a list of urls in a readable file type
                        (e.g. .txt, .csv, .md)
  -u URLS [URLS ...], --urls URLS [URLS ...]
                        Enter a list of urls separated by spaces to get their UA/GA
                        codes (e.g. --urls https://www.google.com
                        https://www.facebook.com)
  -o {csv,txt,json,xlsx}, --output {csv,txt,json,xlsx}
                        Enter an output type to write results to file. Defaults to
                        json.
  -s START_DATE, --start_date START_DATE
                        Start date for time range (dd/mm/YYYY:HH:MM) Defaults to
                        01/10/2012:00:00, when UA codes were adopted.
  -e END_DATE, --end_date END_DATE
                        End date for time range (dd/mm/YYYY:HH:MM). Defaults to None.
  -f {yearly,monthly,daily,hourly}, --frequency {yearly,monthly,daily,hourly}
                        Can limit snapshots to remove duplicates (1 per hr, day, month,
                        etc). Defaults to None.
  -l LIMIT, --limit LIMIT
                        Limits number of snapshots returned. Defaults to -100 (most
                        recent 100 snapshots).
  -sc, --skip_current   Add this flag to skip current UA/GA codes when getting archived
                        codes.Examples:To get current codes for two websites and archived codes between Oct 1, 2012 and Oct 25, 2012:wayback-google-analytics --urls https://someurl.com https://otherurl.org --output json --start_date 01/10/2012 --end_date 25/10/2012 --frequency hourlyTo get current codes for a list of websites (from a file) from January 1, 2012 to the present day, checking for snapshots monthly and returning it as an excel spreadsheet:wayback-google-analytics --input_file path/to/file.txt --output xlsx --start_date 01/01/2012To check a single website for its current codes plus codes from the last 2,000 archive.org snapshots:wayback-google-analytics --urls https://someurl.com --limit -2000Output files & spreadsheetsWayback Google Analytics allows you to export your findings to either.csvor.xlsxspreadsheets. When choosing to save your findings as a spreadsheet, the tool generates two databases: one where each url is the primary index and another where each identified code is the primary index. In an.xlsxfile this is one spreadsheet with two sheets, while the.csvoption generates one file sorted by codes and another sorted by websites. All output files can be found in/output, which is created in the directory from which the code is executed.Example spreadsheetLet's say we're looking into data from 4 websites from 2015 until present and we want to save what we find in an excel spreadsheet. Our start command looks something like this:wayback-google-analytics -u https://yapatriot.ru https://zanogu.com https://whoswho.com.ua https://adamants.ru -s 01/01/2015 -f yearly -o xlsxThe result is a single.xlsxfile with two sheets.Ordered by website:Ordered by code:(back to top)Limitations & Rate LimitsWe recommend that you limit your list of urls to ~10 and your max snapshot limit to <500 during queries. While Wayback Google Analytics doesn't have any hardcoded limitations in regards to how many urls or snapshots you can request, large queries can cause 443 errors (rate limiting). Being rate limited can result in a temporary 5-10 minute ban from web.archive.org and the CDX api.The app currently usesasyncio.Semaphore()along with delays between requests, but large queries or operations that take a long time can still result in a 443. Use your judgment and break large queries into smaller, more manageable pieces if you find yourself getting rate limited.(back to top)ContributingBugs and feature requestsPlease feel free toopen an issueshould you encounter any bugs or have suggestions for new features or improvements. You can alsoreach out to medirectly with suggestions or thoughts.(back to top)DevelopmentTestingRun tests withpython -m unittest discoverCheck coverage withcoverage run -m unittestUsing Poetry for DevelopmentWayback Google Analytics usesPoetry, a Python dependency management and packaging tool. A GitHub workflow automates the tests on PRs and to main (see our workflow here),  be sure to update thesemanticversion number inpyproject.tomlwhen opening a PR.If you have push access, follow these steps to trigger the GitHub workflow that will build and release a new version to PyPI :Change the version number inpyproject.tomlCreate a new tag for that versiongit tag "vX.0.0"Push the taggit push --tagsLicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)ContactYou can contact me through email or social media.email: jclarksummit at gmail dot comTwitter/X:@JustinClarkJOLinkedin:Justin ClarkProject Link:https://github.com/bellingcat/wayback-google-analytics(back to top)AcknowledgmentsBellingcatfor hosting this projectMiguel Ramalhofor constant support, thoughtful code reviews and suggesting the original idea for this project(back to top)