Package: crashstats-tools
Command line tools and library for interacting with Crash Stats
(https://crash-stats.mozilla.org/).Code:https://github.com/willkg/crashstats-toolsDocumentation:Check theREADME.rstfileChangelog:Check theHISTORY.rstfileIssue tracker:https://github.com/willkg/crashstats-tools/issuesLicense:MPLv2Chat:#crashreporting matrix channelCommunity Participation Guidelines:https://github.com/willkg/crashstats-tools/blob/main/CODE_OF_CONDUCT.mdInstallingcrashstats-tools is available onPyPI. You can install
it with:$ pip install --user crashstats-toolsSince it has commands that you want to use, you probably want to
install it withpipx:$ pipx install crashstats-toolsSupportThis is a project that I spent some time on because it seemed helpful to me
and others. I use it regularly for Crash Stats investigations and reprocessing.
However, I’m not you! If you have any problems, please write up an issue in the
issue tracker and I’ll get to it when I’m able.If you use this, like it, appreciate it, or have any positive feeling, please
give it a GitHub star. That helps me:have a proxy for knowing whether it’s being usedprioritize my time working on this projecthave a hand-wavey list of users in case I have questions and need to ask
for adviceToolssupersearchPerform Super Search queries on Crash Stats.Fetch 10 crash ids for Firefox:$ supersearch --num=10 --product=FirefoxFetch all crash reports that havelibc2.30.so/E22A1E7AEF7C58504AF2C60A5AD3A7AE0in themodules_in_stackfield:$ supersearch --modules_in_stack=libc2.30.so/E22A1E7AEF6C58504AF2C60A5AD3A7AE0This is helpful when you need to reprocess crash reports after uploading symbols
for a module that we didn’t have symbols for.Fetch all crash reports that havelibfenix.soin themodules_in_stackfield:$ supersearch --verbose --modules_in_stack='^libfenix.so'Fetch 57 crash ids that match a Super Search query:$ supersearch --num=57 --supersearch-url='https://crash-stats.mozilla.org/search/?release_channel=nightly&version=70.0a1&product=Firefox&_sort=-date'Use single quotes for values so that your shell doesn’t do any shell expansion.Fetch uuid, product, version, and build_id for crash reports that have “OOM” in
the signature:$ supersearch --_columns=uuid --_columns=product --_columns=version \
    --_columns=build_id --signature='~OOM'Results are formatted as tab-delimited by default. JSON output is also
available.Note that this doesn’t support Super Search aggregations–usesupersearchfacetfor that.See Super Search API documentation for details on notation and fields:https://crash-stats.mozilla.org/documentation/supersearch/https://crash-stats.mozilla.org/documentation/supersearch/api/supersearchfacetPerforms facets and daily-facets.See the breakdown of crash reports by product for the last 7 days:$ supersearchfacet --_facets=productSee crashes broken down by product and down by day for the last 7 days:$ supersearchfacet --daily --relative-range=7d --_facets=productSee just Firefox crashes broken down by day for the last 14 days:$ supersearchfacet --daily --relative-range=14d --_facets=product --product=FirefoxResults are formatted as tab-delimited by default. Markdown and JSON output are
also available.Get the table in Markdown for easy cut-and-paste into Markdown things:$ supersearchfacet --daily --format=markdown --relative-range=14d --_facets=product \
    --product=FirefoxSee Super Search API documentation for details on notation and fields:https://crash-stats.mozilla.org/documentation/supersearch/https://crash-stats.mozilla.org/documentation/supersearch/api/fetch-dataFetch data for specified crash reports.This lets you download raw crash, dumps, and processed crash from Crash Stats.Fetch processed crash data for specific crash id:$ fetch-data --no-raw --no-dumps --processed 723cacd6-1684-420e-a1c7-f04240190731Fetch raw crash data using supersearch command to generate crash ids:$ supersearch --product=Firefox --num=10 | \
    fetch-data --raw --no-dumps --no-processed crashdirreprocessLet’s you specify crash reports for reprocessing.Reprocess an individual crash report:$ reprocess 723cacd6-1684-420e-a1c7-f04240190731Reprocess crash reports based on a supersearch:$ supersearch --num=5 | reprocessNoteThereprocesscommand requires that you setCRASHSTATS_API_TOKENin
your environment with an API token that has the “Reprocess Crashes”
permission.NoteIf you intend to reprocess more than 10,000 crash reports, please tell
us first.API tokenForsupersearchandfetch-data, you need to use a API token to:download data containing personally identifiable informationdownload security sensitive dataget out from the shadow of extreme API use rate limitingYou need an API token forreprocess–it doesn’t work without one.If you have access, you can generate an API token here:https://crash-stats.mozilla.org/api/tokens/Once you have acquired one, set theCRASHSTATS_API_TOKENenvironment
variable when using crashstats-tools commands.Remember to abide by the data access policy when using data from Crash Stats!
The policy is specified here:https://crash-stats.mozilla.org/documentation/memory_dump_access/Use casesThese tools are helpful when downloading data for analysis as well as
downloading data to test other tools with.Example 1I want to collect a bunch of crash report data to look at possible values of an
annotation in Firefox crash reports that’s not available in Super Search, yet.Since I’m looking just at annotations, all I need is the raw crash.I would do something like this:$ mkdir crashdata
$ supersearch --product=Firefox --num=1000 | \
    fetch-data --raw --no-dumps --no-processed crashdataThen I can usejqor whatever to look at the crash report data incrashdata/raw_crash/.Example 2I want to test out a new JIT analysis tool that works on minidump files.I would write a script like this:#!/bin/bash

CRASHSTATS_API_TOKEN=foo
DATADIR=./crashdata
CRASHIDS=$(supersearch --product=Firefox --num=1000)

mkdir -p "${DATADIR}"

for crashid in ${CRASHIDS}
do
    echo "crashid ${crashid}"
    fetch-data --raw --dumps --no-processed "${DATADIR}" "${crashid}"

    # Not all crash reports have dumps--we only want to run analysis
    # on the ones that do.
    if [[ -e "crashdata/dump/${crashid}" ]]
    then
        echo "analyze dump ${crashid}..."
        # run my tool on the dump
    fi
doneExample 3I want to get a list of crash ids for today (2019-07-30) whereDOMFissionEnabledexists in the crash report.I would do this:$ supersearch --date=">=2019-07-30" --date='<2019-07-31' \
    --dom_fission_enabled='!__null__'Example 4I want to see number of crash reports for the last 14 days broken down by day
and by product whereDOMFissionEnabledexists in the crash report.I would do this:$ supersearchfacet --period=daily --format=markdown --relative-range=14d \
    --dom_fission_enabled='!__null__' --_facets=productPrior art and related projectshttps://github.com/mozilla/libmozdataPython library which has aSupersearchclass for performing queries and
aProcessedCrashclass for fetching processed crash data.https://github.com/mozilla-services/socorroSocorro (the code base for Crash Stats) has a Docker-based local dev
environment which includes a series of commands for manipulating data.https://socorro.readthedocs.io/en/latest/service/processor.html#processing-crashes