Package: trailscraper
TrailScraperA command-line tool to get valuable information out of AWS CloudTrail
and a general purpose toolbox for working with IAM policiesInstallationOSX$brewinstalltrailscraperInstallation using pipRequirements:Python >= 3.5pip$pipinstalltrailscraperRun directly using docker$dockerrun--rm--env-file<(env|grepAWS_)-v$HOME/.aws:/root/.awsghcr.io/flosell/trailscraper:latestCurrent Versions starting from 0.7.0 are found on GitHub Container
Registry (ghcr.io), older versions onDockerHubUsageGet CloudTrail events matching a filter from CloudTrail
APIDownload some logsDownload some logs in organisational
trailsFind CloudTrail events matching a filter in downloaded
logsGenerate Policy from some CloudTrail
recordsExtend existing policy by guessing matching
actionsFind CloudTrail events and generate an IAM
PolicyGet CloudTrail events matching a filter from CloudTrail API$ trailscraper select --use-cloudtrail-api \
                      --filter-assumed-role-arn some-arn \
                      --from 'one hour ago' \
                      --to 'now'
{
  "Records": [
    {
      "eventTime": "2017-12-11T15:01:51Z",
      "eventSource": "autoscaling.amazonaws.com",
      "eventName": "DescribeLaunchConfigurations",
...Download some logs$ trailscraper download --bucket some-bucket \
                        --account-id some-account-id \
                        --region some-other-region \
                        --region us-east-1 \
                        --from 'two days ago' \
                        --to 'now' \Note: Include us-east-1 to download logs for global services. Seebelowfor detailsDownload some logs in organisational trails$ trailscraper download --bucket some-bucket \
                        --account-id some-account-id \
                        --region us-east-1 \
                        --org-id o-someorgid \
                        --from 'two days ago' \
                        --to 'now'Find CloudTrail events matching a filter in downloaded logs$ trailscraper select --filter-assumed-role-arn some-arn \
                      --from 'one hour ago' \
                      --to 'now'
{
  "Records": [
    {
      "eventTime": "2017-12-11T15:01:51Z",
      "eventSource": "autoscaling.amazonaws.com",
      "eventName": "DescribeLaunchConfigurations",
...Generate Policy from some CloudTrail records$ gzcat some-records.json.gz | trailscraper generate
{
    "Statement": [
        {
            "Action": [
                "ec2:DescribeInstances"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}Extend existing policy by guessing matching actionsCloudTrail logs might not always contain all relevant actions. For
example, your logs might only contain theCreateactions after a
terraform run when you really want the delete and update permissions as
well. TrailScraper can try to guess additional statements that might be
relevant:$ cat minimal-policy.json | trailscraper guess
{
    "Statement": [
        {
            "Action": [
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListObjects"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}
$ cat minimal-policy.json | ./go trailscraper guess --only Get
{
    "Statement": [
        {
            "Action": [
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}Find CloudTrail events and generate an IAM Policy$ trailscraper select | trailscraper generate
{
    "Statement": [
        {
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:iam::1111111111:role/someRole"
            ]
        }
    ],
    "Version": "2012-10-17"
}FAQHow can I generate policies in CloudFormation YAML instead of JSON?TrailScraper doesn’t provide this. But you can usecfn-flipto do
it:$ trailscraper select | trailscraper generate | cfn-flip
Statement:
  - Action:
      - ec2:DescribeInstances
    Effect: Allow
    Resource:
      - '*'How can I generate policies in Terraform HCL instead of JSON?TrailScraper doesn’t provide this. But you can useiam-policy-json-to-terraformto do it:$ trailscraper select | trailscraper generate | iam-policy-json-to-terraform
data "aws_iam_policy_document" "policy" {
  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["*"]

    actions = [
      "ec2:DescribeInstances",
    ]
  }
}Why is TrailScraper missing some events?Make sure you have logs for theus-east-1region. Some global AWS
services (e.g. Route53, IAM, STS, CloudFront) use this region. For
details, check theCloudTrail
DocumentationWhy are some TrailScraper-generated actions not real IAM actions?This is totally possible. Unfortunately, there is no good,
machine-readable documentation on how CloudTrail events map to IAM
actions so TrailScraper is using heuristics to figure out the right
actions. These heuristics likely don’t cover all special cases of the
AWS world.This is where you come in: If you find a special case that’s not covered
by TrailScraper, pleaseopen a new
issueor, even
better, submit a pull request.For more details, check out thecontribution
guideWhy does click think I am in an ASCII environment?Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.Set environment variables that describe your locale, e.g. :export LC_ALL=de_DE.utf-8
export LANG=de_DE.utf-8orLC_ALL=C.UTF-8
LANG=C.UTF-8For details, seehttp://click.pocoo.org/5/python3/#python-3-surrogate-handlingDevelopment$./gosetup# set up venv, dependencies and tools$./gotest# run some tests$./gocheck# run some style checks$./go# let's see what we can do hereChangelogThis changelog contains a loose collection of changes in every release
including breaking changes to the API.The format is based onKeep a Changelog0.8.1ChangedUpdated supporting dependencies0.8.0AddedSupport for Python 3.10Support for Python 3.11RemovedRemoved official support for EOL Python 3.6. TrailScraper might
still run but we no longer actively test for it.0.7.0AddedSupport for Python 3.9RemovedRemoved official support for Python 3.5. TrailScraper might still
run but we no longer actively test for itChangedMoving from DockerHub to GitHub Container registry for Docker images
(since DockerHub is dropping support for autobuilds and restricting
it in other ways)Updated supporting dependencies0.6.4FixedFixed Docker images that threw aModuleNotFoundErrorChangedPerformance tweakstrailscraper downloaduses smarter directory listing to
improve performance with large date ranges and little new datatrailscraper downloadnow downloads files in parallelMinor performance improvements intrailscraper select0.6.2 and 0.6.3(skipeed because of continuing release-script issues)0.6.1(same as 0.6.1, just fixing inconsistent release)0.6.0AddedSupport for Python 3.7 and 3.8Support for org-level trails (#101)Fixedtrailscraper guesswas not working when installed through
homebrew or pip (#110)RemovedRemoved official support for Python 2.7 and 3.4. TrailScraper
might still run but we no longer actively test for it0.5.1AddedNew commandguessto extend existing policy by guessing matching
actions #22FixedFixed parsing events that contain resources without an ARN
(e.g.s3:ListObjects) #510.5.0Breaking CLI changes: split upgenerate-policyintoselectandgenerate(#38)AddedNew commandselectto print all CloudTrail records matching a
filter to stdoutNew commandgenerateto take CloudTrail records from stdin and
generate a policy for itChangedNew commandselectdefaults to not filtering at all whereasgenerate-policyfiltered for recent events by default. Changed to
make filtering more explicit and predictable instead of surprising
users who wonder why their events don’t show upRemovedRemoved commandgenerate-policy, replaced withselectandgenerate. Use pipes to produce the same behavior:$trailscraperselect|trailscrapergenerate0.4.4FixedMade trailscraper timezone-aware. Until now, trailscraper implicitly
treated everything as UTC, meaning relative timestamps (e.g.now,two hours ago) didn’t work properly when filtering logfiles to
download or records to generate from. (#39)AddedNew commandtrailscraperlast-event-timestampto get the last
known event timestamp.New flagtrailscraper download--waitto wait until events for
the specified timeframe are found. Useful if you are waiting for
CloudTrail to ship logs for a recent operation.0.4.3skipped because of release-problems0.4.2FixedFixed various special cases in mapping CloudTrail to IAM Actions:API GatewayApp Stream 2DynamoDB StreamsLexMechanical TurkS3STSTagging0.4.1FixedIgnore record files that can’t be read (e.g. not valid GZIP) in
Python 2.7 (was only working in Python 3.* before)Fixed permissions generated for services that include the API version
date (e.g. Lambda, CloudFront) (#20)0.4.0AddedSupport for CloudTraillookup_eventsAPI that allows users to
generate a policy without downloading logs from an S3 bucket. Note
that this API only returns“create, modify, and delete API
calls”trailscraper downloadnow supports--fromand--toflags
to specify the timeframe that should be downloaded. Accepts precise
(e.g. “2017-10-12”) and relative (e.g. “-2days”) arguments.trailscrapergenerate-policynow supports--fromand--toto filter events to consider for the generated policy. Accepts
precise (e.g. “2017-10-12”) and relative (e.g. “-2days”) arguments.Performance optimizations:generate-policyonly reads logfiles
for the timeframe requestedAdded--versioncommand line argumentChangedSet more flexible dependenciesRemovedRemoved--past-daysparameter intrailscraper download. Was
replaced by--fromand--to(see above)FixedIgnore record files that can’t be read (e.g. not valid GZIP)0.3.0AddedSupport for Python >= 2.7ChangedDo not download CloudTrail Logs from S3 if they already exist in the
target folder (#9)Removed dependency on fork of the awacs-library to simplify
installation and developmentFixedBug that led to policy-statements with the same set of actions not
being combined properly in some cases (#7)0.2.0AddedBasic filtering for role-arns when generating policy (#3)0.1.0Initial ReleaseAddedBasic feature to download CloudTrail Logs from S3 for certain
accounts and timeframeBasic feature to generate IAM Policies from a set of downloaded
CloudTrail logs