Package: s3tk
s3tkA security toolkit for Amazon S3Another day, another leaky Amazon S3 bucket— The Register, 12 Jul 2017Don’t bethe...next...big...data...leak:tangerine: Battle-tested atInstacartInstallationRun:pipinstalls3tkYou can use theAWS CLIorAWS Vaultto set up your AWS credentials:pipinstallawscli
awsconfigureSeeIAM policiesneeded for each command.CommandsScanScan your buckets for:ACL open to publicpolicy open to publicpublic access blockedlogging enabledversioning enableddefault encryption enableds3tkscanOnly run on specific bucketss3tkscanmy-bucketmy-bucket-2Also works with wildcardss3tkscan"my-bucket*"Confirm correct log bucket(s) and prefixs3tk scan --log-bucket my-s3-logs --log-bucket other-region-logs --log-prefix "{bucket}/"Check CloudTrail object-level logging [experimental]s3tkscan--object-level-loggingSkip logging, versioning, or default encryptions3tkscan--skip-logging--skip-versioning--skip-default-encryptionGet email notifications of failures (via SNS)s3tkscan--sns-topicarn:aws:sns:...List PolicyList bucket policiess3tklist-policyOnly run on specific bucketss3tklist-policymy-bucketmy-bucket-2Show named statementss3tklist-policy--namedSet PolicyNote:This replaces the previous policyOnly private uploadss3tkset-policymy-bucket--no-object-aclDelete PolicyDelete policys3tkdelete-policymy-bucketBlock Public AccessBlock public access on specific bucketss3tkblock-public-accessmy-bucketmy-bucket-2Use the--dry-runflag to testEnable LoggingEnable logging on all bucketss3tkenable-logging--log-bucketmy-s3-logsOnly on specific bucketss3tkenable-loggingmy-bucketmy-bucket-2--log-bucketmy-s3-logsSet log prefix ({bucket}/by default)s3tkenable-logging--log-bucketmy-s3-logs--log-prefix"logs/{bucket}/"Use the--dry-runflag to testA few notes about logging:buckets with logging already enabled are not updated at allthe log bucket must in the same region as the source bucket - run this command multiple times for different regionsit can take over an hour for logs to show upEnable VersioningEnable versioning on all bucketss3tkenable-versioningOnly on specific bucketss3tkenable-versioningmy-bucketmy-bucket-2Use the--dry-runflag to testEnable Default EncryptionEnabledefault encryptionon all bucketss3tkenable-default-encryptionOnly on specific bucketss3tkenable-default-encryptionmy-bucketmy-bucket-2This does not encrypt existing objects - use theencryptcommand for thisUse the--dry-runflag to testScan Object ACLScan ACL on all objects in a buckets3tkscan-object-aclmy-bucketOnly certain objectss3tkscan-object-aclmy-bucket--only"*.pdf"Except certain objectss3tkscan-object-aclmy-bucket--except"*.jpg"Reset Object ACLReset ACL on all objects in a buckets3tkreset-object-aclmy-bucketThis makes all objects private. Seebucket policiesfor how to enforce going forward.Use the--dry-runflag to testSpecify certain objects the same way asscan-object-aclEncryptEncrypt all objects in a bucket withserver-side encryptions3tkencryptmy-bucketUseS3-managed keysby default. ForKMS-managed keys, use:s3tkencryptmy-bucket--kms-key-idarn:aws:kms:...Forcustomer-provided keys, use:s3tkencryptmy-bucket--customer-keysecret-keyUse the--dry-runflag to testSpecify certain objects the same way asscan-object-aclNote:Objects will lose any custom ACLDelete Unencrypted VersionsDelete all unencrypted versions of objects in a buckets3tkdelete-unencrypted-versionsmy-bucketFor safety, this will not delete any current versions of objectsUse the--dry-runflag to testSpecify certain objects the same way asscan-object-aclScan DNSScan Route 53 for buckets to make sure you own thems3tkscan-dnsOtherwise, you may be susceptible tosubdomain takeoverCredentialsCredentials can be specified in~/.aws/credentialsor with environment variables. Seethis guidefor an explanation of environment variables.You can specify a profile to use with:AWS_PROFILE=your-profiles3tkIAM PoliciesHere are the permissions needed for each command. Only include statements you need.{"Version":"2012-10-17","Statement":[{"Sid":"Scan","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:GetBucketAcl","s3:GetBucketPolicy","s3:GetBucketPublicAccessBlock","s3:GetBucketLogging","s3:GetBucketVersioning","s3:GetEncryptionConfiguration"],"Resource":"*"},{"Sid":"ScanObjectLevelLogging","Effect":"Allow","Action":["cloudtrail:ListTrails","cloudtrail:GetTrail","cloudtrail:GetEventSelectors","s3:GetBucketLocation"],"Resource":"*"},{"Sid":"ScanDNS","Effect":"Allow","Action":["s3:ListAllMyBuckets","route53:ListHostedZones","route53:ListResourceRecordSets"],"Resource":"*"},{"Sid":"ListPolicy","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:GetBucketPolicy"],"Resource":"*"},{"Sid":"SetPolicy","Effect":"Allow","Action":["s3:PutBucketPolicy"],"Resource":"*"},{"Sid":"DeletePolicy","Effect":"Allow","Action":["s3:DeleteBucketPolicy"],"Resource":"*"},{"Sid":"BlockPublicAccess","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:PutBucketPublicAccessBlock"],"Resource":"*"},{"Sid":"EnableLogging","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:PutBucketLogging"],"Resource":"*"},{"Sid":"EnableVersioning","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:PutBucketVersioning"],"Resource":"*"},{"Sid":"EnableDefaultEncryption","Effect":"Allow","Action":["s3:ListAllMyBuckets","s3:PutEncryptionConfiguration"],"Resource":"*"},{"Sid":"ResetObjectAcl","Effect":"Allow","Action":["s3:ListBucket","s3:GetObjectAcl","s3:PutObjectAcl"],"Resource":["arn:aws:s3:::my-bucket","arn:aws:s3:::my-bucket/*"]},{"Sid":"Encrypt","Effect":"Allow","Action":["s3:ListBucket","s3:GetObject","s3:PutObject"],"Resource":["arn:aws:s3:::my-bucket","arn:aws:s3:::my-bucket/*"]},{"Sid":"DeleteUnencryptedVersions","Effect":"Allow","Action":["s3:ListBucketVersions","s3:GetObjectVersion","s3:DeleteObjectVersion"],"Resource":["arn:aws:s3:::my-bucket","arn:aws:s3:::my-bucket/*"]}]}Access LogsAmazon Athenais great for querying S3 logs. Create a table (thanks tothis postfor the table structure) with:CREATEEXTERNALTABLEmy_bucket(bucket_ownerstring,bucketstring,timestring,remote_ipstring,requesterstring,request_idstring,operationstring,keystring,request_verbstring,request_urlstring,request_protostring,status_codestring,error_codestring,bytes_sentstring,object_sizestring,total_timestring,turn_around_timestring,referrerstring,user_agentstring,version_idstring)ROWFORMATSERDE'org.apache.hadoop.hive.serde2.RegexSerDe'WITHSERDEPROPERTIES('serialization.format'='1','input.regex'='([^ ]*) ([^ ]*) \\[(.*?)\\] ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) \\\"([^ ]*) ([^ ]*) (- |[^ ]*)\\\" (-|[0-9]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) ([^ ]*) (\"[^\"]*\\") ([^ ]*)$')LOCATION's3://my-s3-logs/my-bucket/';Change the last line to point to your log bucket (and prefix) and query awaySELECTdate_parse(time,'%d/%b/%Y:%H:%i:%S +0000')AStime,request_url,remote_ip,user_agentFROMmy_bucketWHERErequester='-'ANDstatus_codeLIKE'2%'ANDrequest_urlLIKE'/some-keys%'ORDERBY1CloudTrail LogsAmazon Athena is also great for querying CloudTrail logs. Create a table (thanks tothis postfor the table structure) with:CREATEEXTERNALTABLEcloudtrail_logs(eventversionSTRING,userIdentitySTRUCT<type:STRING,principalid:STRING,arn:STRING,accountid:STRING,invokedby:STRING,accesskeyid:STRING,userName:String,sessioncontext:STRUCT<attributes:STRUCT<mfaauthenticated:STRING,creationdate:STRING>,sessionIssuer:STRUCT<type:STRING,principalId:STRING,arn:STRING,accountId:STRING,userName:STRING>>>,eventTimeSTRING,eventSourceSTRING,eventNameSTRING,awsRegionSTRING,sourceIpAddressSTRING,userAgentSTRING,errorCodeSTRING,errorMessageSTRING,requestIdSTRING,eventIdSTRING,resourcesARRAY<STRUCT<ARN:STRING,accountId:STRING,type:STRING>>,eventTypeSTRING,apiVersionSTRING,readOnlyBOOLEAN,recipientAccountIdSTRING,sharedEventIDSTRING,vpcEndpointIdSTRING,requestParametersSTRING,responseElementsSTRING,additionalEventDataSTRING,serviceEventDetailsSTRING)ROWFORMATSERDE'com.amazon.emr.hive.serde.CloudTrailSerde'STOREDASINPUTFORMAT'com.amazon.emr.cloudtrail.CloudTrailInputFormat'OUTPUTFORMAT'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'LOCATION's3://my-cloudtrail-logs/'Change the last line to point to your CloudTrail log bucket and query awaySELECTeventTime,eventName,userIdentity.userName,requestParametersFROMcloudtrail_logsWHEREeventNameLIKE'%Bucket%'ORDERBY1Best PracticesKeep things simple and follow the principle of least privilege to reduce the chance of mistakes.Strictly limit who can perform bucket-related operationsAvoid mixing objects with different permissions in the same bucket (use a bucket policy to enforce this)Don’t specify public read permissions on a bucket level (noGetObjectin bucket policy)Monitor configuration frequently for changesBucket PoliciesOnly private uploads{"Version":"2012-10-17","Statement":[{"Effect":"Deny","Principal":"*","Action":"s3:PutObjectAcl","Resource":"arn:aws:s3:::my-bucket/*"}]}PerformanceFor commands that iterate over bucket objects (scan-object-acl,reset-object-acl,encrypt, anddelete-unencrypted-versions), run s3tk on an EC2 server for minimum latency.NotesTheset-policy,block-public-access,enable-logging,enable-versioning, andenable-default-encryptioncommands are provided for convenience. We recommendTerraformfor managing your buckets.resource"aws_s3_bucket""my_bucket"{bucket="my-bucket"acl="private"logging{target_bucket="my-s3-logs"target_prefix="my-bucket/"}versioning{enabled=true}}resource"aws_s3_bucket_public_access_block""my_bucket"{bucket="${aws_s3_bucket.my_bucket.id}"block_public_acls=trueblock_public_policy=trueignore_public_acls=truerestrict_public_buckets=true}UpgradingRun:pipinstalls3tk--upgradeTo use master, run:pipinstallgit+https://github.com/ankane/s3tk.git--upgradeDockerRun:dockerrun-itankane/s3tkawsconfigureCommit your credentials:dockercommit$(dockerps-l-q)my-s3tkAnd run:dockerrun-itmy-s3tks3tkscanHistoryView thechangelogContributingEveryone is encouraged to help improve this project. Here are a few ways you can help:Report bugsFix bugs andsubmit pull requestsWrite, clarify, or fix documentationSuggest or add new featuresTo get started with development:gitclonehttps://github.com/ankane/s3tk.gitcds3tk
pipinstall-rrequirements.txt