Package: bc-detect-secrets
detect-secretsNoticeThis is a fork of thedetect-secretsrepo by Yelp and is
officially supported by Bridgecrew.Aboutdetect-secretsis an aptly named module for (surprise, surprise)detecting secretswithin a
code base.However, unlike other similar packages that solely focus on finding secrets, this package is
designed with the enterprise client in mind: providing abackwards compatible, systematic
means of:Preventing new secrets from entering the code base,Detecting if such preventions are explicitly bypassed, andProviding a checklist of secrets to roll, and migrate off to a more secure storage.This way, you create aseparation of concern:
accepting that there maycurrentlybe secrets hiding in your large repository
(this is what we refer to as abaseline), but preventing this issue from getting any larger,
without dealing with the potentially gargantuan effort of moving existing secrets away.It does this by running periodic diff outputs against heuristically crafted regex statements,
to identify whether anynewsecret has been committed. This way, it avoids the overhead of
digging through all git history, as well as the need to scan the entire repository every time.For a look at recent changes, please seeCHANGELOG.md.If you are looking to contribute, please seeCONTRIBUTING.md.For more detailed documentation, check out our otherdocumentation.ExamplesQuickstart:Create a baseline of potential secrets currently found in your git repository.$detect-secretsscan>.secrets.baselineor, to run it from a different directory:$detect-secrets-C/path/to/directoryscan>/path/to/directory/.secrets.baselineScanning non-git tracked files:$detect-secretsscantest_data/--all-files>.secrets.baselineAdding New Secrets to Baseline:This will rescan your codebase, and:Update/upgrade your baseline to be compatible with the latest version,Add any new secrets it finds to your baseline,Remove any secrets no longer in your codebaseThis will also preserve any labelled secrets you have.$detect-secretsscan--baseline.secrets.baselineFor baselines older than version 0.9, just recreate it.Alerting off newly added secrets:Scanning Staged Files Only:$gitdiff--staged--name-only-z|xargs-0detect-secrets-hook--baseline.secrets.baselineScanning All Tracked Files:$gitls-files-z|xargs-0detect-secrets-hook--baseline.secrets.baselineViewing All Enabled Plugins:$detect-secretsscan--list-all-plugins
ArtifactoryDetector
AWSKeyDetector
AzureStorageKeyDetector
BasicAuthDetector
CloudantDetector
DiscordBotTokenDetector
GitHubTokenDetector
Base64HighEntropyString
HexHighEntropyString
IbmCloudIamDetector
IbmCosHmacDetector
JwtTokenDetector
KeywordDetector
MailchimpDetector
NpmDetector
PrivateKeyDetector
SendGridDetector
SlackDetector
SoftlayerDetector
SquareOAuthDetector
StripeDetector
TwilioKeyDetectorDisabling Plugins:$detect-secretsscan--disable-pluginKeywordDetector--disable-pluginAWSKeyDetectorIf you want toonlyrun a specific plugin, you can do:$detect-secretsscan--list-all-plugins|\grep-v'BasicAuthDetector'|\sed"s#^#--disable-plugin #g"|\xargsdetect-secretsscantest_dataAuditing a Baseline:This is an optional step to label the results in your baseline. It can be used to narrow down your
checklist of secrets to migrate, or to better configure your plugins to improve its signal-to-noise
ratio.$detect-secretsaudit.secrets.baselineUsage in Other Python ScriptsBasic Use:fromdetect_secretsimportSecretsCollectionfromdetect_secrets.settingsimportdefault_settingssecrets=SecretsCollection()withdefault_settings():secrets.scan_file('test_data/config.ini')importjsonprint(json.dumps(secrets.json(),indent=2))More Advanced Configuration:fromdetect_secretsimportSecretsCollectionfromdetect_secrets.settingsimporttransient_settingssecrets=SecretsCollection()withtransient_settings({# Only run scans with only these plugins.# This format is the same as the one that is saved in the generated baseline.'plugins_used':[# Example of configuring a built-in plugin{'name':'Base64HighEntropyString','limit':5.0,},# Example of using a custom plugin{'name':'HippoDetector','path':'file:///Users/aaronloo/Documents/github/detect-secrets/testing/plugins.py',},],# We can also specify whichever additional filters we want.# This is an example of using the function `is_identified_by_ML_model` within the# local file `./private-filters/example.py`.'filters_used':[{'path':'file://private-filters/example.py::is_identified_by_ML_model',},]})assettings:# If we want to make any further adjustments to the created settings object (e.g.# disabling default filters), we can do so as such.settings.disable_filters('detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign','detect_secrets.filters.heuristic.is_likely_id_string',)secrets.scan_file('test_data/config.ini')Installation$pipinstalldetect-secrets
✨🍰✨Install viabrew:$brewinstalldetect-secretsUsagedetect-secretscomes with three different tools, and there is often confusion around which one
to use. Use this handy checklist to help you decide:Do you want to add secrets to your baseline? If so, usedetect-secrets scan.Do you want to alert off new secrets not in the baseline? If so, usedetect-secrets-hook.Are you analyzing the baseline itself? If so, usedetect-secrets audit.Adding Secrets to Baseline$ detect-secrets scan --help
usage: detect-secrets scan [-h] [--string [STRING]] [--only-allowlisted]
                           [--all-files] [--baseline FILENAME]
                           [--force-use-all-plugins] [--slim]
                           [--list-all-plugins] [-p PLUGIN]
                           [--base64-limit [BASE64_LIMIT]]
                           [--hex-limit [HEX_LIMIT]]
                           [--disable-plugin DISABLE_PLUGIN]
                           [-n | --only-verified]
                           [--exclude-lines EXCLUDE_LINES]
                           [--exclude-files EXCLUDE_FILES]
                           [--exclude-secrets EXCLUDE_SECRETS]
                           [--word-list WORD_LIST_FILE] [-f FILTER]
                           [--disable-filter DISABLE_FILTER]
                           [path [path ...]]

Scans a repository for secrets in code. The generated output is compatible
with `detect-secrets-hook --baseline`.

positional arguments:
  path                  Scans the entire codebase and outputs a snapshot of
                        currently identified secrets.

optional arguments:
  -h, --help            show this help message and exit
  --string [STRING]     Scans an individual string, and displays configured
                        plugins' verdict.
  --only-allowlisted    Only scans the lines that are flagged with `allowlist
                        secret`. This helps verify that individual exceptions
                        are indeed non-secrets.

scan options:
  --all-files           Scan all files recursively (as compared to only
                        scanning git tracked files).
  --baseline FILENAME   If provided, will update existing baseline by
                        importing settings from it.
  --force-use-all-plugins
                        If a baseline is provided, detect-secrets will default
                        to loading the plugins specified by that baseline.
                        However, this may also mean it doesn't perform the
                        scan with the latest plugins. If this flag is
                        provided, it will always use the latest plugins
  --slim                Slim baselines are created with the intention of
                        minimizing differences between commits. However, they
                        are not compatible with the `audit` functionality, and
                        slim baselines will need to be remade to be audited.

plugin options:
  Configure settings for each secret scanning ruleset. By default, all
  plugins are enabled unless explicitly disabled.

  --list-all-plugins    Lists all plugins that will be used for the scan.
  -p PLUGIN, --plugin PLUGIN
                        Specify path to custom secret detector plugin.
  --base64-limit [BASE64_LIMIT]
                        Sets the entropy limit for high entropy strings. Value
                        must be between 0.0 and 8.0, defaults to 4.5.
  --hex-limit [HEX_LIMIT]
                        Sets the entropy limit for high entropy strings. Value
                        must be between 0.0 and 8.0, defaults to 3.0.
  --disable-plugin DISABLE_PLUGIN
                        Plugin class names to disable. e.g.
                        Base64HighEntropyString

filter options:
  Configure settings for filtering out secrets after they are flagged by the
  engine.

  -n, --no-verify       Disables additional verification of secrets via
                        network call.
  --only-verified       Only flags secrets that can be verified.
  --exclude-lines EXCLUDE_LINES
                        If lines match this regex, it will be ignored.
  --exclude-files EXCLUDE_FILES
                        If filenames match this regex, it will be ignored.
  --exclude-secrets EXCLUDE_SECRETS
                        If secrets match this regex, it will be ignored.
  --word-list WORD_LIST_FILE
                        Text file with a list of words, if a secret contains a
                        word in the list we ignore it.
  -f FILTER, --filter FILTER
                        Specify path to custom filter. May be a python module
                        path (e.g.
                        detect_secrets.filters.common.is_invalid_file) or a
                        local file path (e.g.
                        file://path/to/file.py::function_name).
  --disable-filter DISABLE_FILTER
                        Specify filter to disable. e.g.
                        detect_secrets.filters.common.is_invalid_fileBlocking Secrets not in Baseline$ detect-secrets-hook --help
usage: detect-secrets-hook [-h] [-v] [--version] [--baseline FILENAME]
                           [--list-all-plugins] [-p PLUGIN]
                           [--base64-limit [BASE64_LIMIT]]
                           [--hex-limit [HEX_LIMIT]]
                           [--disable-plugin DISABLE_PLUGIN]
                           [-n | --only-verified]
                           [--exclude-lines EXCLUDE_LINES]
                           [--exclude-files EXCLUDE_FILES]
                           [--exclude-secrets EXCLUDE_SECRETS]
                           [--word-list WORD_LIST_FILE] [-f FILTER]
                           [--disable-filter DISABLE_FILTER]
                           [filenames [filenames ...]]

positional arguments:
  filenames             Filenames to check.

optional arguments:
  -h, --help            show this help message and exit
  -v, --verbose         Verbose mode.
  --version             Display version information.
  --json                Print detect-secrets-hook output as JSON
  --baseline FILENAME   Explicitly ignore secrets through a baseline generated
                        by `detect-secrets scan`

plugin options:
  Configure settings for each secret scanning ruleset. By default, all
  plugins are enabled unless explicitly disabled.

  --list-all-plugins    Lists all plugins that will be used for the scan.
  -p PLUGIN, --plugin PLUGIN
                        Specify path to custom secret detector plugin.
  --base64-limit [BASE64_LIMIT]
                        Sets the entropy limit for high entropy strings. Value
                        must be between 0.0 and 8.0, defaults to 4.5.
  --hex-limit [HEX_LIMIT]
                        Sets the entropy limit for high entropy strings. Value
                        must be between 0.0 and 8.0, defaults to 3.0.
  --disable-plugin DISABLE_PLUGIN
                        Plugin class names to disable. e.g.
                        Base64HighEntropyString

filter options:
  Configure settings for filtering out secrets after they are flagged by the
  engine.

  -n, --no-verify       Disables additional verification of secrets via
                        network call.
  --only-verified       Only flags secrets that can be verified.
  --exclude-lines EXCLUDE_LINES
                        If lines match this regex, it will be ignored.
  --exclude-files EXCLUDE_FILES
                        If filenames match this regex, it will be ignored.
  --exclude-secrets EXCLUDE_SECRETS
                        If secrets match this regex, it will be ignored.
  -f FILTER, --filter FILTER
                        Specify path to custom filter. May be a python module
                        path (e.g.
                        detect_secrets.filters.common.is_invalid_file) or a
                        local file path (e.g.
                        file://path/to/file.py::function_name).
  --disable-filter DISABLE_FILTER
                        Specify filter to disable. e.g.
                        detect_secrets.filters.common.is_invalid_fileWe recommend setting this up as a pre-commit hook. One way to do this is by using thepre-commitframework:# .pre-commit-config.yamlrepos:-repo:https://github.com/Yelp/detect-secretsrev:v1.4.0hooks:-id:detect-secretsargs:['--baseline','.secrets.baseline']exclude:package.lock.jsonInline AllowlistingThere are times when we want to exclude a false positive from blocking a commit, without creating
a baseline to do so. You can do so by adding a comment as such:secret="hunter2"# pragma: allowlist secretor//  pragma: allowlist nextline secretconstsecret="hunter2";Auditing Secrets in Baseline$detect-secretsaudit--help
usage:detect-secretsaudit[-h][--diff][--stats][--report][--only-real|--only-false][--json]filename[filename...]Auditingabaselineallowsanalyststolabelresults,andoptimizepluginsforthehighestsignal-to-noiseratiofortheirenvironment.

positionalarguments:filenameAuditagivenbaselinefiletodistinguishthedifferencebetweenfalseandtruepositives.

optionalarguments:-h,--helpshowthishelpmessageandexit--diffAllowsthecomparisonoftwobaselinefiles,inordertoeffectivelydistinguishthedifferencebetweenvariouspluginconfigurations.--statsDisplaystheresultsofaninteractiveauditingsessionwhichhavebeensavedtoabaselinefile.--reportDisplaysareportwiththesecretsdetected

reporting:Displayasummarywithallthefindingsandthemadedecisions.Tobeusedwiththereportmode(--report).--only-realOnlyincludesrealsecretsinthereport--only-falseOnlyincludesfalsepositivesinthereport

analytics:Quantifythesuccessofyourpluginsbasedonthelabelledresultsinyourbaseline.Tobeusedwiththestatisticsmode(--stats).--jsonOutputsresultsinamachine-readableformat.ConfigurationThis tool operates through a system ofpluginsandfilters.Pluginsfind secrets in codeFiltersignore false positives to increase scanning precisionYou can adjust both to suit your precision/recall needs.PluginsThere are three different strategies we employ to try and find secrets in code:Regex-based RulesThese are the most common type of plugin, and work well with well-structured secrets.
These secrets can optionally beverified, which increases
scanning precision. However, solely depending on these may negatively affect the recall of your
scan.Entropy DetectorThis searches for "secret-looking" strings through a variety of heuristic approaches. This
is great for non-structured secrets, but may require tuning to adjust the scanning precision.Keyword DetectorThis ignores the secret value, and searches for variable names that are often associated with
assigning secrets with hard-coded values. This is great for "non-secret-looking" strings (e.g.
le3tc0de passwords), but may require tuning filters to adjust the scanning precision.Want to find a secret that we don't currently catch? You can also (easily) develop your own
plugin, and use it with the engine! For more information, check out theplugin documentation.Filtersdetect-secretscomes with several different in-built filters that may suit your needs.--exclude-linesSometimes, you want to be able to globally allow certain lines in your scan, if they match a
specific pattern. You can specify a regex rule as such:$detect-secretsscan--exclude-lines'password = (blah|fake)'Or you can specify multiple regex rules as such:$detect-secretsscan--exclude-lines'password = blah'--exclude-lines'password = fake'--exclude-filesSometimes, you want to be able to ignore certain files in your scan. You can specify a regex
pattern to do so, and if the filename meets this regex pattern, it will not be scanned:$detect-secretsscan--exclude-files'.*\.signature$'Or you can specify multiple regex patterns as such:$detect-secretsscan--exclude-files'.*\.signature$'--exclude-files'.*/i18n/.*'--exclude-secretsSometimes, you want to be able to ignore certain secret values in your scan. You can specify
a regex rule as such:$detect-secretsscan--exclude-secrets'(fakesecret|\${.*})'Or you can specify multiple regex rules as such:$detect-secretsscan--exclude-secrets'fakesecret'--exclude-secrets'\${.*})'Inline AllowlistingSometimes, you want to apply an exclusion to a specific line, rather than globally excluding it.
You can do so with inline allowlisting as such:API_KEY='this-will-ordinarily-be-detected-by-a-plugin'# pragma: allowlist secretThese comments are supported in multiple languages. e.g.constGoogleCredentialPassword="something-secret-here";//  pragma: allowlist secretYou can also use:# pragma: allowlist nextline secretAPI_KEY='WillAlsoBeIgnored'This may be a convenient way for you to ignore secrets, without needing to regenerate the entire
baseline again. If you need to explicitly search for these allowlisted secrets, you can also do:$detect-secretsscan--only-allowlistedWant to write more custom logic to filter out false positives? Check out how to do this in
ourfilters documentation.ExtensionswordlistThe--exclude-secretsflag allows you to specify regex rules to exclude secret values. However,
if you want to specify a large list of words instead, you can use the--word-listflag.To use this feature, be sure to install thepyahocorasickpackage, or simply use:$pipinstalldetect-secrets[word_list]Then, you can use it as such:$catwordlist.txt
not-a-real-secret
$catsample.inipassword=not-a-real-secret# Will show results$detect-secretsscansample.ini# No results found$detect-secretsscan--word-listwordlist.txtGibberish DetectorThe Gibberish Detector is a simple ML model, that attempts to determine whether a secret value
is actually gibberish, with the assumption thatrealsecret values are not word-like.To use this feature, be sure to install thegibberish-detectorpackage, or use:$pipinstalldetect-secrets[gibberish]Check out thegibberish-detectorpackage for
more information on how to train the model. A pre-trained model (seeded by processing RFCs) will
be included for easy use.You can also specify your own model as such:$detect-secretsscan--gibberish-modelcustom.modelThis is not a default plugin, given that this will ignore secrets such aspassword.CaveatsThis is not meant to be a sure-fire solution to prevent secrets from entering the codebase. Only
proper developer education can truly do that. This pre-commit hook merely implements several
heuristics to try and prevent obvious cases of committing secrets.Things That Won't Be Prevented:Multi-line secretsDefault passwords that don't trigger theKeywordDetector(e.g.login = "hunter2")FAQGeneral"Did not detect git repository." warning encountered, even though I'm in a git repo.Check to see whether yourgitversion is >= 1.8.5. If not, please upgrade it then try again.More details here.Windowsdetect-secrets auditdisplays "Not a valid baseline file!" after creating baseline.Ensure the file encoding of your baseline file is UTF-8.More details here.