Package: credentialdigger
Credential DiggerCredential Digger is a GitHub scanning tool that identifies hardcoded credentials (Passwords, API Keys, Secret Keys, Tokens, personal information, etc), filtering the false positive data through machine learning models.TLDR; watch the video ⬇️Credential DiggerWhyRequirementsDownload and InstallationHow to runAdd rulesScan a repositoryDocker containerAdvanced InstallationBuild from sourceExternal postgres databaseHow to update the projectPython library usageAdd rulesScan a repositoryCLI - Command Line InterfaceMicosoft Visual Studio Plug-inpre-commit hookCI/CD Pipeline Intergation on PiperWikiContributingHow to obtain supportNewsWhyIn data protection, one of the most critical threats is represented by hardcoded (or plaintext) credentials in open-source projects. Several tools are already available to detect leaks in open-source platforms, but the diversity of credentials (depending on multiple factors such as the programming language, code development conventions, or developers' personal habits) is a bottleneck for the effectiveness of these tools. Their lack of precision leads to a very high number of pieces of code incorrectly detected as leaked secrets. Data wrongly detected as a leak is calledfalse positivedata, and compose the huge majority of the data detected by currently available tools.The goal of Credential Digger is to reduce the amount of false positive data on the output of the scanning phase by leveraging machine learning models.The tool supports several scan flavors: public and private repositories on
github and gitlab, pull requests, wiki pages, github organizations, local git repositories, local files and folders.
Please refer to theWikifor the complete documentation.For the complete description of the approach of Credential Digger (versions <4.4),you can read this publication.@InProceedings {lrnto-icissp21,
    author = {S. Lounici and M. Rosa and C. M. Negri and S. Trabelsi and M. Önen},
    booktitle = {Proc. of the 8th The International Conference on Information Systems Security and Privacy  (ICISSP)},
    title = {Optimizing Leak Detection in Open-Source Platforms with Machine Learning Techniques},
    month = {February},
    day = {11-13},
    year = {2021}
}RequirementsCredential Digger supports Python >= 3.8 and < 3.12, and works only with Linux and MacOS systems.
In case you don't meet these requirements, you may consider running aDocker container(that also includes a user interface).Download and InstallationFirst, you need to install some dependencies (namely,build-essentialandpython3-dev). No need to explicitely install hyperscan anymore.sudoaptinstall-ybuild-essentialpython3-devThen, you can install Credential Digger module usingpip.pipinstallcredentialdiggerHow to runAdd rulesOne of the core components of Credential Digger is the regular expression scanner. You can choose the regular expressions rules you want (just follow the templatehere). We provide a list of patterns in therules.ymlfile, that are included in the UI. The scanner supports rules of 4 different categories:password,token,crypto_key, andother.Before the very first scan, you need to add the rules that will be used by the scanner.This step is only needed once.credentialdiggeradd_rules--sqlite/path/to/data.db/path/to/rules.yamlScan a repositoryAfter adding the rules, you can scan a repository:credentialdiggerscanhttps://github.com/user/repo--sqlite/path/to/data.dbMachine learning models are not mandatory, but highly recommended in order to reduce the manual effort of reviewing the result of a scan:credentialdiggerscanhttps://github.com/user/repo--sqlite/path/to/data.db--modelsPathModelPasswordModelAs for the models, also the similarity feature is not mandatory, but highly recommended in order to reduce the manual effort while assessing the discoveries after a scan:credentialdiggerscanhttps://github.com/user/repo--sqlite/path/to/data.db--similarity--modelsPathModelPasswordModelDocker containerTo have a ready-to-use instance of Credential Digger, with a user interface, you can use a docker container.
This option requires the installation ofDockerandDocker Compose.Credential Digger is published ondockerhub. You can pull the latest releasesudodockerpullsaposs/credentialdiggerOr build and run containers with docker-composegitclonehttps://github.com/SAP/credential-digger.gitcdcredential-digger
cp.env.sample.env
sudodocker-composeup--buildThe UI is available athttp://localhost:5000/It is preferrable to have at least 8 GB of RAM free when using docker containersAdvanced InstallationCredential Digger is modular, and offers a wide choice of components and adaptations.Build from sourceAfter installing thedependencieslisted above, you can install Credential Digger as follows.Configure a virtual environment for Python 3 (optional) and clone the main branch of the project:virtualenv-ppython3./venvsource./venv/bin/activate

gitclonehttps://github.com/SAP/credential-digger.gitcdcredential-diggerInstall the tool from source:pipinstall.Then, you can add the rules and scan a repository as described above.External postgres databaseAnother ready-to-use instance of Credential Digger with the UI, but using a dockerized postgres database instead of a local sqlite one:gitclonehttps://github.com/SAP/credential-digger.gitcdcredential-digger
cp.env.sample.env
vim.env# set credentials for postgressudodocker-compose-fdocker-compose.postgres.ymlup--buildWARNING: Differently from the sqlite version, here we need to configure the.envfile with the credentials for postgres (by modifyingPOSTGRES_USER,POSTGRES_PASSWORDandPOSTGRES_DB).Most advanced users may also wish to use an external postgres database instead of the dockerized one we provide in ourdocker-compose.postgres.yml.How to update the projectIf you are already running Credential Digger and you want to update it to a
newer version, you canrefer to the wiki for the needed steps.Python library usageWhen installingcredentialdiggerfrom pip (or from source), you can instantiate the client and scan a repository.Instantiate the client proper for the chosen database:# Using a Sqlite databasefromcredentialdiggerimportSqliteClientc=SqliteClient(path='/path/to/data.db')# Using a postgres databasefromcredentialdiggerimportPgClientc=PgClient(dbname='my_db_name',dbuser='my_user',dbpassword='my_password',dbhost='localhost_or_ip',dbport=5432)Add rulesAdd rules before launching your first scan.c.add_rules_from_file('/path/to/rules.yml')Scan a repositorynew_discoveries=c.scan(repo_url='https://github.com/user/repo',models=['PathModel','PasswordModel'],debug=True)WARNING: Make sure you add the rules before your first scan.Please refer to theWikifor further information on the arguments.CLI - Command Line InterfaceCredential Digger also offers a simple CLI to scan a repository. The CLI supports both sqlite and postgres databases. In case of postgres, you need either to export the credentials needed to connect to the database as environment variables or to setup a.envfile. In case of sqlite, the path of the db must be passed as argument.Refer to theWikifor all the supported commands and their usage.Micosoft Visual Studio PluginVS Code extension for project "Credential Digger" is a free IDE extension that let you detect secrets and credentials in your code before they get leaked! Like a spell checker, the extension scans your files using the Credential Digger and highlights the secrets as you write code, so you can fix them before the code is even committed.The VS Code extension can be donwloaded from theMicrosoft VS Code Marketplacepre-commit hookCredential Digger can be used with thepre-commitframework to scan staged files before each commit.Please, refer to theWiki page of the pre-commit hookfor further information on its installation and execution.CI/CD Pipeline Intergation on PiperCredential Digger is intergrated with the continuous delivery CI/CD pipelinePiperin order to automate secrets scans for your Github projects and repositories.
In order to activate the Credential Diggger Step please refer to thisCredential Digger step documentation for PiperHow Piper works with JenkinsOnce the step for credentialdigger is reached, its docker image is downloaded from the internal SAP registry. (A public instance will be avaialble soon)Jenkins runs this container and runs a scan using credentialdigger, based on the step configuration. Indeed, the step supports full scan of a repo, scan of a snapshot and scan of a pull request. It is also supporting orchestrators.The result of the scan (an excel file) is stored in Jenkins workspace as an output artifactJenkins destroys the container after the scanThere is no need to deploy or install a Credential Digger instance !!WikiFor further information, please refer to theWikiContributingWe invite your participation to the project through issues and pull requests. Please refer to theContributing guidelinesfor how to contribute.How to obtain supportAs a first step, we suggest toread the wiki.
In case you don't find the answers you need, you can open anissueor contact themaintainers.NewsCredential Digger announcementCredential Digger is now supporting Keras machine learning modelsCredential Digger approach has been published at ICISSP 2021 conference