Package: resc-vcs-scanner
Repository Scanner Version Control System Scanner (RESC-VCS-SCANNER)Table of contentsAbout the componentGetting startedPrerequisitesRun locally from sourceRun locally using dockerTestingAbout the componentThe RESC-VCS-Scanner component uses the Gitleaks binary file to scan the source code for secrets.Getting startedThese instructions will help you to get a copy of the project up and running on your local machine for development and testing purposes.PrerequisitesGitDocker DesktopPython 3.9.0Run locally from sourcePreviewPrerequisites:RabbitMQ and RESC web service must be up and running locally.If you have already deployed RESC through helm in Kubernetes, then rabbitmq and resc webservice are already running for you.Install Gitleaksv8.18.0on your system.Download the rule config toml file to/tmp/temp_resc_rule.tomllocation by running below command from a Git Bash terminal.Send some repositories to 'repositories' topics of RabbitMQ server by referring the README of RESC-VCS-SCRAPER component.curlhttps://raw.githubusercontent.com/zricethezav/gitleaks/master/config/gitleaks.toml>/tmp/temp_resc_rule.tomlClone the repository, open the Git Bash terminal from /components/resc-vcs-scanner folder, and run below commands.1. Create virtual environment:cdcomponents/resc-vcs-scanner
pipinstallvirtualenv
virtualenvvenvsourcevenv/Scripts/activate2. Install resc_vcs_scanner package:pipinstall-e.3. Set below environment variables:exportRESC_RABBITMQ_SERVICE_HOST=127.0.0.1#  The hostname/IP address of the rabbitmq serverexportRESC_RABBITMQ_SERVICE_PORT_AMQP=30902#  The amqp port of the rabbitmq serverexportRABBITMQ_DEFAULT_VHOST=resc-rabbitmq#  The virtual host name of the rabbitmq serverexportRABBITMQ_USERNAME=queue_user#  The username used to connect to the rabbitmq projects and repositories topicsexportRABBITMQ_PASSWORD=""# The password used to connect to the rabbitmq projects and repositories topics can be found for the value of queues_password field in /deployment/kubernetes/example-values.yaml fileexportRABBITMQ_QUEUE=repositories# The name of the queue from which secret scanner will read repositoriesexportRESC_API_NO_AUTH_SERVICE_HOST=127.0.0.1#  The hostname/IP address where RESC web service is runningexportRESC_API_NO_AUTH_SERVICE_PORT=30900#  The port number where RESC web service is runningexportVCS_INSTANCES_FILE_PATH=""# The absolute path to vcs_instances_config.json file containing the vcs instances definitionsexportGITHUB_PUBLIC_USERNAME=""# Your GitHub usernameexportGITHUB_PUBLIC_TOKEN=""#  Your GitHub personal access tokenexportGITLEAKS_PATH=""# The absolute path to gitleaks binary executableYou need to replace the following values with your custom values: RABBITMQ_PASSWORD, VCS_INSTANCES_FILE_PATH, GITHUB_PUBLIC_USERNAME, GITHUB_PUBLIC_TOKEN and GITLEAKS_PATH.Structure of vcs instances config jsonThe vcs_instances_config.json file must have the following format:Note:You can add multiple vcs instances.PreviewExample:{"vcs_instance_1":{"name":"GITHUB_PUBLIC","scope":["kubernetes"],"exceptions":[],"provider_type":"GITHUB_PUBLIC","hostname":"github.com","port":"443","scheme":"https","username":"GITHUB_PUBLIC_USERNAME","token":"GITHUB_PUBLIC_TOKEN","organization":""}}scope: List of GitHub accounts you want to scan.
For example, lets'say you want to scan all the repositories for the following GitHub accounts.https://github.com/kuberneteshttps://github.com/dockerThen you need to add those accounts to scope like: ["kubernetes", "docker"]. All the repositories from those accounts will be scanned.exceptions (optional): If you want to exclude any account from scan, then add it to exceptions. Default is empty exception.Theoutputmessages ofcollect_projectscommand has the following format:{"project_key":"kubernetes","vcs_instance_name":"GITHUB_PUBLIC",}4. Run the secret scan task:This task reads the repositories from a RabbitMQ channel called 'repositories', runs scan usingGitleaksand saves the findings' metadata to database.This can be done via the following command:celery-Avcs_scanner.secret_scanners.celery_workerworker--loglevel=INFO-E-Qrepositories--concurrency=1--prefetch-multiplier=1Run locally using dockerPreviewRun the RESC VCS Scanner docker image locally by running the following commands:Pull the docker image from registry:dockerpullrescabnamro/resc-vcs-scanner:latestAlternatively, build the docker image locally by running:dockerbuild-trescabnamro/resc-vcs-scanner:latest.Run the vcs-scanner by using below command:dockerrun-v<pathtovcs_instances_config.jsoninyourlocalsystem>:/tmp/vcs_instances_config.json-eRESC_RABBITMQ_SERVICE_HOST="host.docker.internal"-eRESC_RABBITMQ_SERVICE_PORT_AMQP=30902-eRABBITMQ_DEFAULT_VHOST=resc-rabbitmq-eRABBITMQ_USERNAME=queue_user-eRABBITMQ_PASSWORD="<the password of queue_user>"-eRABBITMQ_QUEUE="repositories"-eRESC_API_NO_AUTH_SERVICE_HOST="host.docker.internal"-eRESC_API_NO_AUTH_SERVICE_PORT=30900-eVCS_INSTANCES_FILE_PATH="/tmp/vcs_instances_config.json"-eGITHUB_PUBLIC_USERNAME="<your github username>"-eGITHUB_PUBLIC_TOKEN="<your github personal access token>"-eGITLEAKS_PATH="/vcs_scanner/gitleaks_config/seco-gitleaks-linux-amd64"--nameresc-vcs-scannerrescabnamro/resc-vcs-scanner:latestcelery-Avcs_scanner.secret_scanners.celery_workerworker--loglevel=INFO-E-Qrepositories--concurrency=1--prefetch-multiplier=1To create vcs_instances_config.json file please refer to:Structure of vcs_instances_config.jsonRun locally as a CLI tool (Still in development)PreviewIt is also possible to run the component as a CLI tool to scan VCS repositories.1. Create virtual environment:cdcomponents/resc-vcs-scanner
pipinstallvirtualenv
virtualenvvenvsourcevenv/bin/activate2. Install resc_vcs_scanner package:pipinstall-e.3. Run CLI scanner:The CLI has 3 modes of operation, please make use of the --help argument to see all the options for the modes:Scanning a non-git directory:secret_scannerdir--help
secret_scannerdir--gitleaks-rules-path=<pathtogitleakstomlrule>--gitleaks-path=<pathtogitleaksbinary>--ignored-blocker-path=<pathtoresc-ignore.dsvfile>--dir=<directorytoscan>Scanning an already cloned git repository:secret_scannerrepolocal--help
secret_scannerrepolocal--gitleaks-rules-path=<pathtogitleakstomlrule>--gitleaks-path=<pathtogitleaksbinary>--ignored-blocker-path=<pathtoresc-ignore.dsvfile>--dir=<directoryofrepositorytoscan>Scanning a remote git repository:secret_scannerreporemote--help
secret_scannerreporemote--gitleaks-rules-path=<pathtogitleakstomlrule>--gitleaks-path=<pathtogitleaksbinary>--ignored-blocker-path=<pathtoresc-ignore.dsvfile>--repo-url=<urlofrepositorytoscan>Most CLI arguments can also be provided by setting the corresponding environment variable.
Please see the --help options on the arguments that can be provided using environment variables, and the expected environment variable names.
These will always be prefixed with RESC_Example: the argument--gitleaks-pathcan be provided using the environment variableRESC_GITLEAKS_PATHIgnoring findingsPreviewIt is possible to ignore some blocker findings (e.g. false positive) by providing
aresc-ignore.dsvfile. The bockers will be downgraded to a warning level and marked asignored. Such file has the following structure:# This is a commentfinding_path|finding_rule|finding_line_number|expiration_date
finding_path_2|finding_rule_2|finding_line_number_2finding_pathcontains the path to the file with the blocking finding.finding_rulecontains the name of the blocking rule.finding_line_numbercontains the line number of the finding.expiration_dateis optional, contains the date in ISO 8601 format until which this ignore rule should be considered valid.For example, if we want to ignore the finding in file/etc/passwdfor ruleroot_value_foundon line1until April 1st 2024 at 23:59 the following line should be used./etc/passwd|root_value_found|1|2024-04-01T23:59:00To ignore this findingad vitam aeternam:/etc/passwd|root_value_found|1TestingRun below commands to make sure that the unit tests are running and that the code matches quality standards:Note:To run these tests you need to installtox. This can be done on Linux and Windows with Git Bash.pipinstalltox# install tox locallytox-v-esort# Run this command to validate the import sortingtox-v-elint# Run this command to lint the code according to this repository's standardtox-v-epytest# Run this command to run the unit teststox-v# Run this command to run all of the above tests