Package: vcs-scraper
Repository Scanner Version Control System Scraper (RESC-VCS-SCRAPER)Table of ContentsAbout the componentGetting startedPrerequisitesRun locally from sourceRun locally using dockerTestingAbout the componentThe RESC-VCS-Scraper component collects all projects and repositories from multiple VCS providers. The supported VCS providers are Bitbucket, Azure Repos, and GitHub.This component includes two main modules, the project collector and the repository collector.
The project collector collects all projects and sends them to the project queue. The repository collector collects projects from the projects queue, fetches its corresponding repositories, and sends them to the repository queue.Getting startedThese instructions will help you to get a copy of the project up and running on your local machine for development and testing purposes.PrerequisitesGitDocker DesktopPython 3.9.0Run locally from sourcePreviewPrerequisites:RabbitMQ must be up and running locally.If you have already deployed RESC through helm in Kubernetes, then rabbitmq is already running for you.Clone the repository, open the Git Bash terminal from /components/resc-vcs-scraper folder, and run below commands.1. Create virtual environment:cdcomponents/resc-vcs-scraper
pipinstallvirtualenv
virtualenvvenvsourcevenv/Scripts/activate2. Install resc_vcs_scraper package:pipinstall-e.3. Set below environment variables:exportRESC_RABBITMQ_SERVICE_HOST=127.0.0.1#  The hostname/IP address of the rabbitmq serverexportRESC_RABBITMQ_SERVICE_PORT_AMQP=30902#  The amqp port of the rabbitmq serverexportRABBITMQ_DEFAULT_VHOST=resc-rabbitmq#  The virtual host name of the rabbitmq serverexportRABBITMQ_QUEUES_USERNAME=queue_user#  The username used to connect to the rabbitmq projects and repositories topicsexportRABBITMQ_QUEUES_PASSWORD=""# The password used to connect to the rabbitmq projects and repositories topics, can be found for the value of queues_password field in /deployment/kubernetes/example-values.yaml fileexportVCS_INSTANCES_FILE_PATH=""# The absolute path to vcs_instances_config.json file containing the vcs instances definitionsexportGITHUB_PUBLIC_USERNAME=""# Your GitHub usernameexportGITHUB_PUBLIC_TOKEN=""#  Your GitHub personal access tokenYou need to replace with correct values for RABBITMQ_QUEUES_PASSWORD, VCS_INSTANCES_FILE_PATH, GITHUB_PUBLIC_USERNAME and GITHUB_PUBLIC_TOKEN.4. Run thecollect_projectstask:collect_projectstask collects all projects from a given Version Control System Instance, then writes the found projects to a RabbitMQ channel called 'projects'.This can be done via the commandcollect_projectsStructure of vcs instances config jsonThe vcs_instances_config.json file must have the following format.Note:You can add multiple vcs instances.PreviewExample:{"vcs_instance_1":{"name":"GITHUB_PUBLIC","scope":["kubernetes"],"exceptions":[],"provider_type":"GITHUB_PUBLIC","hostname":"github.com","port":"443","scheme":"https","username":"GITHUB_PUBLIC_USERNAME","token":"GITHUB_PUBLIC_TOKEN","organization":""}}scope: List of GitHub accounts you want to scan.
For example, lets'say you want to scan all the repositories for the following github accounts.https://github.com/kuberneteshttps://github.com/dockerThen you need to add to the scope the following accounts like : ["kubernetes", "docker"]. All the repositories from those accounts will be scanned.exceptions (optional): If you want to exclude any account from scan, then add it to exceptions. Default is empty exception.Theoutputmessages ofcollect_projectscommand has the following format:{"project_key":"kubernetes","vcs_instance_name":"GITHUB_PUBLIC",}5. Run collect all repositories task:This task collects all repositories from a single VCS project, then writes the found repositories to a RabbitMQ channel called 'repositories'.This can be done via the command:celery-Avcs_scraper.repository_collector.commonworker--loglevel=INFO-E-QprojectsRun locally using DockerPreviewRun the RESC VCS Scraper Docker image locally by running the following commands:Pull the Docker image from registry:dockerpullrescabnamro/resc-vcs-scraper:1.0.0Alternatively, build the Docker image locally by running:dockerbuild-trescabnamro/resc-vcs-scraper:1.0.0.Run the vcs-scraper by using below command:dockerrun-v<pathtovcs_instances_config.jsoninyourlocalsystem>:/tmp/vcs_instances_config.json-eRESC_RABBITMQ_SERVICE_HOST="host.docker.internal"-eRESC_RABBITMQ_SERVICE_AMQP_PORT=30902-eRABBITMQ_DEFAULT_VHOST=resc-rabbitmq-eRABBITMQ_QUEUES_USERNAME=queue_user-eRABBITMQ_QUEUES_PASSWORD="<the password of queue_user>"-eVCS_INSTANCES_FILE_PATH="/tmp/vcs_instances_config.json"-eGITHUB_PUBLIC_USERNAME="<your github username>"-eGITHUB_PUBLIC_TOKEN="<your github personal access token>"--nameresc-vcs-scraperrescabnamro/resc-vcs-scraper:0.0.1collect_projectsTo create vcs_instances_config.json file, refer:Structure of vcs_instances_config.jsonTesting(Back to top)Run below commands to make sure that the unit tests are running and that the code matches quality standards:Note:To run these tests you need to installtox. This can be done on Linux and Windows with Git Bash.pipinstalltox# install tox locallytox-v-esort# Run this command to validate the import sortingtox-v-elint# Run this command to lint the code according to this repository's standardtox-v-epytest# Run this command to run the unit teststox-v# Run this command to run all of the above tests