Package: rptf
Remote Pipeline Test FrameworkEnd-to-end test framework for GitLab pipeline template authors.Trigger pipelines, make assertions.WhyAs a GitLab pipeline author, your pipeline and jobs will be used across several projects under varying configurations. Your pipeline template understands the context (branches, tags, vars, source files etc.) and responds accordingly.The resultant pipeline's status might vary, the number of stages and jobs might change, statuses of the jobs might vary, artifacts may or may not be produced and environments may or may not be provisioned etc.Should you not setup proper tests for each of these scenarios? Should these tests not be executed on every change of the pipeline template?Thus,rptf.Table of ContentsQuickstartUsing RPTF with GitLabSetup GitLab Access TokenUsage With Self-Managed GitLabAuthoring TestsDetailed ExampleTest OutputCLI OptionsJUnit ReportsLicenseContributingWho Uses RPTFQuickstartBeing authored in Python,rptfis distributed via PyPi:https://pypi.org/project/rptf/Installrptflocally withpip.pipinstallrptfThis makesrptfavailable for local execution:rptf--helpTest cases are defined in a Yaml file, which by default is calledrptf.yml.# rptf.ymlshould_succeed_with_25_jobs:trigger:project_id:...assertions:pipeline_status:successjob_count:25should_fail_with_1_job:trigger:project_id:...assertions:pipeline_status:failedjob_count:1should_fail_with_3_jobs_deploy_skipped:trigger:project_id:...assertions:pipeline_status:failedjob_count:3job_status:build:successtest:faileddeploy:skippedEach test case contains two sections:triggerandassertions. The example above shows two test project's pipelines being triggered and asserts being make expect the pipelines have25jobs and1job respectively.Pipeline status is also asserted, and the second test case expects a failed pipeline.Using RPTF with GitLabWe can executerptfwithin a GitLab pipeline, thus, you have a primary test project which contains therptf.ymlfile and includes a.gitlab-ci.ymldefinition that triggers pipelines in other GitLab projects and makes assertions.# .gitlab-ci.ymlstages:-testrptf:stage:testimage:python:3before_script:-pip install rptfscript:-rptfTherptfjob above triggers pipelines defined inrptf.ymlas long as theGITLAB_ACCESS_TOKENprovided has sufficient permissions. It's advisable to set the token at the project or group level viaGitLab -> Settings -> CICD.Setup GitLab Access TokenThe project containing therptfjob needs to provideGITLAB_ACCESS_TOKEN. This token must be authorized to execute pipelines across all projects defined inrptf.yml.Usage With Self-Managed GitLabEnvironment variableGITLAB_API_URLis read to determine the GitLab instance where the pipelines are created.You can set this to getrptfto work with your self-managed GitLab instance.# set GITLAB_API_URLexportGITLAB_API_URL=https://path-to-your-instance/api/v4# execute rptfrptfAuthoring TestsThe Yaml specification declares the following attributes that can be used to author test cases:triggerdefines the target project where pipeline should be triggeredproject_idrequiredrefdefaults tomasterbranchdeprecated in favor ofrefvariablesdefaults to an emppty list `[]'assertionsdefines assertions to make after pipeline finishes executionpipeline_statusrequired, expected pipeline statusjob_countexpected number of jobsjob_statusdict of expected job statusTODOartifact_countexpected number of artifactsTODOenvironment_statusdict of expected environment statusDetailed Exampletwo_jobs_two_artifacts:trigger:project_id:11112222ref:feature-branch-abcvariables:hello:worldfoo:barassertions:pipeline_status:successjob_count:2job_status:artifact-one:successartifact-two:successartifact_count:2# TODOenvironment_status:# TODOsuccess:available# TODOreview_env_one:stopped# TODOCLI Options# helprptf--help# help# run testsrptf# by default, executes rptf.ymlrptf--targetFILE# path to test definition yaml# toggle JUnit test reportingrptf# generates JUnit reportrptf--junit# generates JUnit reportrptf--no-junit# JUnit reporting skippedTest OutputJUnit ReportsBy default, all test failures are reported in a JUnit compatible XML file calledrptf-report.junit.xml.This can be integrated in therptfjob to enable deep integration with GitLab (or any other tool that supports JUnit reporting format).# .gitlab-ci.ymlstages:-testrptf:stage:testimage:python:3before_script:-pip install rptfscript:-rptfartifacts:reports:junit:rptf-report.junit.xmlExample of RPTF has failing tests integrated back into a GitLab Merge Request:https://gitlab.com/sri-at-gitlab/projects/remote-pipeline-test-framework/framework/-/merge_requests/1JUnit reporting can be turned off by passing therptf --no-junitflag.LicenseMITContributingContributions welcomeherein the form of feature requests, feature implementations, documentation improvements, examples etc.# execute locallypipenvinstall
pipenvrunpython-mrptf.rptfWho Uses RPTFIn-exhaustive list of RPTF users:GitLab Five Minute Production AppLeapMotionAPNICBrainbitsEverbridgeBoltAddyour organization.