Package: jobs-done10
Job's DoneAboutJob's Done is a tool heavily inspired byTravis, and works in the same way
in that configuring a.jobs_done.yamlfile in your repository's root to create and trigger Continuous Integration jobs.Example of a.jobs_done.yamlfile:matrix:platform:-"win64"-"redhat64"platform-win64:build_batch_commands:-|python -m venv .env3   || goto eofcall .env3\Scripts\activate   || goto eofpytest --junitxml=tests-{platform}.xmlplatform-redhat64:build_shell_commands:-|python3 -m venv .env3source .env3/bin/activatepytest --junitxml=tests-{platform}.xmljunit_patterns:-"tests.*.xml"Considering this file is in the root of repositorymyprojectand was pushed to branchfeat-71, this will generate two Jenkins jobs:myproject-feat-71-win64myproject-feat-71-linux64Command-lineJobs done can be executed in the command-line.To use it, from the repository's folder that you want to create jobs for, execute:$jobs_donejenkins--usernameUSERhttps://example.com/jenkinsThis will create/update existing jobs.Below are the possible installation options.PyPICreate a virtual environment using Python 3.10 and activate it:$python-mvenv.env$.env\Scripts\activate# Windows$source.env/bin/activate# LinuxInstall jobs_done10:$pipinstalljobs_done10DevelopmentClone the repository:git clone git@github.com:ESSS/jobs_done10.gitcd jobs_done10Create a virtual environment using Python 3.10 and activate it:$python-mvenv.env$.env\Scripts\activate# Windows$source.env/bin/activate# LinuxInstall dependencies and the project in editable mode:$pipinstall-rrequirements.txt-e.Upgrading dependenciesWe usepip-toolsto pin versions, follow the instructions in the
docs to add new libraries or update existing versions, adding--extra devto include development dependencies.Serverjobs done includes aflaskserver injobs_done10.serverwhich can be deployed usingDocker.It provides two end-points forPUSHandGET:/stash: tailored to receive the push event from BitBucket Server (formerly known as Stash)./github: tailored to receive the push event from GitHub.Important: the webhook must be configured with content-type ofapplication/jsonand asecret.AGETto either ends points will return the JobsDone version, useful to check the installed version and
that the end-point is correct.Posting to/is the same as posting to/stashfor backwards compatibility (might be removed in the future).ConfigurationConfiguration is done by having a.envfile (cortesy ofpython-dotenv)
in the root of this repository with the following variables:JD_JENKINS_URL=https://example.com/jenkinsJD_JENKINS_USERNAME=jenkins-userJD_JENKINS_PASSWORD=some passwordJD_STASH_URL=https://example.com/stashJD_STASH_USERNAME=stash-userJD_STASH_PASSWORD=some passwordJD_GH_TOKEN=github-user-personal-access-tokenJD_GH_WEBHOOK_SECRET=webhook-secretJD_EMAIL_USER=mail-sender@example.comJD_EMAIL_FROM=JobsDone Bot <mail-sender@example.com>JD_EMAIL_PASSWORD=email passwordJD_EMAIL_SERVER=smtp.example.comJD_EMAIL_PORT=587BuildClone the repository and checkout the tag:$gitclonehttps://github.com/ESSS/jobs_done10.git$cdjobs_done10$gitcheckout<VERSION>Build a docker image:$dockerbuild.--tagjobsdone:<VERSION>--build-argSETUPTOOLS_SCM_PRETEND_VERSION=<VERSION>Run server$dockerrun--publish5000:5000jobsdone:<VERSION>Hello WorldThis is an example of a Job's Done file, and what you might expect of its contents.build_batch_commands:-"echoMESSAGE:Hello,world!"description_regex:"MESSAGE\\:(.*)"Adding this file to a repository hooked into our CI system will create a single job that when executed run a
Windows batch command, and later on catches the message echoed and sets that as the build description.TestsThis is an example of a simple application with tests:build_batch_commands:-"pytest--junitxml=pytest_results.xml"junit_patterns:-"pytest_results.xml"This jobs runs pytest in the repository and outputs test results to a file. We also configure the job to look for that
file, and present test results to us at then end of the build.Multiple platformsThe same application as above, but now running on multiple platforms.platform-win64:build_batch_commands:-"pytest--junitxml=pytest_results-{platform}.xml"platform-redhat64:build_shell_commands:-"pytest--junitxml=pytest_results-{platform}.xml"junit_patterns:-"pytest_results.*.xml"matrix:platform:-"win64"-"redhat64"Here we add amatrixsection to define variations of this job. In this case, we have the platform variable,
with two possible values,win64andredhat64.One job will be created for each possible combination in the matrix (only two jobs in this case).Since we can't run batch commands in linux, we add another builder section,build_shell_commands. Using some flags
before defining sections we can choose which one will be available in each job.Values from the matrix can also be used as variables, in this case,{platform}will be replaced by the platform used
in that job (win64orredhat64).Branch patternsBranch patterns are used to filter which branches will produce jobs. This list of regular expressions
(using Python syntax), ensures that a branch will only produce jobs if at least one of the regular
expressions matches the name of the branch.Here's an example that filter onlymasterand feature branches:branch_patterns:-"master"-"fb-*"If this section is not defined in the file, all branches will produce jobs.Job MatrixAs shown in the examples above, the job matrix can be used to create multiple variations of a job.
One job for each combination of entries in this matrix is created.matrix:mode:-"app"-"cases"platform:-"win64"-"linux64"In this case 4 jobs will be generated:app-win64app-linux64cases-win64cases-linux64Note that you can use any variables you need, Job's done has no idea whatmodeorplatformmeans.There's anexcludeclause which can be used to remove particular entries from the matrix:matrix:mode:-"app"-"cases"platform:-"win64"-"linux64"mode-cases:platform-win.*:exclude:"yes"This will exclude all cases jobs from windows.String replacementVariables defined in the job matrix can be used to replace strings in the job file.On top of matrix variables, there are a few special string templates that can be used in any job:name- Name of the repositorybranch- Branch being builtmatrix:platform:-"win64"-"linux64"platform-win.*:build_batch_commands:-"echoBuildingproject{name}inbranch{branch}onplatform{platform}"Note that we use Python's format syntax, so if you need an actual{or}use double braces:{{,}}.Condition FlagsVariables defined in your job matrix can also be used to control some of the contents in your job file.
A common example here is using different builder for windows (bash) and linux (shell).This is done by adding a prefix to sections of the YAML file, with the variable name and value necessary to use it:platform-win.*:build_batch_commands:-"dir."platform-linux.*:build_shell_commands:-"ls-la."matrix:platform:-"win32"-"linux64"Matrix variables can also define aliases, useful to reduce duplication when using such flags.
To add aliases, simply use commas to separate additional names for matrix values:platform-windows:build_batch_commands:-"dir."platform-linux:build_shell_commands:-"ls-la."matrix:platform:-"win32,windows"-"win64,windows"-"linux64,linux"On top of that you can use a special variablebranchthat's always available, and points to your branch:branch-master:build_batch_commands:-"echoBuild"branch-deploy:build_batch_commands:-"echoBuild+Deploy"Condition values can use Python regex syntax for extra flexibility:branch-master:build_batch_commands:-"echoBuild"branch-fb.*:build_batch_commands:-"echoFeaturebranch!"branch-rb.*:build_batch_commands:-"echoReleasebranch!"DevelopmentCreate a virtual environment and install it in development mode:$python-mvirtualenv.env36$source.env36/bin/activate$pipinstall-e.Run tests:$pytestsrcDeployOpen a PR updating the CHANGELOG and after it passes, push a tag to the repository.This will automatically publish it to PyPI and to the configured Docker Registry in the organization.All optionsadditional_repositoriesAdditional repositories to be checked out in this job.The repository where this .jobs_done file is included by default.RequiresMultiple SCMs PluginandGit Plugin.Uses same options asgit.additional_repositories:-git:url:"https://project.git"branch:"{branch}"auth_tokenJob authentication token required to triggers builds remotely.auth_token:"my_token"boosttest_patternsList of boosttest file patterns to look for test results.Requires thexUnit Plugin.boosttest_patterns:-"*.xml"branch_patternsList of regexes used to match branch names.
Only branches that match one of these will create
jobs.branch_patterns:-"master"-"fb-*"build_batch_commandsList of Windows batch commands used to build the job.
If errorcode is not 0 after any command, the build fails.build_batch_commands:-"pytestsrc"-"echoFinished"build_shell_commandsList of shell commands used to build the job.
If errorcode is not 0 after any command, the build fails.build_shell_commands:-"pytestsrc"-"echoFinished"build_python_commandsList of python commands used to build the job.RequiresPython Plugin.build_python_commands:-"print(5)"console_colorEnable support for ANSI escape sequences, including color, to Console Output.RequiresAnsiColor Plugin.Accepted values:xterm(default)vgacssgnome-terminalconsole_color:"css"coverageEnables code coverage report.RequireCobertura Plugin.Options:report_pattern: mandatory, pattern where XML coverage files are searched. These XML files are usually inCoberturaformat, which is also format bypytest-covXML output (because pytest-cov uses coverage library).healthy: optional, specifies desired method, line and conditional metric. Any omitted metric defaults to80.unhealthy: optional, specifies desired method, line and conditional metric. Any omitted metric defaults to0. Builds below these thresholds are marked as unhealthy.failing: optional, specifies desired method, line and conditional metric. Any omitted metric defaults to0. Builds below these thresholds are marked as failed.coverage:report_pattern:"**/build/coverage/*.xml"healthy:method:100line:100conditional:90unhealthy:method:95line:95conditional:85failing:method:90line:90conditional:80cronSchedules to build to run periodically.cron:|# Everyday at 22pm* 22 * * *custom_workspaceDefines a custom workspace directory for the job. To maintain the same base directory as the default workspace directories prefix it with"workspace/".custom_workspace:"workspace/devspace-user"description_regexRegular expression for searching job output for a description.
If a match is found, the contents of the first group will be set as the description.RequiresDescription Setter Plugin.description_regex:"OUTPUT:(.*)"display_nameConfigures the display name of the job.display_name:"{branch}{name}"email_notificationSends emails for failed builds.email_notification:"email1@example.comemail2@example.com"# oremail_notification:recipients:"email1@example.comemail2@example.com"notify_every_build:truenotify_individuals:trueexcludeExcludes a job from the matrix.platform-linux64:exclude:"yes"gitAdditional git options for the main project.RequiresGit Plugin.Options available here are shared withadditional_repositories.git:branch:masterclean_checkout:falselfs:truerecursive_submodules:truereference:"/home/reference.git"shallow_clone:50tags:truetarget_dir:"main_application"timeout:30url:"ssh://server/somerepo.git"Note: by default,tagsisfalsebecause we noticed that fetching tags takes quite some time and
we don't use tags for anything. If tags are needed, settagstotrue.jsunit_patternsList of jsunit file patterns to look for test results.RequiresJSUnit Plugin.jsunit_patterns:-"*.xml"junit_patternsList of junit file patterns to look for test results.RequiresxUnit Plugin.junit_patterns:-"*.xml"label_expressionConfigures the label expression of the job.The label-expression is used to determine which workers can run the job.label_expression:"{platform}"matrixConfigures variations of a job.matrix:python:-"27"-"36"notify_githubNotifies GitHub of job build status.Currently it does not support any parameters, using default values.RequiresGitHub Plugin.notify_github:notify_stashNotifies a Stash instance of job build status.When no parameters are given, uses configurations set in the Jenkins instance.RequiresStashNotifier Plugin.notify_stash:url:"example.com/stash"username:"user"password:"pass"# Using default from Jenkinsnotify_stash:parametersJob parameters for Jenkins jobs.Currently, onlychoiceandstringare implemented.parameters:-choice:name:"PARAM_BIRD"choices:-"African"-"European"description:"Areawherethebirdisfrom"-string:name:"PARAM_VERSION"default:"dev"description:"Appversion"scm_pollSchedules to periodically poll SCM for changes, and trigger builds.scm_poll:|# Everyday at 22pm* 22 * * *slackConfigure notification with slack.Configure your Jenkins integration on SlackObtain the tokenConfigure your job to notify slack using this option.slack:team:essschannel:devtoken:XXXurl:https://example.com/jenkinstimeoutJob timeout in minutes.timeout:60timestampsShow timestamps on the left side of the console output.Requires theTimestamper Plugin.timestamps:trigger_jobsTrigger other jobs after the current job finishes. Parameters are optional.trigger_jobs:names:-myrepo-{branch}-synthetic-{platform}condition:SUCCESS# can be one of: SUCCESS, UNSTABLE, FAILED, ALWAYS. Defaults to SUCCESS.parameters:# optional-PARAM1=VALUE1-PARAM2=VALUE2warningsConfigures parsing of warnings and static analysis in a CI job.RequiresWarnings Plugin.warnings:console:-parser:Clang (LLCM based)-parser:PyLintfile:-parser:CppLintfile_pattern:*.cpplint-parser:CodeAnalysisfile_pattern:*.codeanalysis