Package: c2cciutils
C2C CI utilsPublishingThe main goals of C2C CI utils is to offer the commands and the workflows to have the following project structure:Have stabilization branches named by default<major>.<minor>.
Have the release named by default<major>.<minor>.<patch>.With C2C CI utils you can publish a python package and a Docker image from the same repository.The default publishing are:Push on the<major>.<minor>branch will publish Docker images.Create the tag<major>.<minor>.<patch>will publish the Docker images, and the Python package.Push on a feature branch (whatever other name) will publish the Docker images.Delete a feature branch will delete the Docker images.Push on themasterbranch will publish the Docker images with the master tag (Publishing a python package is also possible).The version at the last line of theSECURITY.mdof themasterbranch will be also published using thelatesttag,
this will respect thetagspresent in the configurationIn theSECURITY.mdfile of themasterbranch we can also add a columnAlternate Tagto publish the Docker images with another tag,
this will respect thetagspresent in the configuration (only for Docker).The Docker images are published on Docker Hub and GitHub Container Registry.You can run the publishing locally in dry-run mode:GITHUB_REF=...c2cciutils-publish--dry-run...ChangelogWhen we create a tag by default with thechangelogworkflow a release is created on GitHub, a changelog is
generated and added to the release.SecurityThe security is managed by thec2cciutils-auditcommand with Snyk, it will audit the dependencies of the project on every
stabilization branches, if possible a pull request is created automatically to update the dependencies.When we publish a Docker image the generated image is monitored by Snyk, this means that Snyk will search
for all the dependencies and send the list to the Snyk web site to be monitored.
We also do a test of the image and log the result (This will never cause the build to fail).ChecksC2C CI utils will no more provide a tool to do a check of the project, this is replaced bypre-commit,
a base configuration is provided in the example project.Pull request checksA workflow is provided to run the checks on the pull requests, it will run thec2cciutils-pr-checkscommand.Check that the commit message and the pull request title start with a capital letter.Check that there aren't any spelling issue in the commit message and in the pull request title.Add a message to the pull request with a link to the JIRA issue if the pull request branch name starts with[a-zA-Z]+-[0-9]+-or end with-[a-zA-Z]+-[0-9]+.DependenciesIn the example project there is a basic Renovate configuration, it will update the dependencies of the project.
There is also a workflow to add a review on the Renovate pull requests to make the auto merge working on
repository that required a review.BackportsA workflow is provided to backport the pull requests on the stabilization branches, it will be triggered by
adding a label namedbackport <destination_branch>on the pull request.Old workflowsGitHub will retain all the old workflows, so we need to delete them, thedelete-old-workflows-runworkflow will delete the workflows older than 500 days.WorkflowsC2cciutils make easier to have those workflows in a project:audit.yaml: Audit the stabilization branches of the application against vulnerabilities in the python and node dependencyauto-review.yaml: Auto review the Renovate pull requestsbackport.yaml: Trigger the backports (work with labels)clean.yaml: Clean the Docker images related on a deleted feature branchmain.yaml: Main workflow especially with the c2cciutils-checks commandchangelog.yaml: Generate the changelog and create the release on GitHubdelete-old-workflows-run.yaml: Delete the old workflowspr-checks.yaml: Run the checks on the pull requestsAll the provided commands used in the workflow:c2cciutils: some generic tools.c2cciutils-version: Create a new version of the project.c2cciutils-checks: Run the checks on the code (those checks don't need any project dependencies).c2cciutils-audit: Do the audit, the main difference with checks is that it can change between runs on the same code.c2cciutils-publish: Publish the project.c2cciutils-clean: Delete Docker images on Docker Hub after corresponding branch have been deleted.UtilitiesThe following utilities are provided:c2cciutils: some generic tools.c2cciutils-download-applications: Download the applications with version managed by Renovate, see below.c2cciutils-docker-logs: Display the logs of the application in Docker (compose).c2cciutils-k8s-install: Install a k3d / k3s cluster, see below.c2cciutils-k8s-logs: Display the logs of the application in the k8s cluster, see below.c2cciutils-k8s-db: Create a database in the k8s cluster, see below.c2cciutils-k8s-wait: Wait that the application started correctly in the cluster, see below.c2cciutils-docker-versions-gen: Generate the Docker package versions file (ci/dpkg-versions.yaml), see below.c2cciutils-pin-pipenv: Display all the dependencies that's in thePipenv.lockbut not in thePipenvto be able to pin them.c2cciutils-trigger-image-update: Trigger the ArgoCD repository about image update on the CI (automatically done in the publishing).c2cciutils-google-calendar: Tool to test the Google credentials for calendar API and refresh them if needed. Seec2cciutils-google-calendar -hfor more information.New projectThe content ofexample-projectcan be a good base for a new project.New versionRequirements: the right version (>= 1.6) ofc2cciutilsshould be installed with theversionextra.To create a new minor version you just should runc2cciutils-version --version=<version>.You are welcome to runc2cciutils-version --helpto see what's it's done.Note that it didn't create a tag, you should do it manually.To create a patch version you should just create tag.SecretsIn the CI we need to have the following secrets::HAS_SECRETSto be set to 'HAS_SECRETS', to avoid error errors from external
pull requests, already set globally on Camptocamp organization.GOPASS_CI_GITHUB_TOKENandCI_GPG_PRIVATE_KEYrequired to initialize the gopass password store,
the secrets exists in the Camptocamp organization but not shared on all project, then you should add
your project to the shared list.Use locally, in the projects that use c2cciutilsInstall it:python3 -m pip install --user --requirement ci/requirements.txtDry run publish:GITHUB_REF=... c2cciutils-publish --dry-run ...ConfigurationYou can get the current configuration withc2cciutils --get-config, the default configuration depends on your project.
Note that it didn't contain the default defined the schema and visible in thegenerated documentation.You can override the configuration with the fileci/config.yaml.At the base of the configuration you have:version: Contains some regular expressions to find the versions branches and tags, and to convert them into application versions.audit: The audit configuration, seec2cciutils/audit.pyfor more information.publish: The publishing configuration, seec2cciutils/publish.pyfor more information.Many actions can be disabled by setting the corresponding configuration part toFalse.SECURITY.mdTheSECURITY.mdfile should contain the security policy of the repository, especially the end of
support dates.For compatibility withc2cciutilsit should contain an array with at least the columnsVersionandSupported Until. TheVersioncolumn will contain the concerned version.
TheSupported Untilwill contain the date of end of supportdd/mm/yyyy.
It can also contain the following sentences:Unsupported: no longer supported => no audit, no rebuild.Best effort: the support is ended, it is still rebuilt and audited, but this can be stopped without any notice.To be defined: not yet released or the date will be set related of another project release date (like for GeoMapFish).See alsoGitHub DocumentationIDEThe IDE should be configured as:usingblackandisortwithout any arguments,using theeditorconfigconfiguration.VScodeRecommend extensions to work well with c2cciutils:PrettierAnd use EditorConfigshell-formatWith the configuration"shellformat.flag": "-bn".Better TOMLOther recommend extensions:hadolintCode Spell CheckerSelect a formatter:CTRL+MAJ+PFormat document With...Configure Default Formatter...Select the formatterPublishingTo pypiThe config is like this:versions:# List of kinds of versions you want to publish, that can be:# rebuild (specified with --type),# version_tag, version_branch, feature_branch, feature_tag (for pull request)It we have asetup.pyfile, we will be in legacy mode:
When publishing, the version computed from arguments orGITHUB_REFis put in environment variableVERSION, thus you should use it insetup.py, example:VERSION=os.environ.get("VERSION","1.0.0")Also we consider that we usepoetrywithpoetry-dynamic-versioningto manage the version, andpoetry-plugin-tweak-dependencies-versionto manage the dependencies versions.Example of configuration:[tool.poetry-dynamic-versioning]enable=truevcs="git"pattern="^(?P<base>\\d+(\\.\\d+)*)"format-jinja="""{%- if env.get("VERSION_TYPE") == "version_branch" -%}{{serialize_pep440(bump_version(base, 1 if env.get("IS_MASTER") == "TRUE" else 2), dev=distance)}}{%- elif distance == 0 -%}{{serialize_pep440(base)}}{%- else -%}{{serialize_pep440(bump_version(base), dev=distance)}}{%- endif -%}"""Note that we can access to the environment variablesVERSION,VERSION_TYPEandIS_MASTER.Then by default:Tag with1.2.3=> release1.2.3Commit on feature branch just do a validationCommit onmasterbranch after the tag 1.3.0 => release1.4.0.dev1Commit on1.3branch after the tag 1.3.0 => release1.3.1.dev1To make it working in theDockerfileyou should have in thepoetrystage:ENVPOETRY_DYNAMIC_VERSIONING_BYPASS=devRUNpoetryexport--extras=checks--extras=publish--extras=audit--output=requirements.txt\&&poetryexport--with=dev--output=requirements-dev.txtAnd in therunstageARGVERSION=devRUN--mount=type=cache,target=/root/.cache\POETRY_DYNAMIC_VERSIONING_BYPASS=${VERSION}python3-mpipinstall--disable-pip-version-check--no-deps--editable=.And in theMakefile:VERSION=$(strip$(shellpoetryversion--short)).PHONY:buildbuild:## Build the Docker imagesdockerbuild--build-arg=VERSION=$(VERSION)--tag=$(GITHUB_REPOSITORY).To Docker registryThe config is like this:latest:Trueimages:-# The base name of the image we want to publishname:repository:<internal_name>:# The fqdn name of the server if not Docker hubserver:# List of kinds of versions you want to publish, that can be: rebuild (specified using --type),# version_tag, version_branch, feature_branch, feature_tag (for pull request)version:# List of tags we want to publish interpreted with `template(version=version)`# e.g. if you use `{version}-lite` when you publish the version `1.2.3` the source tag# (that should be built by the application build) is `latest-lite`, and it will be published# with the tag `1.2.3-lite`.tags:# If your images are published by different jobs you can separate them in different groups# and publish them with `c2cciutils-publish --group=<group>`group:By default, the last line of theSECURITY.mdfile will be published (docker) with the taglatest. SetlatesttoFalseto disable it.With thec2cciutils-cleanthe images on Docker hub forfeature_branchwill be removed on branch removing.Download applicationsIn case some executables or applications from GitHub releases or any other URLs are required on the CI host
and are not handled by any dependency manager, we provide a set of tools to install them and manage upgrades
through Renovate.Create an application file (e.-g.applications.yaml) with:# yaml-language-server: $schema=https://raw.githubusercontent.com/camptocamp/c2cciutils/master/c2cciutils/schema-applications.json# Application from GitHub release<organization>/<project>:get-file-name:<file name present in the release>to-file-name:<The file name you want to create in ~/.local/bin>finish-command:# The command you want to run after the file is downloaded--chmod# To be executable (usually required)-+x-<to-file-name>--<to-file-name># Print the version of the application---version# Application from GitHub release in a tar file (or tar.gz)<organization>/<project>:get-file-name:<file name present in the release>type:tartar-file-name:<The file name available in the tar file>to-file-name:<The file name you want to create in ~/.local/bin>finish-command:[...]# The command you want to run after the file is downloaded# Application from an URL<application reference name>:url-pattern:<The URL used to download the application>to-file-name:<The file name you want to create in ~/.local/bin>finish-command:[...]# The command you want to run after the file is downloadedIn the attributesurl-pattern,get-file-nameyou can use the following variables:{version}: The version of the application present in the version file.{version_quote}: The URL encoded version.{short_version}: The version without thevprefix.Theapplications-versions.yamlfile is a map of applications and their versions.Add in your Renovate configuration:regexManagers: [
    {
      fileMatch: ['^applications-versions.yaml$'],
      matchStrings: [
        '(?<depName>[^\\s]+): (?<currentValue>[^\\s]+) # (?<datasource>[^\\s]+)',
      ],
    },
  ],Now you need to callc2cciutils-download-applications --applications-file=applications.yaml --versions-file=applications-version.yamlto install required applications on CI host before using them (an already installed application is installed only if needed).Use Renovate to trigger a new build instead of the legacy rebuildRun the commandc2cciutils-docker-versions-gen camptocamp/image[:tag]to generate a file that is a kind of package lock of the Debian packages in the fileci/dpkg-versions.yaml.Add in your renovate configuration:regexManagers:[{fileMatch:['^ci/dpkg-versions.yaml$'],matchStrings:[" *(?<depName>[^'\\s]+): '?(?<currentValue>[^'\\s/]*[0-9][^'\\s/]*)'?"],datasourceTemplate:'repology',versioningTemplate:'loose',},],When a new version of a Debian package will be available:Renovate will automatically open a pull request to update the fileci/dpkg-versions.yaml.And the continuous integration will build a new fresh Docker image with latest versions of all Debian packages.KubernetesC2cciutils provide some commands for Kubernetes.You can define a workflow like that:-name:Install k3s/k3d (Kubernetes cluster)run:c2cciutils-k8s-install-name:Create a database to do the testsrun:c2cciutils-k8s-db --script=<my_script>.sql-name:Install the application in the Kubernetes clusterrun:kubectl apply -f <my_application>.yaml-name:Wait that the application is readyrun:c2cciutils-k8s-wait-name:Print the application status and logsrun:c2cciutils-k8s-logsif:always()-name:Uninstall the applicationrun:kubectl delete -f <my_application>.yaml || true-name:Cleanup the databaserun:c2cciutils-k8s-db --cleanupc2cciutils-k8s-installcan be configured in theci/config.yamlfile, in sectionk8s/k3d/install-commands, default is:--k3dclustercreatetest-cluster--no-lb--no-rollbackSee also:K3d cluster create documentation.c2cciutils-k8s-dbcan be configured in theci/config.yamlfile, in sectionk8s/db/chart-options, default is:persistence.enabled:'false'tls.enabled:'true'tls.autoGenerated:'true'postgresqlPassword:mySuperTestingPasswordvolumePermissions.enabled:'true'See also:Parameters documentations.ContributingInstall the pre-commit hooks:pipinstallpre-commit
pre-commitinstall--allow-missing-config