Package: drkns
drkns is a simple build tool aimed at easing the continuous integration of
monorepos through dependency aware caching.Le problemMonorepositories contain many projects, it rarely makes sense to
rebuild an every projects when changes are committed.But we can't build necessarily limit ourselves to run builds and tests
in directories where changes occurred because some projects depends on others.Quick startYou wild kid!ConfuciusChecka dummy projecton which our tests are based.Create your drkns.yml filesFrom your monorepo root, indrkns.yml:dependencies:
  project1: projects/project1
  project2: projects/project2
  library1: library/library1Sampleprojects/project1/drkns.yml:checkSteps:
  installDependencies: pipenv install
  test: pipenv run pytest
buildSteps:
  build: pipenv run paver build
  deploy: pipenv run paver deploy_to_pypi
dependencies:
  - ../../library/library1Test that everything is alrightdrkns check  # Check the configuration file
drkns list   # Check list the identified steps
drkns run    # Run everything.gitignorehygieneecho '.drknspersistence/*' >> .gitignoreIn your CIWith environment variables:DRKNS_S3_PATH: S3 path to a directory you can write toAWS_ACCESS_KEY_IDAWS_DEFAULT_REGIONAWS_SECRET_ACCESS_KEYpip install drkns

drkns check             # We never know
drkns sync in           # Get past execution statuses from S3
drkns debug             # If needed, you can check which steps are
                        #   going to run here
drkns run               # Run all steps
drkns clean             # Will remove four weeks old execution statuses to 
                        #   limit disk usage
drkns sync out --delete # Persist all execution statuses from S3,
                        #  **use --delete** to also delete data removed by 
                        #  `clean`DocumentationUnitEachdrkns.ymlforms aunit, it should represent a specific unit of your
code, a directory containing a lib or a project, that has specific build steps
or can be depended on.drkns.ymlaccepted fields:directory: string, Current directory by default, only used to compute the
hash associated with this build filedependencies: strings sequence, path containing anotherdrkns.ymlfile
to load, you can also only indicate a directory name.checkSteps: dictionary of steps, see below.buildSteps: dictionary of steps, see below.cleanupSteps: dictionary of steps, see below.StepsSteps :Have acommandto run, string, required.Have abackgroundflag, default to false.Can also be a single string, thecommandargument.Are ordered.Are deduplicated at execution. E.g.library/front-util/drkns.ymlis required
by bothprojects/aviation-customer/drkns.ymlandprojects/retail-customer/drkns.ymlit will only appear in the first one to
be parsed. Note that this can impact.gitignoreresolution.Three different step types are available, by order of execution:checkSteps: steps necessary to check the integrity of theunit.
Will always be ran if no checkSteps has already failed in thisunit.
Typically used to build
your application, your docker images and run your unit tests.
If a steps fail, the following ones will not be executed.buildSteps: steps executed only ifcheckSteps, dependencies'checkStepsand previous buildSteps succeeded. You
can use this to upload or deploy the build artifacts. Beware there is no
guarantee thatdependencieswill be executed, there execution might have
been startedcleanupSteps: steps executed no matter what (except if theunithas not
changed since last execution). You can suppress things that have
taken too much memory or things in the like.Warning: implicit cleanup of background tasksIf any of your steps used the background attributes they are killed after the
cleanup steps at the end of the execution of your `unit.CachingThe point ofdrknsis to not rerun everything for every change.Upon execution a.drknspersistencefolder in the directory of each unit will
be created, this folder will contain the execution status of each step for a
specific unit, step and a specific hash.At each execution a hash for the unit is computed, it changes if files within
the unit or its dependencies hashes have changed. If it is present in the
persisted files, it will prevent the execution.Ignoring files.drknsignoreOften, it happens that the execution of steps will alter some files in the
directory changing the mentioned above hash.It is not necessarily something that we wish.You can add partial or full file or directory names (as long as they have a
trailing/) to be ignored in a.drknsignorefile.
A unit will ignore specific patterns.Warning:.drknsignoreare also inherited from parents.Because of this
path are not supported to avoid ambiguity.The ignored elements are passed todirhash'signoreargument. The relevant
documentation is available athttps://pypi.org/project/dirhash/.Sample.drknsignore:*.tmp
build/PersistencePersistence is enabled through S3 buckets.You need aDRKNS_S3_PATHenvironment variable, or provide it as a command line
argument.export DRKNS_S3_PATH=s3://some-drkns-bucket/someProjectDirectorydrknsexpectsAWS environment variablesto be defined:AWS_ACCESS_KEY_ID,AWS_DEFAULT_REGIONandAWS_SECRET_ACCESS_KEYfor the AWS client to work properly.Parallel CI generationThis is an advanced usage. You are invited to skip this if this is your first
dive indrkns.Once your CI is setup, maybe you want to parallelize job execution,drknscan
generate your CI configuration, provided you write a template.The template must be located at.drknsgeneration/*.template.*, only a single
template file must be in this directory.If you need a more concrete example, checkthe dummy project sample main.template.yml.Fields to be defined in the template, always in the form%FIELD_NAME%:DRKNS_GROUP_BEGINand%DRKNS_GROUP_END%: around a group template
definition. A group will containunitsthat are
ran together. The lines containing those tags will be removed. For now, only
one unit per group is supported, so do not be surprised.DRKNS_UNIT_BEGINandDRKNS_UNIT_END: around a unit template,
mostly callingdrkns run %DRKNS_UNIT_NAME%. The lines containing those tags
will be removed.%DRKNS_GROUP_UNITS%: where group units will be inserted -
must be within a group template. The line containing this tag will be removed.%DRKNS_UNIT_NAME%: the unit's name%DRKNS_DEPENDENCY_GROUPS_NAMES{, }%: the dependency group names,
with a comma separator (as an example),
if you need a
separator:%DRKNS_DEPENDENCY_GROUPS_NAMES?{, }%will only insert a,.%DRKNS_ALL_GROUPS_NAMES{, }%: all group names, here with a comma separatordrknsCLICommandsdrkns check: checksdrkns.ymlfiles by loading the one in the current
directory and printing errors if anything bad happens.drkns clean: cleans older than four weeks execution steps from the persisted
data.drkns debug: prints debug information, mostly to control if execution cache
has been altered.drkns forget UNIT_NAME: forgets the previous execution ofUNIT_NAME,allfor everything. Nota the name of the root config unit ismain. This
can be used to force the new execution of some entries.drkns list: prints all the available steps plan. Beware repeating steps are
removed.drkns run [--limit-output] [--force-success] [--summary] STEP_NAME: run the
given steps and its requirements or all steps.drkns sync DIRECTION [--delete ][DRKNS_S3_PATH]: syncs the persisted
execution cache,DIRECTIONcan beinorout.drkns generate: generates a file based on the template described aboveOptions--force-success: flag, forces exit code0.--limit-output: flag, less restrictive thansummary, limits the outputs to
the failed steps' output at this execution (ignored cached ones), steps and
execution statuses.--summary: flag, prints only the summary: steps and execution statuses.--delete: flag, deletes remote execution status files that do not exist
locally. It is of paramount importance to use afterclean. It is a flag and
not a default behavior as it makes sense to sync in / sync out at each step
to prevent duplicate execution when a parallel processing is happening, or in
the case the whole process is canceled.Outputdrknsoutput is composed of the failed steps output only and a summary.Failed steps outputOutput for unitName1@stepName1:
/bin/sh: line 1: stop: command not foundIn case a step got canceled because of a previous failure:Output for unitName2@stepName2:
Previous failure of dependency1 / dependency1FailingCheckSummarydependency1@dependency1FailingCheck: Error (restored)
dependency1@dependency1Build: Ignored
dependency1@dependency1Cleanup: OK (restored)
project1@project1Check: OK (restored)
project1@project1ShouldNotRunBuild: Ignored (restored)
project1@project1Cleanup: OK (restored)
project3@project3Build: OK
project3@project3failingCleanup: Error
project3@project3NeverExecutedCleanUp: OK
main@mainCheck1: OK
main@mainBuild1: IgnoredStep statuses:Error: an error occurred on executionIgnored: an error occurred in a previous step, preventing this step from
being executedOK: all good(restored)indicates that the execution did not happen this round but was
cache from a previous step.