Package: im-jote
Informatics Matters Job Tester (“jote”)TheSquonk2 Job Tester(jote) is a Python utility used to rununit teststhat are defined in Data Managerjob implementation repositoriesagainst
the job’s container image, images that are typically built from the same
repository.joteis designed to run job implementations in a file-system
environment that replicates what they find when they’re run by the Data Manager.
But jobsare notrunning in the same operating-system environment, e.g. they
are not bound by the same processor and memory constraints they’ll encounter in
the Data Manager, which runs inKubernetes.A successful test should give you confidence that itshouldwork in the
Data Manger but without writing a lot of tests you’ll never be completely
confident that it will always run successfully.joteis a tool we designed to provideuswith confidence that we can
deploy jobs to a Data Manager instance and know that they’re basically fit
for purpose. Jobs that have no tests will not normally be deployed to the
Data Manager.To use a job in Squonk you need to create at least onemanifest fileand
onejob definition file. These reside in thedata-managerdirectory of the repository you’re going to test.joteexpects the
default manifest file to be calledmanifest.yamlbut you can use a
different name and have more than one.If you want to provide your own Squonk jobs and corresponding
job definitions ourVirtual Screeningrepository
(https://github.com/InformaticsMatters/virtual-screening) is a good
place to start. The repository is host to a number of job-based
container images and severalmanifestsandjob definition files.Here’s an examplemanifestfrom a recentVirtual Screeningrepository:---
kind: DataManagerManifest
kind-version: '2021.1'

job-definition-files:
- im-virtual-screening.yaml
- rdkit.yaml
- xchem.yamlEach Manifest must list at least one file. To be included in Squonk every
job must contain at least one test.joteruns the tests but also ensures
the repository structure is as expected and applies strict rules for the
formatting of the YAML files.Bothjoteand the Data Manager rely on the schemas that can be found
in ourJob Decoderrepository
(https://github.com/InformaticsMatters/data-manager-job-decoder).Here’s a snippet from a job definition file illustrating a
job (max-min-picker) that has a test calledsimple-execution.The test defines an input option (a file) and some other command options.
Thecheckssection is used to define the exit criteria of the test.
In this case the container must exit with code0and the filediverse.smimust be found in the generated test directory, i.e
it mustexistand contain100lines.jotewill fail the test unless
these checks are satisfied:jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        inputs:
          inputFile: data/100000.smi
        options:
          outputFile: diverse.smi
          count: 100
        checks:
          exitCode: 0
          outputs:
          - name: diverse.smi
            checks:
            - exists: true
            - lineCount: 100Running testsRunjotefrom the root of a clone of the Data Manager Job implementation
repository that you want to test:joteYou can display the utility’s help with:jote --helpJote container networkjotetests are executed on the networkdata-manager_jote. This is
defined in the docker-compose file that it generates to run your tests.Built-in variablesJob definition command-expansion provided by thejob decoderrelies on a number ofbuilt invariables. Some are provided by the
Data Manager when the job runs under its control
(i.e.DM_INSTANCE_DIRECTORY) others are provided byjoteto simplify
testing.The set of variables injected into the command expansion byjoteare: -DM_INSTANCE_DIRECTORY. Set to the path of the simulated instance
directory created byjote, normally created by the Data ManagerCODE_DIRECTORY. Set to the root of the repository that you’re running
the tests in. This is a convenient variable to locate your out-of-container
nextflow workflow file, which is likely to be in the root of your repositoryIgnoring testsOccasionally you may want to disable some tests because they need some work
before they’re complete. To allow you to continue testing other jobs under
these circumstances you can mark individual tests and have them excluded
by adding anignoredeclaration:jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        ignore:
        [...]You don’t have to remove theignoredeclaration to run the test injote.
If you want to see whether an ignored test now works you can runjotefor specific tests by using--testand naming the ignored test you want
to run. When a test is named explicitly it is run, regardless of whetherignorehas been set or not.Test run levelsTests can be assigned arun-level. Run-levels are numerical value (1..100)
that can be used to group your tests. You can use therun-levelas an indication of execution time, with short tests having low values and
time-consuming tests with higher values.By default all tests that have no run-level defined and those with
a run-level of1are executed.  If you set the run-level for longer-running
tests to a higher value, e.g.5, these will be skipped. To run these more
time-consuming tests you specify the run-level when runningjoteusing--run-level5.When you givejotea run-level only tests up to and including the
level, and those without any run-level, will be run.You define the run-level in the root block of the job’s test specification:jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        run-level: 5
        [...]Test timeoutsjotelets each test run for 10 minutes before cancelling (and failing) them.
If you expect that your test needs to run for more than 10 minutes you must
use thetimeout-minutesproperty in the job definition to define your own
test-specific value:jobs:
  [...]
  max-min-picker:
    [...]
    tests:
      simple-execution:
        timeout-minutes: 120
        [...]You should try and avoid creating too many long-running tests. If you cannot,
consider whether it’s a appropriate to userun-levelto avoidjoterunning them by default.Test groupsTests are normally executed and the environment torn-down between them.
If you have tests that depend on the results from a prior test you can run
tests as agroup, which preserves the project directory between the tests.To run a sequence of test (as agroup) you need to define atest-groupin your Job Definition file and then refer to that group in your test. Here,
we define a test group calledexperiment-a, at the top of the
definition file:test-groups:
- name: experiment-aWe then place a test in that group with arun-groupdeclaration
in the corresponding test block:jobs:
  max-min-picker:
    [...]
    tests:
      test-a:
        run-groups:
        - name: experiment-a
          ordinal: 1We need to provide anordinalvalue. This numeric value (from 1 ..N)
puts the test in a specific position in the test sequence. When tests are
placed in arun-groupyou have to order your tests, i.e. declare thattest-afollowstest-b. This is done with unique ordinals for each
test in therun-group. A test with ordinal1will run before a test
with ordinal2. Ordinals have to be unique within arun-group.You can run the tests for a specific group by using  the--run-groupoption:jote --run-group experiment-aRunning additional containers (group testing)Test groups provide an ability to launch additional support containers during
testing. You might want to start a background database for example, that can
be used by tests in yourtest-group. To take advantage of this feature
you just need to provide adocker-composefile (in the Job definitiondata-managerdirectory) and name that file in you r``test-groups``
declaration.Here we declare a docker-compose file calleddocker-compose-experiment-a.yaml:test-groups:
- name: experiment-a
  compose:
    file: docker-compose-experiment-a.yamlThe compose filename must begindocker-composeand end.yaml.The compose file is run before any tests in the corresponding test group
have been run and will be stopped after the last test in the group.The compose file you provide is run in adetachedstate sojotedoes
not wait for the containers to start (or initialise). As the first test
in the test group can begin very soon after the compose file is started
you can minimise the risk that your containers are not ready for the tests
by adding a fixed delay betweenjotestarting the compose file and
running the first test:test-groups:
- name: experiment-a
  compose:
    file: docker-compose-experiment-a.yaml
    delay-seconds: 10Nextflow test executionJob image types can besimpleornextflow. Simple jobs are executed in
the container image you’ve built and should behave much the same as they do
when run within the Data Manager. Nextflow jobs on the other hand are executed
using the shell, relying on Docker as the execution run-time for the processes
in your workflow.Be aware that nextflow tests run byjoterun under different conditions
compared to when it runs under the Data Manager’s control. In the Data Manager
nextflow jobs will be executed within a Kubernetes environment. When run byjotenextflow is expected using the operating system shell. This introduces a
variability that you need to take into account - i.e. underjotethe
nextflow controller runs in the shell, andare notexecuted in the same
environment or under the same memory or processor constraints.You might need to provide a custom nextflow configuration file
for your tests to run successfully. You do this by adding anextflow-config-filedeclaration in the test. Here, we name the filenextflow-test.config:jobs:
  max-min-picker:
    [...]
    tests:
      simple-load:
        nextflow-config-file: nextflow-test.config
        [...]The config file must be located in the Job repository’sdata-managerdirectory.Prior to running the corresponding testjotecopies it to the
Job’s project directory as the filenextflow.config(a standard file
expected by nextflow).jotewill notlet you have a nextflow config in your home directory
as any settings found there would be merged with the filejotewrites,
potentially disturbing the execution behaviour.NoteIt’s your responsibility to install a suitable nextflow that’s available
for shell execution.joteexpects to be able to run nextflow when
executing the correspondingcommandthat’s defined in the job
definition.Installationjoteis published onPyPIand can be installed from there:pip install im-joteThis is a Python 3 utility, so try to run it from a recent (ideally 3.10)
Python environment.To use the utility you will need to have installedDockerand,
if you want to test nextflow jobs,nextflow.Get in touchReport bugs, suggest features or view the source codeon GitHub.