Package: girder-slicer-cli-web
A Girder plugin for exposing slicer execution model CLIs over the web using docker and girder_worker.InstallationSlicer CLI Web is both aGirderplugin and aGirder Workerplugin.  It allows dockerized tasks to be run from the Girder user interface.LinuxIn linux with Python 3.6, 3.7, 3.8, 3.9, or 3.10:Prerequisites:An appropriate version of Python must be installed.MongoDB must be installed and running.RabbitMQ or another broker is needed to use Girder Worker.Docker must be installed and the current user must be part of the docker group.To use Girder:pipinstallgirder-slicer-cli-web[girder]girderbuildgirderserveTo use Girder Worker:pipinstallgirder-slicer-cli-web[worker]GW_DIRECT_PATHS=truegirder_worker-linfo-Ofair--prefetch-multiplier=1The first time you start Girder, you’ll need to configure it with at least one user and one assetstore (see theGirderdocumentation).  Additionally, it is recommended that you install some dockerized tasks, such as theHistomicsTKalgorithms.  This can be done going to the Admin Console, Plugins, Slicer CLI Web settings.  Set a default task upload folder, then import thedsarchive/histomicstk:latestdocker image.Girder PluginImporting Docker ImagesWhen installed in Girder, an admin user can go to the Admin Console -> Plugins -> Slicer CLI Web to add Docker images.  Select a Docker image and an existing folder and then select Import Image.  Slicer CLI Web will pull the Docker image if it is not available on the Girder machine.For each docker image that is imported, a folder is created with the image tag.  Within this folder, a subfolder is created with the image version.  The subfolder will have one item per CLI that the Docker image reports.  These items can be moved after they have been imported, just like standard Girder items.Running CLIsWhen you visit the item page of an imported CLI, an extraConfigure Tasksection is shown.  You can set all of the inputs for the CLI and indicate where output files should be stored.  SelectingRun Taskwill have Girder Worker execute the CLI.  See the Jobs panel for progress and error messages.Docker CLIsSlicer CLI Web executes programs from docker images.  Any docker image can be used that complies with some basic responses from itsENTRYPOINT.Specifically, when a docker image is invoked like:docker run <docker image tag> --list_cliit needs to respond with with a JSON dictionary of CLIs, where the keys of this dictionary are the names of the CLIs.  Seecli_list.jsonfor an example.Each available CLI needs to report what it takes as inputs and outputs.  When the docker image is invoked like:docker run <docker image tag> <cli name> --xmlit needs to return an XML specification to stdout.  SeeExample1.xmlfor an example.The XML must conform to theSlicer Execution Schema, with a few minor additions:Some types (image,file,transform,geometry,table,directory,item) can have areferenceproperty.  Whenimage,file,item,directoryare used with Girder, if thereferenceproperty is_girder_id_, then value will be passed as a Girder ID string rather than converted to a Girder resource.Theregiontype can have acoordinateSystemproperty.Theregiontype has an optionalshapesproperty that is a comma-separated list of values that can includedefault,rectangle,polygon,line,polyline, andpoint, plusmultiand one ofsubmit(orsibmitoff),submiton, orautosubmit.
In the official schema, region is a vector of six values of the form x,y,z,rx,ry,rz, defining a rectangle based on its center and radius in each of three dimensions.  This is thedefaultshape.  Therectangleshape allows a vector of four values defining a rectangle of the form x,y,width,height, where x,y is the left and top of the rectangle in pixel coordinates.  Many algorithms that accept this value accept -1,-1,-1,-1 as a default to specify the whole conceptual space.  Thepolygonshape allows for a list of x,y values.  Polygons must always have at least four points so that the vector of values cannot be confused with the default; repeat the first vertex at the end to specify a triangle.  Thelineshape allows a two-vertex line.  To disambiguate this from a rectangle, the values -2,-2 are added after the line.  Thepolylineshape allows a multi vertex line, indicated again by a -2,-2 value after the line.  Apointis a single vertex.multiallow multiple shapes, indicated by separating coordinates of each shape by -1,-1.  Note that neither -1,-1 nor -2,-2 are allowed as coordinates within a shape – to use those, specify them with decimals (e.g., -1.0,-1.0).
The submit options will add suggestions on how the UI should handle changes.  If present, the option to auto-run a job as soon as a valid shape is set should be present.autosubmitmeans this should always happen.submitorsubmitoffoffers this as a setting but is default to not submit the job.submitonoffers this as a setting and defaults to submitting the job.Some input types (image,file,item,directory) can havedefaultNameMatch,defaultPathMatch, anddefaultRelativePathproperties.  The first two are regular expressions designed to give a UI a value to match to prepopulate default values from files or paths that match the regex.defaultNameMatchis intended to match the final path element, whereasdefaultPathMatchis used on the entire path as a combined string.defaultRelativePathis used to find a value that has a path relative to some base.  In the Girder UI, this might be from an item.Input types can have adatalistproperty.  If this is present, when the CLI is first loaded or, possibly periodically after parameters have been changed, the CLI may be called with optional parameters.  The CLI is expected to return a new-line separated list of values that can be used as recommended inputs.  As an example, astringinput might have adatalistof--enumerate-options; the cli would be called with the existing parameters PLUS the extra parameter specified bydatalist.  If the result is sensible, the input control would expose this list to the user.  Thedatalistproperty is a json-encoded dictionary that overrides other parameters.  This should override parameters that aren’t needed to be resolved to produce the datalist (e.g., input and output files) as that will speed up the call.  The CLI should respond to the modified call with a response that contains multiple<element>sometext</element>values that will be the suggested data for the control.There are some special string parameters that, if unspecified or blank, are autopopulated.  String parameters with the names ofgirderApiUrlandgirderTokenare populated with the appropriate url and token so that a running job could use girder_client to communicate with Girder.Internally, thectk_climodule is used.  This has two differences from the Slicer Execution Schema that are technically bugs.Enumerations have bare elements under the appropriate parent tag.  That is, instead of a structure like<string-enumeration>...<enumeration><element>Value1</element><element>Value2</element>...</enumeration></string-enumeration>, the<enumeration>tag is omitted:<string-enumeration>...<element>Value1</element><element>Value2</element>...</string-enumeration>.Booleans specify a true or false value after the flag or long flag.  The Slicer Execution Schema states that booleans should be false by default and the presence of the flag should make them true.  Thectk_clispecifies that they take a singletrueorfalseparameter.  This doesn’t change the xml; it changes what is passed to the CLI.  Instead of passing--longflagto set the flag to true,--longflagtruemust be passed.Docker CLIs with GPU supportWhen girder_worker runs docker images, the containers are started with gpu support only if the docker image has a label saying it should use an nvidia driver (addLABEL com.nvidia.volumes.needed=nvidia_driverto the Dockerfile).–list_cli response formatThe response from a docker image invoked with the--list_clioption needs to be a JSON response returning a single object.  The object must contain a key for each CLI.  Each key has a value used to parse or handle the CLI.Here is a commented example:{
  // the key is the name of the CLI
  "Example1": {
    // type is typically either "python" or "cxx".  The default program
    // either runs "python <CLI key>/<CLI key>.py" for python or
    // "<CLI key>/<CLI key>" for cxx.
    "type": "python"
  },
  "Example2": {
    "type": "python",
    // The desc-type defaults to xml but can be any of "xml", "json", or
    // "yaml".  To get the CLI command line options, the CLI is invoked via
    //   docker run <docker image tag> <cli name> --<desc-type>
    "desc-type": "json"
  },
  "AnotherName": {
    // The alias allows the CLI to be invoked as either the key or the
    // alias.  This runs Example2 when invoked as AnotherName.
    "alias": "Example2",
    "type": "python"
  },
  "Example3": {
    "type": "python",
    // docker-params is a dictionary of parameters passed to the docker API
    // when the docker container is created and run.  Not all possible tags
    // are passed through.  See the docker python module for options:
    // https://docker-py.readthedocs.io/en/stable/containers.html
    "docker-params": {
      "ipc_mode": "host"
    }
  }
}CLI EndpointsEach exposed CLI is added as an endpoint using the REST path ofslicer_cli_web/<dockerimage tag andversion>/<clicommand>/runand also using the REST path ofslicer_cli_web/<internalitemid>/run, where<docker image tag and version>is the combined tag and version with slashes, colons, and at signs replaced by underscores.  All command line parameters can be passed as endpoint query parameters.  Input items, folders, and files are specified by their Girder ID.  Input images are specified by a Girder file ID.  Output files are specified by name and with an associated parameter with the same name plus a_foldersuffix with a Girder folder ID.Small Example CLI DockerThe small example CLI docker image can be built locally viadocker build--force-rm-tgirder/slicer_cli_web:small ., or pulled from Docker Hub.Batch ProcessingAll CLIs that take any single item, image, or files as inputs can be run on a set of such resources from a single directory.  For non-batch processing, the
ID of the image, item, or file is passed to<param>.  For batch processing, the ID of a folder is passed to<param>_folderand a regular expression is passed to <param>.  All items in that folder whose name matches the regex are processed.  For images, only items that contain large_images are considered.  For files, the first file in each considered item is used.If two inputs have batch specifications, there must be a one-to-one correspondence between the each of the lists of items determined by the folder ID and regular expression.  All of the lists are enumerated sorted by the lower case item name.When running a batch job, a parent job initiates ordinary (non-batch) jobs.  The parent job will only start another child job when the most recent child job is no longer waiting to start.  This allows non-batch jobs or multiple batch jobs’ children to naturally interleave.  The parent job can be canceled which will stop it from scheduling any more child jobs.Templated InputsAny CLI parameter that takes a value that isn’t a Girder resource identifier can be specified with a Jinja2-style template string.For instance, instead of typing an explicit output file name, one can specify something like{{title}}-{{reference_base}}-{{now}}{{extension}}.  If this were being run on a task called “Radial Blur” on an image called “SampleImage.tiff”, where the output image referenced the image image and had a list of file extensions starting with “.png”, this would end up being converted to the valueRadialBlur-SampleImage-20210428-084321.png.The following template values are handled identically for all parameters:{{title}}: the displayed CLI task title.{{task}}: the internal task name (this usually doesn’t have spaces in it){{image}}: the tag of the Docker image used for the task{{now}}: the local time the job started in the form yyyymmdd-HHMMSS.  You can useyyyy,mm,dd,HH,MM,SSfor the four digit year, and two digit month, day, 24-hour, minute, and second.{{parameter_<name of cliparameter>}}: any parameter that isn’t templated can be referenced by its name.  For instance, in Example1 in the small-docker cli in this repo,{{parameter_stringChoice}}would get replaced by the value passed to the stringChoice parameter.{{parameter_<name of cli parameter>_base}}is the same as the previous item except that if the right-most part of the parameter looks like a file extension, it is removed.  This can be used to get the base name of file parameters.The following template parameters are only handled on the web client:
-{#control:<selector>#}: If specified for the value of a parameter, use the value of the selected field from the DOM.  For instance,{#control:.h-zoom-value#}could get the current image zoom level.There are also template values specific to individual parameters:{{name}}: the name of this parameter.  This usually doesn’t have any spaces in it.{{label}}: the label of the is parameter.  This is what is displayed in the user interface.{{description}}: the description of the parameter.{{index}}: the index, if any, of the parameter.{{default}}: the default value, if any, of the parameter.{{extension}}: the first entry in thefileExtensionvalue of the parameter, if any.{{reference}}: if the parameter has a reference to another parameter, this returns that parameter’s value.  It is equivalent to{{parameter_<reference>}}.{{reference_base}}: the reference value mentioned previously striped of the right-most file extension.If the local (server) environment has any environment variables that begin withSLICER_CLI_WEB_, these are accessible in the templates as{{env_(name)}}.  For instance,SLICER_CLI_WEB_DASK_SERVERwould be accessible as{{env_DASK_SERVER}}.