Package: law
[!NOTE]This project is currently under development.Version 1.0.0 will support Python ≥3.7 and is developed in therelease_prep branch.There will be a legacy branch with Python 2.7 and ≤3.6 support that, however, won't receive any new features.The release is targetted for spring 2024.Use law to build complex and large-scale task workflows.
It is build on top ofluigiand adds abstractions forrun locations,storage locationsandsoftware environments.
Law strictly disentangles these building blocks and ensures they remain interchangeable and resource-opportunistic.Key features:CLI with auto-completion and interactive status and dependency inspection.Remote targets with automatic retries and local cachingWebDAV, HTTP, Dropbox, SFTP, all WLCG protocols (srm, xrootd, dcap, gsiftp, webdav, ...)Automatic submission to batch systems from within tasksHTCondor, LSF, gLite, ARC, Slurm, CMS-CRABEnvironment sandboxing, configurable on task levelDocker, Singularity, Sub-Shells, Virutal envsContentsFirst stepsInstallation and dependenciesUsage at CERNOvercomplete example configProjects using lawExamplesFurther topicsAuto completion on the command-lineDevelopmentTestsContributorsFirst stepsInstallation and dependenciesInstall viapippipinstalllaworconda / (micro)mambaconda install -c conda-forge lawIf you plan to use remote targets, the (default) implementation also requiresgfal2andgfal2-python(optional) to be installed, either via pip or conda / (micro)mamba.condainstall-cconda-forgegfal2gfal2-utilUsage at CERNSee thewiki.Overcomplete example configSeelaw.cfg.example.Projects using lawCMS Di-Higgs Inference Tools:Basis for statistical analysis for all Di-Higgs searches in CMS, starting at datacard-levelrepo,docscolumnflow (+ all analyses using it):Python based, fully automated, columnar framework, including job submission, resolution of systematics and ML pipelines, starting at NanoAOD-level with an optimized multi-threaded column readerrepo,docs,task structureCMS B-Tag SF Measurement:Automated workflow for deriving shape-calibrating b-tag scale factors, starting at MiniAOD-levelrepoCMS Tau POG ML Tools:Preprocessing pipeline for ML trainings in the TAU grouprepoCMS HLT Config Parser:Collects information from various databases (HLT, bril, etc.) and shows menus, triggers paths, filter names for configurable MC datasets or data runsrepoRWTH-CMS Analysis Framework:Basis for multiple CMS analyses ranging from Di-Higgs, to single Higgs and b-tag SF measurements, starting at NanoAOD-level and based on coffea processorsrepoCIEMAT-CMS Analysis Framework:Python and RDataFrame based framework starting from NanoAOD and targetting multiple CMS analysesrepoCMS 3D Z+jet 13TeV analysisAnalysis workflow management from NTuple production to final plots and fitsrepoNP-correction derivation toolMC generation with Herwig and analysis of generated events with RivetrepoCMS SUSY Searches at DESYAnalysis framework for CMS SUSY searches going from custom NanoAODs -> NTuple production -> DNN-based inference -> final plots and fitsrepoKingmaker (CMS Ntuple Production with CROWN)Ntuple conversion from CMS nanoAOD to analysis Ntuples using the CROWN framework. Also includes the training of an event classifier on those ntuples.repo,CROWNIf your project uses law but is not yet listed here, feel free to open a pull request or mention your project details in a newissueand it will be added.ExamplesAll examples can be run either in a Jupyter notebook or a dedicated docker container.
For the latter, dodockerrun-tiriga/law:example<example_name>loremipsum: Thehello worldexample of law.workflows: Law workflows.workflow_parameters: Alternative way of parametrizing workflows with explicit branch parameters.notebooks: Examples showing how to use and work with law in notebooks.dropbox_targets: Working with targets that are stored on Dropbox.wlcg_targets: Working with targets that are stored on WLCG storage elements (dCache, EOS, ...). TODO.htcondor_at_vispa: HTCondor workflows at theVISPA service.htcondor_at_cern: HTCondor workflows at the CERN batch infrastructure.CMS Crab at CERN: CMS Crab workflows executed from lxplus at CERN.sequential_htcondor_at_cern: Continuation of thehtcondor_at_cernexample, showing sequential jobs that eagerly start once jobs running previous requirements succeeded.htcondor_at_naf: HTCondor workflows at GermanNational Analysis Facility (NAF).slurm_at_maxwell: Slurm workflows at theDesy Maxwell cluster.grid_at_cern: Workflows that run jobs and store data on the WLCG.lsf_at_cern: LSF workflows at the CERN batch infrastructure.docker_sandboxes: Environment sandboxing using Docker. TODO.singularity_sandboxes: Environment sandboxing using Singularity. TODO.subshell_sandboxes: Environment sandboxing using Subshells. TODO.parallel_optimization: Parallel optimization usingscikit optimize.notifications: Demonstration of slack and telegram task status notifications..CMS Single Top Analysis: Simple physics analysis using law.Further topicsAuto completion on the command-linebashsource"$(lawcompletion)"zshzsh is able to load and evaluate bash completion scripts viabashcompinit.
In order forbashcompinitto work, you should runcompinstallto enable completion scripts:autoload-Uzcompinstall&&compinstallAfter following the instructions, these lines should be present in your~/.zshrc:# The following lines were added by compinstallzstyle:compinstallfilename'~/.zshrc'autoload-Uz+Xcompinit&&compinit
autoload-Uz+Xbashcompinit&&bashcompinit# End of lines added by compinstallIf this is the case, just source the law completion script (which internally enablesbashcompinit) and you're good to go:source"$(lawcompletion)"DevelopmentSource hosted atGitHubReport issues, questions, feature requests onGitHub IssuesTestsTo run and test law, there are various dockerriga/lawimages available on theDockerHub, corresponding to different OS and Python versions (based onmicromamba).
Start them viadockerrun-tiriga/law:<the_tag>OSPythonTagsAlmaLinux 93.11a9-py311, a9-py3, a9, py311, py3, latestAlmaLinux 93.10a9-py310, py310AlmaLinux 93.9a9-py39, py39AlmaLinux 93.8a9-py38, py38AlmaLinux 93.7a9-py37, py37CentOS 83.11c8-py311, c8-py3, c8CentOS 83.10c8-py310CentOS 83.9c8-py39CentOS 83.8c8-py38CentOS 83.7c8-py37CentOS 73.10c7-py310, c7-py3, c7CentOS 73.9c7-py39CentOS 73.8c7-py38CentOS 73.7c7-py37CentOS 73.6c7-py36, py36 (removed soon)CentOS 72.7c7-py27, c7-py2, py27, py2 (removed soon)ContributorsMarcel Rieger💻👀🚧📖Peter Fackeldey💻Yannik Rath💻Jaime Leon Holgado💻Louis Moureaux💻Lukas Geiger💻Valentin Iovene💻This project follows theall-contributorsspecification.