Package: razel
RazelA command executor with caching. It is:Fast: caching avoids repeated execution of commands which haven't changedReliable: commands are executed in a sandbox to detect missing dependenciesEasy to use: commands are specified using a high-level TypeScript or Python API and convenience functions/tasks are
built-inMade for: data processing pipelines with executables working on files and many dependencies between thoseRazel is not the best choice for building software, especially there's no built-in support for compiler setup and header
dependencies.Getting StartedThe native input format for Razel is arazel.jsonlfile, see the exampletest/razel.jsonl.
It can be run withrazel exec -f test/razel.jsonl.The preferred way is to use one of the high-level APIs. Both allow specifying the commands in an object-oriented style
and provide arun()function which creates therazel.jsonlfile, downloads the nativerazelbinary
and uses it to execute the commands.Paths of inputs files are relative to the workspace (directory ofrazel.jsonl). Output files are created in<cwd>/razel-out.TypeScript APIInstallDenoto use theTypeScript API.
Run theexample Deno script:denorun-A--checktest/deno.ts---vPython APIThePython APIrequires Python >= 3.8.
Install the package and run theexample Python script:pipinstall--upgraderazel
pythontest/python.py-vBatch file (experimental)In addition torazel.jsonl, Razel can directly execute a batch file containing commands.
Input and output files need to be specified, which is WIP.Execute the exampletest/batch.shwith Razel:razelexec-ftest/batch.shRunning in Docker/Podman containerThe workspace directory can be mounted into a container:podmanrun-t-v$PWD:$PWD-w$PWDdenoland/denodenorun-Atest/deno.tsBuilding Razel from sourceUserustupto install Rust. Installprotobuf-compiler. Then runcargo install razel.Project StatusRazel is in active development andnotready for production. CLI and format ofrazel.jsonlwill likely change.OSStatusNoteLinux✓stable, main development platformMac✓used and tested in CIWindows(✓)tested in CI onlyFeatureStatusNotecommand execution in sandbox✓multithreaded execution✓local caching✓remote caching✘WIPremote execution✘TODOOOM handling: retry with less concurrency✓ Linuxrequiressudo cgcreate -a $USER -t $USER -g memory:razelWhy not ...?Bazelis a multi-language build tool. However, for the use case Razel targets, there are some
issues:additional launcher script required for some simple tasksusing stdout of action as input for another actionparsing measurements from stdout of actionCTest features like FAIL_REGULAR_EXPRESSION, WILL_FAILdifficult to get command lines for debuggingno automatic disk usage limit/cleanup for local cache - all temp output needs to fit on diskno native support for response filesresources cannot be reserved to run real-time critical testscontent of bazel-bin/out directories is not defined (contains mixture of current build and cache)CTestis nice for building C/C++ code and CTest can be used
for testing,
but it does not support caching and managing dependencies between tests is difficult.FeaturesMeasurementsRazel parses the stdout of executed commands to capture runtime measurements and writes them torazel-out/razel-metadata/measurements.csv.
Currently, the<CTestMeasurement>and<DartMeasurement>tags as used byCTest/CDashare supported:<CTestMeasurement type="numeric/double" name="score">12.3</CTestMeasurement>
<CTestMeasurement type="text/string" name="result">ok</CTestMeasurement>Supporting custom formats is planned.TagsTags can be set on commands. Any custom string can be used as tag, a colon should be used for grouping.
The tags are added torazel-out/razel-metadata/execution_times.json.
Using tags for filtering commands and creating reports is planned.Tags withrazel:prefix are reserved and have special meaning:razel:quiet: don't be verbose if command succeededrazel:verbose: always show verbose outputrazel:condition: keep running and don't be verbose if command failedrazel:no-cache: always execute a command without cachingrazel:no-remote-cache: don't use remote cacherazel:no-sandbox: disable sandbox and also cache - for commands with unspecified input/output filesConditional execution / Skipping commandCommands can be skipped based on the execution result of another command. Set therazel:conditiontag on a command
and use that one as dependency for other commands.Param/Response filesCommands with huge number of arguments might result in command lines which are too long to be executed by the OS.
Razel detects those cases and replaces the arguments with a response file. The filename starts with @.AcknowledgementsThe idea to build fast and correct is based onBazel. Razel uses data structures from
theBazel Remote Execution APIfor caching.