Package: rapids-dependency-file-generator
rapids-dependency-file-generatorrapids-dependency-file-generatoris a Python CLI tool that generates condaenvironment.yamlfiles andrequirements.txtfiles from a single YAML file, typically nameddependencies.yaml.When installed, it makes therapids-dependency-file-generatorCLI command available which is responsible for parsing adependencies.yamlconfiguration file and generating the appropriate condaenvironment.yamlandrequirements.txtdependency files.Table of ContentsInstallationUsagedependencies.yamlFormatfileskeychannelskeydependencieskeyHow Dependency Lists Are MergedAdditional CLI NotesExamplesInstallationrapids-dependency-file-generatoris available onPyPI. To install, run:pipinstallrapids-dependency-file-generatorUsageWhenrapids-dependency-file-generatoris invoked, it will read adependencies.yamlfile from the current directory and generate children dependency files.Thedependencies.yamlfile has the following characteristics:it is intended to be committed to the root directory of repositoriesit can define matrices that enable the output dependency files to vary according to any arbitrary specification (or combination of specifications), including CUDA version, machine architecture, Python version, etc.it contains bifurcated lists of dependencies based on the dependency's purpose (i.e. build, runtime, test, etc.). The bifurcated dependency lists are merged according to the description in theHow Dependency Lists Are Mergedsection below.dependencies.yamlFormatTheExamplessection below has instructions on where exampledependency.yamlfiles and their corresponding output can be viewed.Thedependencies.yamlfile has three relevant top-level keys:files,channels, anddependencies. These keys are described in detail below.filesKeyThe top-levelfileskey is responsible for determining the following:which types of dependency files should be generated (i.e. condaenvironment.yamlfiles and/orrequirements.txtfiles)where the generated files should be written to (relative to thedependencies.yamlfile)which variant files should be generated (based on the provided matrix)which of the dependency lists from the top-leveldependencieskey should be included in the generated filesHere is an example of what thefileskey might look like:files:all:# used as the prefix for the generated dependency file names for conda or requirements files (has no effect on pyproject.toml files)output:[conda,requirements]# which dependency file types to generate. required, can be "conda", "requirements", "pyproject", "none" or a list of non-"none" valuesconda_dir:conda/environments# where to put conda environment.yaml files. optional, defaults to "conda/environments"requirements_dir:python/cudf# where to put requirements.txt files. optional, but recommended. defaults to "python"pyproject_dir:python/cudf# where to put pyproject.toml files. optional, but recommended. defaults to "python"matrix:# (optional) contains an arbitrary set of key/value pairs to determine which dependency files that should be generated. These values are included in the output filename.cuda:["11.5","11.6"]# which CUDA version variant files to generate.arch:[x86_64]# which architecture version variant files to generate. This value should be the result of running the `arch` command on a given machine.includes:# a list of keys from the `dependencies` section which should be included in the generated files-build-test-runtimebuild:# multiple `files` children keys can be specifiedoutput:requirementsconda_dir:conda/environmentsrequirements_dir:python/cudfmatrix:cuda:["11.5"]arch:[x86_64]py:["3.8"]includes:-buildThe result of the above configuration is that the following dependency files would be generated:conda/environments/all_cuda-115_arch-x86_64.yamlconda/environments/all_cuda-116_arch-x86_64.yamlpython/cudf/requirements_all_cuda-115_arch-x86_64.txtpython/cudf/requirements_all_cuda-116_arch-x86_64.txtpython/cudf/requirements_build_cuda-115_arch-x86_64_py-38.txtTheall*.yamlandrequirements_all*.txtfiles would include the contents of thebuild,test, andruntimedependency lists from the top-leveldependencykey. Therequirements_build*.txtfile would only include the contents of thebuilddependency list from the top-leveldependencykey.The value ofoutputcan also benoneas shown below.files:test:output:noneincludes:-testWhenoutput: noneis used, theconda_dir,requirements_dirandmatrixkeys can be omitted. The use case foroutput: noneis described in theAdditional CLI Notessection below.extrasA given file may include anextrasentry that may be used to provide inputs specific to a particular file typeHere is an example:files:build:output:pyprojectincludes:# a list of keys from the `dependencies` section which should be included in the generated files-buildextras:table:table_namekey:key_nameCurrently the supported extras by file type are:pyproject.tomltable: The table in pyproject.toml where the dependencies should be written. Acceptable values are "build-system", "project", and "project.optional-dependencies".key: The key corresponding to the dependency list intable. This may only be provided for the "project.optional-dependencies" table since the key name is fixed for "build-system" ("requires") and "project" ("dependencies"). Note that this implicitly prohibits including optional dependencies via an inline table under the "project" table.channelsKeyThe top-levelchannelskey specifies the channels that should be included in any generated condaenvironment.yamlfiles.It might look like this:channels:-rapidsai-conda-forgeIn the absence of achannelskey, some sensible defaults for RAPIDS will be used (seeconstants.py).dependenciesKeyThe top-leveldependencieskey is where the bifurcated dependency lists should be specified.Underneath thedependencieskey are sets of key-value pairs. For each pair, the key can be arbitarily named, but should match an item from theincludeslist of anyfilesentry.The value of each key-value pair can have the following children keys:common- contains dependency lists that are the same across all matrix variationsspecific- contains dependency lists that are specific to a particular matrix combinationThe values of each of these keys are described in detail below.commonKeyThecommonkey contains a list of objects with the following keys:output_types- a list of output types (e.g. "conda" forenvironment.yamlfiles or "requirements" forrequirements.txtfiles) for the packages in thepackageskeypackages- a list of packages to be included in the generated output filespecificKeyThespecifickey contains a list of objects with the following keys:output_types-same asoutput_typesfor thecommonkey abovematrices- a list of objects (described below) which define packages that are specific to a particular matrix combinationmatricesKeyEach list item under thematriceskey contains amatrixkey and apackageskey.
Thematrixkey is used to define which matrix combinations fromfiles.[*].matrixwill use the associated packages.
Thepackageskey is a list of packages to be included in the generated output file for a matching matrix.
This is elaborated on inHow Dependency Lists Are Merged.An example of the above structure is exemplified below:dependencies:build:# dependency list namecommon:# dependencies common among all matrix variations-output_types:[conda,requirements]# the output types this list item should apply topackages:-common_build_dep-output_types:condapackages:-cupy-pip:# supports `pip` key for conda environment.yaml files-some_random_depspecific:# dependencies specific to a particular matrix combination-output_types:conda# dependencies specific to conda environment.yaml filesmatrices:-matrix:cuda:"11.5"packages:-cudatoolkit=11.5-matrix:cuda:"11.6"packages:-cudatoolkit=11.6-matrix:# an empty matrix entry serves as a fallback if there are no other matrix matchespackages:-cudatoolkit-output_types:[conda,requirements]matrices:-matrix:# dependencies specific to x86_64 and 11.5cuda:"11.5"arch:x86_64packages:-a_random_x86_115_specific_dep-matrix:# an empty matrix/package entry to prevent error from being thrown for non 11.5 and x86_64 matchespackages:-output_types:requirements# dependencies specific to requirements.txt filesmatrices:-matrix:cuda:"11.5"packages:-another_random_dep=11.5.0-matrix:cuda:"11.6"packages:-another_random_dep=11.6.0test:common:-output_types:[conda,requirements]packages:-pytestHow Dependency Lists Are MergedThe information from the top-levelfilesanddependencieskeys are used to determine which dependencies should be included in the final output of the generated dependency files.Consider the following top-levelfileskey configuration:files:all:output:condaconda_dir:conda/environmentsrequirements_dir:python/cudfmatrix:cuda:["11.5","11.6"]arch:[x86_64]includes:-build-testIn this example,rapids-dependency-file-generatorwill generate two conda environment files:conda/environments/all_cuda-115_arch-x86_64.yamlandconda/environments/all_cuda-116_arch-x86_64.yaml.Since theoutputvalue isconda,rapids-dependency-file-generatorwill iterate through anydependencies.build.commonanddependencies.test.commonlist entries and use thepackagesof any entry whoseoutput_typeskey iscondaor[conda, ...].Further, for the11.5andx86_64matrix combination, anybuild.specificandtest.specificlist items whose output includescondaand whosematriceslist items matches any of the definitions below would also be merged:specific:-output_types:condamatrices:-matrix:cuda:"11.5"packages:-some_dep1-some_dep2# orspecific:-output_types:condamatrices:-matrix:cuda:"11.5"arch:"x86_64"packages:-some_dep1-some_dep2# orspecific:-output_types:condamatrices:-matrix:arch:"x86_64"packages:-some_dep1-some_dep2Everymatriceslist must have a match for a given input matrix (only the first matching matrix in the list ofmatriceswill be used).
If no matches are found for a particular matrix combination, an error will be thrown.
In instances where an error should not be thrown, an emptymatrixandpackageslist item can be used:-output_types:condamatrices:-matrix:cuda:"11.5"arch:x86_64py:"3.8"packages:-a_very_specific_115_x86_38_dep-matrix:# an empty matrix entry serves as a fallback if there are no other matrix matchespackages:Merged dependency lists are sorted and deduped.Additional CLI NotesInvokingrapids-dependency-file-generatorwithout any arguments is meant to be the default behavior for RAPIDS developers. It will generate all of the necessary dependency files as specified in the top-levelfilesconfiguration.However, there are CLI arguments that can augment thefilesconfiguration values before the files are generated.Consider the example whenoutput: noneis used:files:test:output:noneincludes:-testThetestfile generated by the configuration above is useful for CI, but it might not make sense to necessarily commit those files to a repository. In such a scenario, the following CLI arguments can be used:ENV_NAME="cudf_test"rapids-dependency-file-generator\--file_key"test"\--output"conda"\--matrix"cuda=11.5;arch=$(arch)">env.yaml
mambaenvcreate--fileenv.yaml
mambaactivate"$ENV_NAME"# install cudf packages built in CI and test them in newly created environment...The--file_keyargument is passed thetestkey name from thefilesconfiguration. Additional flags are used to generate a single dependency file. When the CLI is used in this fashion, it will print tostdoutinstead of writing the resulting contents to the filesystem.The--file_key,--output, and--matrixflags must be used together.--matrixmay be an empty string if the file that should be generated does not depend on any specific matrix variations.The--prepend-channelsargument accepts a semi-colon separated list of additional channels to use, likerapids-dependency-file-generator --prepend-channels "my_channel;my_other_channel".
If both--outputand--prepend-channelsare provided, the output format must be conda.
Prepending channels can be useful for adding local channels with packages to be tested in CI workflows.Runningrapids-dependency-file-generator -hwill show the most up-to-date CLI arguments.ExamplesThetests/examplesdirectory has exampledependencies.yamlfiles along with their corresponding output files.To create newexampletests do the following:Create a new directory with adependencies.yamlfile intests/examplesEnsure theoutputdirectories (e.g.conda_dir,requirements_dir, etc.) are set to write tooutput/actualRunrapids-dependency-file-generator --config tests/examples/<new_folder_name>/dependencies.yamlto generate the initial output filesManually inspect the generated files for correctnessCopy the contents ofoutput/actualtooutput/expected, so it will be committed to the repository and used as a baseline for future changesAdd the new folder name totest_examples.py