Package: pypi2nix
pypi2nixis a command line tool that generatesNix expressionsfrom
differentrequirements.txt. This is useful for:Building a Nix derivation for a program written in Python as part of
packaging it.Setting up a development environment to hack on a program written in Python.The only way we can fix bugs with pypi2nix is if you report them. Please
create an issue if you discover problems.pypi2nixwill (until further notice) only work with latestunstablechannel. This is due to ongoing changes in Python infrastructure happening in
Nixpkgs.TheNixpkgs manual section about Pythonmakes good reading if you
haven’t seen it already.Contents1. InstallationSystem RequirementsAd hoc Installation (Simple)Declarative Installation (Advanced)2. UsageWhat is being generatedBuilding generated packagesUsing generated packagesCommand line options3. When it doesn’t workNon-Python/system dependenciesAdditional environment variablesUsing requirements_override.nixShared overrides4. Advanced UseCreating default.nix for your projectConvert generated requirements.nix into nixpkgs overlay5. Help developing pypi2nixTestingIntegration testsMaintainance scriptsVersion bumping1. Installationpypi2nixis part ofnixpkgs.  If you just want to usepypi2nixon your system, it is recommended that you install it via
the regular means, e.g.nix-env-iAnixos.pypi2nixon NixOS ornix-env-iAnixpkgs.pypi2nixon other systems utilizing nix.System RequirementsMake sure Nix is installed:% curl https://nixos.org/nix/install | shCurrentlypypi2nixis only tested againstlinuxsystems.
Supportednixpkgschannels arenixos-19.09andnixos-unstable.  Due to the nature ofnixos-unstablethe
occasional breakage ofpypi2nixis to be expected.  We try to
provide fixes in that regard in a timely manner.Ad hoc Installation (Simple)For just installing the package with a command, usenix-env:nix-env -if https://github.com/nix-community/pypi2nix/tarball/masterDeclarative Installation (Advanced)If you prefer to explicitly declare each installed package in your
Nix(OS) or project configuration, you can do the following:First, import the package from itsdefault.nixby fetching the
whole git repository withpkgs.fetchgit.  Afterwards you can just
add the imported attribute the list of installed software.Below you find an example for NixOS’configuration.nix. Other
methods likehome-managerwork similar:let
  pypi2nix = import (pkgs.fetchgit {
    url = "https://github.com/nix-community/pypi2nix";
    # adjust rev and sha256 to desired version
    rev = "v2.0.1";
    sha256 = "sha256:0mxh3x8bck3axdfi9vh9mz1m3zvmzqkcgy6gxp8f9hhs6qg5146y";
  }) {};
in
  environment.systemPackages = [
    # your other packages
    pypi2nix
  ];2. UsageThe easiest way to generate a Nix expressions is to invoke:% pypi2nix -e packageA -e packageB==0.1If you also have arequirements.txtfile for your Python project you can use
the-roption.% pypi2nix -e packageA -e packageB==0.1 \
    -r requirements.txt -r requirements-dev.txtWhat is being generatedOption-Vtells pypi2nix which python version to be used. To see which
Python versions are available consultpypi2nix--help.Once Nix expressions are generated you should be able to see 3 new files:requirements_frozen.txt- full frozen set for your for your pypi2nix call.
This is the output you would expect frompip freeze.requirements.nixis a file which contains a nix expression to for the package set that was built.requirements_override.nix- this is an empty file which lets you
override generated nix expressions.Building generated packagesBuild one package:% nix build -f requirements.nix packages.empyBuild all packages:% nix build -f requirements.nix packagesBuild python interpreter with all packages loaded:% nix build -f requirements.nix interpreter
% ./result/bin/python -c "import empy"Enter development environment:% nix run -f requirements.nix interpreter
[user@localhost:~/dev/nixos/pypi2nix) % python -c "import empy"Using generated packagesIf you are working on a project where its dependencies are defined inrequirements.txtthen you can create adefault.nixand add generated
packages asbuildInputs, as demonstrated here:{}:
let
  python = import ./requirements.nix { inherit pkgs; };
in python.mkDerivation {
  name = "ProjectA-1.0.0";
  src = ./.;
  buildInputs = [
    python.packages."coverage"
    python.packages."flake8"
    python.packages."mock"
    python.packages."pytest"
    python.packages."pytest-asyncio"
    python.packages."pytest-cov"
    python.packages."pytest-mock"
    python.packages."pytest-xdist"
    python.packages."virtualenv"
  ];
  propagatedBuildInputs = [
    python.packages."aiohttp"
    python.packages."arrow"
    python.packages."defusedxml"
    python.packages."frozendict"
    python.packages."jsonschema"
    python.packages."taskcluster"
    python.packages."virtualenv"
  ];
  ...
}As you can see you can access all packages viapython.packages."<name>". If
you want to depend onallpackages you can even do:propagatedBuildInputs = builtins.attrValues python.packages;Command line options-vIncrease amount and detail of information output to the user.
Verbosity levels areERROR,WARNING,INFOandDEBUGin that order.  The default verbosity isINFO.-qReduce amount and detail of information output to the user.  See-vfor more information.-I/--nix-pathTEXTAdd entries to theNIX_PATHenvironment variable similarly to
how-Iworks withnixexecutables likenix-build.
This can be useful for generating package sets based on a
differentnixpkgsversion than the one used one the local
system.--nix-shellPATHPath to an alternative version of thenix-shellcommand.  The
default is the first executable that will be found in the currentPATHof the system.--versionShow the current version ofpypi2nix--basenameTEXTThis option determins the name the produced files.  So with--basenameenvironmentyou would get the filesenvironment.nix,environment_frozen.nixandenvironment_override.nix.--extra-build-inputs/-ETEXTExtra build inputs that the required python packages need to run,
e.g.libffiorlibgl.  In that case you would provide-E"libffi libgl".  These nix packages will be available in the
build environment for the wheels.--emit-extra-build-inputs/--no-emit-extra-build-inputsThese options let you control if external build dependencies
specified via-Ewill end up in the generated nix package set.
Please note that if you select this option, your overrides need to
make sure that python packages find their respective external
dependencies.--extra-env/-NTEXTExtra environment variables that will be passed to the build
environment.  Note that you can use nix expressions in this
string, e.g.-N'BERKELEYDB_DIR=${pkgs.db.dev}'.--enable-tests/-TSpecify this flag if you want to enable the check phase of all
packages in the generated nix expression.  Please note that this
feature is highly exprimental and will probably not work for your
use case.--python-version/-VSpecify the python version you want the requirement set to be
built with.  The default is3which translates to thepython3derivation ofnixpkgs.--requirements/-rFILESpecify a requirements file, similar as you would withpip.pypi2nixtries to be fully compatible with the file format ofpip.--editable/-eTEXTThis option allows you to specify individual requirements that get
added to the requirement set, e.g.pypi2nix-eattrs,pypi2nix-e$HOME/src/myproject#egg=myprojectorpypi2nix-e.#egg=myegg.--setup-requires/-sTEXTAllows you to specify python packages that need to be present in
the build environment of other packages, a good example of this
would besetuptools-scm.  Note thatpypi2nixtries to
detect these dependencies on its own.  You only need to specify
this flag in cases where a package author or maintainer forgot to
mention build time dependencies in their setup or neithersetup.cfgnorpyproject.tomlis used.--overrides/-OURLAllows you to specify additional overrides that conform to the
general structure ofrequirements_override.nix.  We support
regular URLs withhttpandhttpsscheme and alsogit.
An example for usinghttpswould bepypi2nix-Ohttps://myoverrides.test/overrides.nix.  Reusing an overlay from
a git repository would be done like so:pypi2nix-Ogit+https://github.com/nix-community/pypi2nix.git&path=requirement_override.nix.
Please keep in mind that these overrides are incorporated in a nix
expression with a precalculated hash value.  So if the file
changes upstream your generated package can not be built anymore.--default-overrides/--no-default-overridesPull in overrides fromhttps://github.com/nix-community/pypi2nix-overrides.  This
feature is enabled by default.--wheels-cache/-WTEXTA location where prebuilt wheels can be found.  This option will
ultimately be passed topip--find-links.  Only point to
wheels that are built throughpypi2nixon your own or a very
similar system.--build-directoryTEXTWarningA bug inpypi2nixcurrently prevents some packages
from being built with this option set.  It is recommended to not
use this flag.The directory where pypi2nix would build the python environment to
generate the desired nix expression.  If not specified, the build
directory will be temporary and is deleted before the program
exits.3. When it doesn’t workI hope nobody is expectingpypi2nixto do always a perfect job. In Python
packaging, there are just too many different cases that we will never be able to
cover. Whatpypi2nixtries to do is to get you very close.Sometimespypi2nixfails entirely. If this happens, open a bug –
it’s almost always a bug inpypi2nix. However, sometimespypi2nixsucceeds but the resultingrequirements.nixfile
fails during the building of your Python package. Depending on what
the problem is, this section may be helpful.Non-Python/system dependenciesQuite a few Python packages require non-Python dependencies to be
present at build time. These packages will fail to build with error
messages about not being able to findfoo.hor somefooconfigfile. To work around this,pypi2nixhas-Eoptions which can
be used to include extra non-Python dependencies.For example,psycopg2requirespg_configbinary to be present at installation time:% pypi2nix -v -V 2.7 -e psycopg2 -E postgresqllxmlrequireslibxml2andlibxsltsystem package:% pypi2nix -v -V 2.7 -e lxml -E libxml2 -E libxsltAdditional environment variablesSome packages expect additional environment variables to be set:% pypi2nix -v -V 2.7 -e bsddb3 -N 'BERKELEYDB_DIR=${pkgs.db.dev}'Using requirements_override.nixSome other failures might be caused because the derivation thatpypi2nixwrote was incomplete. A very common situation is thatpypi2nixdidn’t include all the dependencies of some package. As
an example,execnetdepends onsetuptools-scm, butpypi2nixmay not detect this.When this happens, Nix will fail to buildexecnet, perhaps with an
error message from distutils/setuptools complaining that it can’t find
a distribution forsetuptools-scm. What’s happening here is that
normallyexecnetwould fetchsetuptools-scmfrom PyPI, but Nix
disables network access to guarantee reproducability. So when you
buildexecnet, it fails to findsetuptools-scm.For these situations,pypi2nixprovides arequirements_override.nixfile, which lets you override anything
that it generated. You can even add new packages to the dependency set
this way.As an example, let’s addsetuptools-scmas a build-time dependency
ofexecnet. Here’s therequirements_override.nix:{ pkgs, python }:

self: super: {

  "execnet" = python.overrideDerivation super."execnet" (old: {
    buildInputs = old.buildInputs ++ [ self."setuptools-scm" ];
  });

}In a similar way, you can add or remove any Python package.Shared overridesIn addition to the empty autogeneratedrequirements_overrides.nixfile, you can include pre-existing overrides files.  These overrides
will be included the same way as yourrequirements_overrides.nix.Thepypi2nixauthor also maintains a set of “default” overrides athttps://github.com/nix-community/pypi2nix-overrides/blob/master/overrides.nix–
you can include these by using the--default-overridesargument topypi2nix. These overrides are designed in such a way that they
only override dependencies that were already present in yourrequirements.nix.You can also include an overrides file using the-Ocommand line
argument.pypi2nixcan fetch these overrides from a local file or
over certain common protocols.httpandhttpspypi2nix-V3--overrideshttps://raw.githubusercontent.com/nix-community/pypi2nix-overrides/master/overrides.nixNote that the generated Nix expression will check if contents of
the overrides file differs from when the Nix expression was built, and
fail if this was the case (or the file does not exist anymore).Local filespypi2nix-V3--override../some/relative/path--override/some/absolute/pathGit repositoriespypi2nix-V3--overridegit+https://github.com/nix-community/pypi2nix.git#path=overrides.nixIf you want to import a file from a specific git repository you have
to prefix its URL withgit+, quite similar to how you would do
in arequirements.txtfile forpip.4. Advanced UseCreating default.nix for your projectNothing speaks better than an example:{ }:

let
  pkgs = import <nixpkgs> {};
  python = import ./requirements.nix { inherit pkgs; };
in python.mkDerivation {
  name = "projectA-1.0.0";
  src = ./.;
  buildInputs = [
    python.packages."coverage"
    python.packages."flake8"
    python.packages."mock"
    python.packages."pytest"
    python.packages."pytest-asyncio"
    python.packages."pytest-cov"
    python.packages."pytest-mock"
    python.packages."pytest-xdist"
  ];
  propagatedBuildInputs = [
    python.packages."aiohttp"
    python.packages."arrow"
    python.packages."defusedxml"
    python.packages."frozendict"
    python.packages."jsonschema"
  ];
  checkPhase = ''
    export NO_TESTS_OVER_WIRE=1
    export PYTHONDONTWRITEBYTECODE=1

    flake8 src/
    py.test --cov=src -cov-report term-missing
    coverage html
  '';
}Important to know here is that you instantiate all generated packages
aspython = import ./requirements.nix { inherit pkgs; };which
gives you a Python environment with all the packages generated bypypi2nixas well as some common utilities.To create a package you usepython.mkDerivationwhich works like
thepythonPackages.buildPythonPackagefunction innixpkgs. All
generated packages are available as one attribute set underpython.packages.One of future goals ofpypi2nixproject is to also improve the UX of our
Python tooling in nixpkgs. While this is very hard to do withinnixpkgsit
is almost trivial to experiment with this outsidenixpkgs.Convert generated requirements.nix into nixpkgs overlayA working example is worth 1000 words.overlay.nix:self: super:
{
  customPython =
      (import ./requirements.nix { pkgs = self; });
}shell.nix:with (import <nixpkgs> { overlays = [ (import ./overlay.nix) ]; });
customPython.interpreter5. Help developing pypi2nixClonepypi2nix repositoryand usingnix-shellcommand enter development
environment.:% git clone https://github.com/nix-community/pypi2nix
% cd pypi2nix
% nix-shellCode is located insrc/pypi2nix.TestingPypi2nix comes with two kinds of tests: unit tests and integration
tests.  They can be found in the folders/unittestsand/integrationtestsrespectively.Unit tests are straight forward.  They are run viapytestand (try
to) followpytestbest practices.  Idealy all of pypi2nix’s code
should be covered by unittests.  If possible unittests should not go
online and fetch data from the internet.  If this cannot be avoided
use the@nixdecorator, found inunittests.switchesto mark
tests that require network access.Integration testsIntegration tests are a little bit more involved.  We implemented a
small framework to write new tests and maintain old ones.  Check outintegrationtests.frameworkfor information on how to write custom
integration tests.  To run all integration tests runrun_integration_tests.pyfrom thescriptsdirectory.  If you
usenix-shellto create your development environment then thescriptsdirectory should be in youPATHvariable.Please note that all integration test cases are classes deriving fromintegrationtests.framework.IntegrationTest.  Also all these tests
must end withTestCase, e.g.MyCustomTestCase.Maintainance scriptsThescriptsfolder contains programs that help to maintain the
repository.  We expect the user to have all the packages from the
build environment of pypi2nix installed.  We register thescriptsdirectory in the usersPATHif they choose to enternix-shellin
the top level directory of this project.Version bumpingWe usebumpvto manage the current version of this project.  This
program should be part of the development environment.