Package: pymsbuild
pymsbuildThis is a PEP 517 backend for building packages via MSBuild ordotnet build.Configuration fileThe file is named_msbuild.py, and is executed by runningpython -m pymsbuild.The package definition specifies all the files that end up in the released packages.frompymsbuildimport*METADATA={"Metadata-Version":"2.1","Name":"package","Version":"1.0.0","Author":"My Name","Author-email":"myemail@example.com","Description":File("README.md"),"Description-Content-Type":"text/markdown","Classifier":["Development Status :: 3 - Alpha","Intended Audience :: Developers","Programming Language :: Python :: 3.9",],}PACKAGE=Package("my_package",PyFile(r"my_package\*.py"),PydFile("_accelerator",CSourceFile(r"win32\*.c"),IncludeFile(r"win32\*.h"),),Package("subpackage",PyFile(r"subpackage\*.py"),),)Note that subpackagesmustbe specified as aPackageelement, as the
nesting ofPackageelements determines the destination path. Otherwise you
will find all of your files flattened. Recursive wildcards are supported, however,
be aware that it is not always intuitive how the paths are going to be remapped.Also note that without asource=named argument, all source paths are
relative to the configuration file.pyproject.toml fileYou will need this file in order forpipto build your sdist, but otherwise it's
generally easier and faster to usepymsbuilddirectly.[build-system]
requires = ["pymsbuild"]
build-backend = "pymsbuild"On Windows, aVisual Studioinstallation will be
required. It will be automatically detected, or the path to MSBuild can be
provided as theMSBUILDenvironment variable.On other platforms, the.NET SDKwill be
required. Thedotnetcommand must be available onPATHor specified as theMSBUILDenvironment variable.If you have additional requirements for building either sdist or wheels, add
them asBuildSdistRequiresorBuildWheelRequiresvalues inMETADATA. They
will be parsed afterinit_METADATAand/orinit_PACKAGEhave been called, so
may be modified by these functions.[project] table supportThere is no support for the[project]table at this time. All metadata that is written into the final distribution
files comes from your_msbuild.pyfile.However, thepyproject.tomlincluded in sdists is a direct copy of the one
from the root of your project. Other than the project table, sdists have no
predictable metadata for analysis tools to use, so if you want your project to
provide that metadata, feel free to list it in thepyproject.tomlas well as
in your_msbuild.py(remembering to mark asdynamicanything that is updated by your build process).A future release may automatically use_msbuild.pymetadata to fill out
missing fields in apyproject.tomlproject table, andpymsbuild initmay use
the project table to initialise the configuration file. However, at this point,
both files are totally independent and the configuration file is the canonical
source of metadata.UsageRebuild the current project in-place.python -m pymsbuildInteractively generate the_msbuild.pyfile with project spec.(Or at least, it will, once implemented.)python -m pymsbuild initBuild the project and output an sdistpython -m pymsbuild sdistOutput is put intodistby default, but can be overridden with--dist-dir(-d).Build the project and output a wheelpython -m pymsbuild wheelOutput is put intodistby default, but can be overridden with--dist-dir(-d).Clean any recent buildspython -m pymsbuild cleanAdvanced ExamplesDynamic METADATAMetadata may be dynamically generated, either on import or with theinit_METADATAfunction. This function is called and must either
return the metadata dict to use, or updateMETADATAdirectly.However, if aPKG-INFOfile is found adjacent to the configuration
file, it will be used verbatim. Sdist generation adds this file, so all
metadata is static from that point onward.init_METADATAis not
called in this case.frompymsbuildimport*METADATA={"Metadata-Version":"2.1","Name":"package","Version":os.getenv("VERSION","1.0.0"),"Author":"My Name","Author-email":"myemail@example.com","Description":File("README.md"),"Description-Content-Type":"text/markdown","Classifier":["Development Status :: 3 - Alpha","Intended Audience :: Developers","Programming Language :: Python :: 3.9",],}definit_METADATA():ifos.getenv("BUILD_BUILDNUMBER"):METADATA["Version"]=f"1.0.{os.getenv('BUILD_BUILDNUMBER','')}"# Updated METADATA directly, so no need to return anythingAlso see the earlier section regarding thepyproject.tomlproject table (and
the fact that it is not used bypymsbuild, but will be added to your sdist
without modification).Separate packagesPackages are just Python objects, so they may be kept in variables and
used later. They also expose amembersattribute, which is a list, so
that members can be added or inserted later, as well asfind,findallandinsertmethods to help (see theDynamic Packagessection below).After the entire module is executed, the package inPACKAGEis the
only one used to generate output.P1=Package("submodule",PyFile(r"src\submodule\__init__.py"))P2=Package("submodule_2",PyFile(r"src\submodule_2\__init__.py"))PACKAGE=Package("my_package",P1)PACKAGE.members.append(P2)Wildcard handlingFiles can be added recursively using wildcard operators. These are
evaluated at generation time bypymsbuildand not by MSBuild/dotnet build, as it allows greater control over target names.PACKAGE=Package("my_package",# All .py files, relative to the 'src' directoryPyFile(r"**\*.py"),# All license files, if any, with path separators converted to '-'File(r"**\license*",flatten="-",allow_none=True),# All .bin files from all data directories, moved to the rootFile(r"**\data\*.bin",flatten=True),source="src")flattenspecifies the string sequence to replace path separators in
the name. PassingTrueindicates that only the file name should be
retained.allow_nonemerely suppresses a build-time error when the wildcard
fails to match any files. This is usually an important problem, and
should be suppressed with care.Theflattenandallow_noneproperties are not written to the build
file. However, they are case-sensitive while MSBuild is not, so the
capitalised versions will be ignored for this processing and passed
through.Final install location (also known as the element's name) are generated
from the default name (source file name ornameargument and all
package names in the hierarchy) combined with the pattern according to
these rules:if the pattern contains no wildcards, the default name is preservedeach segment from the first one containing a wildcard will be joined
to the parent of the default nameif the pattern filename contains no wildcards, it is preserved in the
final name. Otherwise, it is replaced by matched filesThese rules ensure consistency across many forms of paths, making it
reliable to use calculated absolute paths with wildcards (for example,
a package extending the build system to add its own files). To create
a directory in the destination, use a newPackageelement:# Installs as 'A/__init__.py'
PACKAGE = Package("A", PyFile("B/__init__.py"))
PACKAGE = Package("A", PyFile("B/source.py", name="__init__.py"))

# Installs as 'A/*.txt'
PACKAGE = Package("A", File("B/*.txt"))

# All of these install as 'A/B/*.txt'
PACKAGE = Package("A", Package("B", File("B/*.txt")))
PACKAGE = Package("A", Package("B", File("*.txt"), source="B"))
PACKAGE = Package("A", File("*/*.txt"))  # assuming no other matchesSpecifying theNamemetadata (as opposed toname, which is a
keyword argument) will override the destination name of every matched
file. This is applied before flattening, and so will preserve the
relative path in whatever form is specified byflatten. To bypass
this additional processing and use the name as an MSBuild literal,
wrap it in aConditionalValuewith no condition:PACKAGE=Package("my_package",File("**/*.dat",Name=ConditionalValue("%(Filename)-1.dat")),)For more complex transforms on filename, we recommend using theinit_PACKAGEfunction described below.Dynamic packagesAfter metadata processing, if aninit_PACKAGE(tag=None)function
exists it will be called with the intended platform tag. It must modify
or returnPACKAGE. This function is called for in-place, sdist and
wheel generation, however, for sdists (and any scenario that should not
generate binaries),tagwill beNone. Otherwise, it will be a
string likecp38-cp38-win32.PACKAGE=Package("my_package",PyFile(r"my_package\*.py"),)definit_PACKAGE(tag=None):iftagandtag.endswith("-win_amd64"):data_file=generate_data_amd64()PACKAGE.members.append(File(data_file))Note that all files to be included in an sdist must be referenced whentagisNone. Conditional compilation is best performed using conditions
in the package elements, rather than usinginit_PACKAGE. However, if you
are going to useinit_PACKAGE, you shouldremoveelements rather than
adding them if they should be included in your sdist.Files added as part of a wildcard can be removed by adding aRemoveFileelement. These may be added dynamically duringinit_PACKAGE, and must
appear after the element that included the files.PACKAGE=Package("my_package",PyFile(r"my_package\*.py"),RemoveFile(PyFile,r"my_package\_internal.py"),)To exclude files from a wildcard in the first place, chain the.excludingmethod on the original element. The pattern will be evaluated in exactly the
same way as the inclusion pattern, and any paths that match will be omitted.PACKAGE=Package("my_package",PyFile(r"my_package\*.py").excluding("my_package\internal*.py"),)Alternatively, a condition may be added to the file pattern to only include
files matching MSBuild style conditions. Because these will be applied to
item groups, the%()metadata syntax should be used to access information
for the element being added. Either the.if_method or theConditionalValuewrapper may be used.PACKAGE=Package("my_package",PyFile(r"my_package\*.py").if_("%(Filename) != '_internal'"),File(ConditionalValue("*.txt",condition="%(Filename.StartsWith(`internal`))")),)Package members can be located during the dynamic stage using thefindandfindallfunctions. These take a path of member identifiers
(typically their name property) and will return those that match.'**'segments are supported for recursive searches.PACKAGE=Package("my_package",Package("sub1",File("license.txt")),Package("sub2",File("license.txt")),)definit_PACKAGE(tag=None):foreinPACKAGE.findall("sub*/license.txt"):e.name="LICENSE"When inserting members, theinsertfunction combines afindwith
the insert, and supports offset and range options. In general, only
subclassed element types should insert additional elements, and only
into themselves at construction.classMyPydFile(PydFile):def__init__(self,name,*members,**options):super().__init__(name,*members,**options)self.insert(# Member path to insert before - this one is inherited from PydFileself.CommonToolsetImports.name,# Member to insert (in this case, an iterable)[Property(PROP1,VALUE1),Property(PROP2,VALUE2)],# Offset it by 1, so inserts after the found element (default 0)offset=1,# Iterate over the insertion value; otherwise insert it as-isrange=True)Source offsetsIf you keep your source in asrcfolder (recommended), provide thesource=argument toPackagein order to properly offset filenames.
Because it is a named argument, it must be provided last.This is important for sdist generation and in-place builds, which need
to match package layout with source layout. Simply prefixing filename
patterns with the additional directory is not always sufficient.Note that this will also offset subpackages, and that subpackages may
include additionalsourcearguments. However, it only affects
sources, while the package name (the first argument) determines where
in the output the package will be located. In-place builds will create
new folders in your source tree if it does not match the final
structure.PACKAGE=Package("my_package",PyFile(r"my_package\__init__.py"),source="src",)Project file overrideBothPackageandPydFiletypes generate MSBuild project files and
execute them as part of build, including sdists. For highly customised
builds, this generation may be overridden completely by specifying theproject_filenamed argument. All members are then ignored.By doing this, you take full responsibility for a valid build,
including providing a number of undocumented and unsupported targets.Recommendations:lock yourpymsbuilddependency to a specific version inpyproject.tomlgenerate project files first and modify, rather than writing by hand
(pass--temp-dirto specify the path where they will be generated)read thepymsbuildsource code, especially thetargetsfolderconsider contributing/requesting your feature or developing an extension
(seepymsbuild-winuiandpymsbuild-rustfor examples)PACKAGE=Package("my_package",PydFile("_accelerator",project_file=r"src\accelerator.vcxproj"))Compiler/linker argumentsRather than overriding the entire project file, there are a number of
ways to inject arbitrary values into a project. These require
familiarity with MSBuild files and the toolsets you are building with.ThePropertyelement inserts a<PropertyGroup>with the value you
specifiy at the position in the project the element appears.Note that project files also interpret (most) named arguments as
properties, so the two properties shown here are equivalent.PYD=PydFile("module",Property("WindowsSdkVersion","10.0.18363.0"),...# Alternative to Property(), but named arguments must be# specified last, so using Property() may be more readableWindowsSdkVersion="10.0.18363.0",)TheItemDefinitionelement inserts an<ItemDefinitionGroup>with
the type and metadata you specify at the position in the project the
element appears. These generally apply metadata to all subsequent items
of that type.PYD=PydFile("module",ItemDefinition("ClCompile",PreprocessorDefinitions="Py_LIMITED_API"),...)TheConditionalValueitem may wrap any element value to add
conditions or concatenate the value. This may also be used on source
arguments for file elements....Property("Arch",ConditionalValue("x86",condition="$(Platform) == 'Win32'")),Property("Arch",ConditionalValue("x64",if_empty=True)),...ItemDefinition("ClCompile",AdditionalIncludeDirectories=ConditionalValue(INCLUDES+";",prepend=True),ProprocessorDefinitions=ConditionalValue(";Py_LIMITED_API",append=True),),...ThePrependshortcut may be used to reduce the amount of text for
property values. Remember to include the appropriate separator. It is
usually a semicolon....ItemDefinition("ClCompile",AdditionalIncludeDirectories=Prepend(INCLUDES+";"),ProprocessorDefinitions=Prepend("Py_LIMITED_API;"),),...ConditionalValuemay also be used to dynamically update values in theinit_PACKAGEfunction, allowing you to keep the structure mostly
static but insert values from the currentMETADATA(which is fully
evaluated by the timeinit_PACKAGEis called). This saves having to
access internal members of other types in order to replace literal
values.Assign aConditionalValueto a variable without specifying any
condition, then use the variable in a laterPropertyelement and
modify itsvalueattribute ininit_PACKAGE.VER=ConditionalValue("1.0.0")PYD=PydFile("module",Property("Version",VER),CSourceFile(r"src\*.c"),IncludeFile(r"src\*.h"),)definit_PACKAGE(tag):VER.value=METADATA["Version"]As a last resort, theLiteralXmlelement inserts plain text directly
into the generated file. It will be inserted as a child of the
top-levelProjectelement....LiteralXml("<Import Project='my_props.props' />"),...Version info for DLLs/PYDsPlatform: WindowsTo embed version info into a compiled extension module, add aVersionInfoelement into thePydFile. All the fields fromhttps://learn.microsoft.com/en-us/windows/win32/menurc/versioninfo-resourceare available, using the names as shown in the tables (e.g.FILEVERSIONfor the'1,0,0,0'fields andFileVersionfor the string
table entry).The recommended usage is to add a default instance into your project and then
useinit_METADATAto find it again and update based on the final metadata.PACKAGE = Package(
    "package",
    PydFile("mod1", VersionInfo()),
    PydFile("mod2", VersionInfo()),
)

def init_METADATA():
    # Update our metadata
    METADATA["Version"] = calculate_current_version()
    # Ensure built modules reflect these updates
    for vi in PACKAGE.findall("*/VersionInfo"):
        vi.from_metadata(METADATA)from_metadatawill fill in any empty fields from the set of metadata that is
passed in.Alternate config fileTo use a configuration file other than_msbuild.py, specify the--config(-c) argument or thePYMSBUILD_CONFIGenvironment
variable.python-mpymsbuild--configbuild-spec.pysdistpython-mpymsbuild--configbuild-spec.pywheel# Alternatively$env:PYMSBUILD_CONFIG="build-spec.py"python-mpymsbuildsdistwheelGenerated sdists will rename the configuration file back to_msbuild.pyin the package to ensure that builds work correctly.
There is no need to override the configuration file path when
building from sdists.Note that this is different from thePYMSBUILD_CONFIGURATIONvariable, which is used to select debug/release settings for compiled
modules.Cross-compiling wheelsCross compilation may be used by overriding the wheel tag, ABI tag,
or build platform, as well as the source for Python's includes and
libraries. These all use environment variables, to ensure that the
same setting can flow through a package installer's own process.It is also possible to permanently override the wheel tag by adding
a'WheelTag'metadata value, or the ABI tag by adding an'AbiTag'metadata value.The wheel tag is used for the generated wheel file, and to fill in a
missing ABI tag and platform.The ABI tag is used for any native extension modules, and to fill in
a missing platform.Any*elements in the wheel tag are filled in from other locations.
For example, specifying*-none-anywill infer the interpreter field
from the current runtime, whilpy3-none-*will infer the platform
from the currnet system (or a specific ABI tag).The platform is used to determine the MSBuild target platform. It
cannot yet automatically select the correct Python libraries, and so
you will need to setPYTHON_INCLUDESandPYTHON_LIBS(or with aPYMSBULID_prefix) environment variables as well to locate the
correct files.You can override the platform toolset with the'PlatformToolset'metadata value, for scenarios where this information ought to be
included in an sdist.The set of valid platforms for auto-generated.pydproject files are
hard-coded intopymsbuildand are currentlyWin32,x64,ARMandARM64. Custom project files may use whatever they like. These
platforms should behave properly cross-platform, though in general onlyx64andARM64are supported.# Directly specify the resulting wheel tag# This is used for the wheel filename/metadata$env:PYMSBUILD_WHEEL_TAG="py38-cp38-win_arm64"# Directly set the ABI tag (or else taken from wheel tag)# This is used for extension module filenames$env:PYMSBUILD_ABI_TAG="cp38-win_arm64"# Specify the Python platform (or else taken from ABI tag)# This is used for MSBuild options$env:PYMSBUILD_PLATFORM="win_arm64"# Specify the paths to ARM64 headers and libs$env:PYTHON_INCLUDES="$pyarm64\Include"$env:PYTHON_LIBS="$pyarm64\libs"# If necessary, specify an alternate C++ toolset$env:PLATFORMTOOLSET="Intel C++ Compiler 19.1"CythonCython support is available from thepymsbuild.cythonmodule.frompymsbuildimportPydFile,ItemDefinitionfrompymsbuild.cythonimportCythonIncludeFile,CythonPydFile,PyxFilePACKAGE=CythonPydFile("cython_module",ItemDefinition("PyxCompile",IncludeDirs=PYD_INCLUDES),CythonIncludeFile("mod.pxd"),PyxFile("mod.pyx"),)TheCythonPydFiletype derives from the regularPydFileand also
generates a C++ project, so all options that would be available there may
also be used.ThePyxCompile.IncludeDirsmetadata specifies search paths for Cython
headers (*.pxd). You may also need to specifyClCompile.AdditionalIncludeDirectoriesfor any C/C++ headers.Two-Step BuildsBy default, thesdistandwheelcommands will perform the entire
process in a single invocation. However, sometimes there are build steps
that must be manually performed between compilation and packaging.To run the build in two stages, invoke as normal, but add the--layout-dirargument followed by a directory. The package will be
laid out in this directory so that you can perform any extra processing.Later, use thepackcommand and specify the--layout-diragain. If
you have added new files into the layout directory, specify each with an--addoption (filenames starting with@are treated as
newline-separated, UTF-8 encoded text files listing each new file). These
paths may be absolute or relative to the layout directory, but only files
located within the layout directory will be included.All other options are retained from the original invocation.python -m pymsbuild sdist --layout-dir tmp

# Generate additional metadata in tmp/EXTRA.txt

python -m pymsbuild pack --layout-dir tmp --add tmp/EXTRA.txt

# List many additional files in build/TO_ADD.txt

python -m pymsbuild pack --layout-dir tmp --add @build/TO_ADD.txtExperimental FeaturesDLL PackingExperimental. (POSIX isveryexperimental)DLL Packing is a way to compile a complete Python package (.pysource
and resource files) into an extension module. It is basically equivalent
to packing in a ZIP file, except that additional native code may also be
included (though not an entire native module), and the whole file may be
cryptographically signed and validated by the operating system.DllPackageis a drop-in substitute for thePackagetype. It will
generate a native extension module the same as thePydFiletype,
but only includes Python source and resource files.frompymsbuildimport*frompymsbuild.dllpackimport*PACKAGE=DllPackage("packed_package",PyFile("__init__.py"),File("data.txt"),...)An entire existing library, such ascryptographycould be packed
like this:frompymsbuildimport*frompymsbuild.dllpackimport*MODULE_TO_PACK="cryptography"fromimportlib.utilimportfind_specspec=find_spec(MODULE_TO_PACK)ifnotspec:raiseRuntimeError(f"{MODULE_TO_PACK}must be installed")PACKAGE=DllPackage(MODULE_TO_PACK,PyFile("**/*.py"),PydRedirect("**/*.pyd"),source=spec.submodule_search_locations[0],)See theazure-packsample in our source repository for a more
complete example.DllPackageis a subclass ofPydFile, and so all logic or elements
by that type are also available.ClCompileelements will be compiled
and linked into the output and functions may be exposed in the root of
the package using theFunctionelement.// extra.cPyObject*my_func(PyObject*,PyObject*args,PyObject**kwargs){...}PACKAGE=DllPackage("packed_package",PyFile("__init__.py"),CSourceFile("extra.c"),CFunction("my_func"),...)Nested extension modulesTo allow referencing other extension modules that would normally be
nested within the module, add aPydRedirectelement and reference the
extension module. The filename does not have to match the original
name, or even need to be a normally importable name, as it will be
passed directly to the module loader. The file will be included in your
wheel in the expected location (alongside the packed DLL). Wildcards
are supported.PACKAGE=DllPackage("packed",PydRedirect(source="packed/nested.pyd",name="packed-nested.pyd"),...)OtherPydFilemodules may be nested inside theDllPackage, which
will automatically add a redirect, as well as building the module.
The nested module will be built using the name specified and sit
adjacent to the packed module, but should be imported via the
packed module.TheImportNamemetadata may be specified on either aPydRedirector
aPydFileto specify the name that must be used to import the module.
Redirected extension modules do not need to have an importable name
whenImportNameis specified. You might include an invalid character
in the filename to ensure the module is not importable directly.
When specifyingImportName, the name of the packed DLL must be used
as the first part.PACKAGE=DllPackage("packed",PydRedirect("module/nested.pyd",ImportName="packed.nested"),...)EncryptionTo encrypt your content using symmetric AES encryption, provide the
name of the environment variable holding your key as theEncryptionKeyVariableoption. The key will need to be a valid size
(usually 16, 24 or 32 bytes) when encoded to UTF-8 or decoded from
base 64. Base 64 keys should start withbase64:.The same variable will need to be set when importing the module. It is
your responsibility to protect the key! The benefit of this encryption
is best realised when you avoid storing the key to disk. That way, an
attacker who steals a copy of your module is unlikely to have access to
the key. An attacker with access to a running copy of your module will
be able to easily extract the key.PACKAGE=DllPackage("package",...,EncryptionKeyVariable="MY_KEY_VARIABLE")>$env:MY_KEY_VARIABLE="base64:MDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDAwMDA=">python-mpymsbuild>delenv:\MY_KEY_VARIABLE>python-c"import package"ImportError:ModulecannotbedecryptedRedirected or nested extension modules are not encrypted.Cross-platform buildsExperimental.With the.NET SDKinstalled,pymsbuildis able to run builds on platforms other than Windows.
Thedotnetcommand must be available onPATHor specified as theMSBUILDenvironment variable.In general, no platform-specific modifications to a build script are
required. Cython and pyd builds are transparently mapped to the target
system. To run build-time actions for specific platforms, add them toinit_PACKAGEand check the tag argument to determine the target
platform.When building native components on POSIX, apython3-configscript is
needed to determine compilation options. By default, only the location
adjacent to the running interpreter is checked. This may be overridden
by setting thePYTHON_CONFIGvariable to the preferred command.