Package: rustimport
rustimport - Import Rust directly from Python!rustimport was heavily inspired by and is partly based uponcppimport. Check it out if you're interested in the same functionality for C and C++!InstallationInstall withpip install rustimport.A quick exampleSave the Rust code below assomecode.rs.// rustimport:pyo3usepyo3::prelude::*;#[pyfunction]fnsquare(x:i32)->i32{x*x}Then open a Python interpreter and import the Rust extension:>>>importrustimport.import_hook>>>importsomecode# This will pause for a moment to compile the module>>>somecode.square(9)81Hurray, you've called some Rust code from Python using a combination ofrustimportandpyo3This workflow enables you to edit both Rust files and Python and recompilation happens automatically and transparently! It's also handy for quickly whipping together an optimized version of a slow Python function.To easily create a new single-file extension (like above), or a complete crate, use the provided tool:$python3-mrustimportnewmy_single_file_extension.rs# or create a full rust crate:$python3-mrustimportnewmy_crateAnd import it from Python:>>>importrustimport.import_hook>>>importmy_single_file_extension,my_crate>>>my_single_file_extension.say_hello()Hellofrommy_single_file_extension,implementedinRust!>>>my_crate.say_hello()Hellofrommy_crate,implementedinRust!Smooth!An explanationOkay, now that I've hopefully convinced you on how exciting this is, let's get into the details of how to do this yourself. First, the comment at top is essential to opt in to rustimport. Don't forget this! (See below for an explanation of why this is necessary.)// rustimport:pyo3The bulk of the file is a generic, simplepyo3extension. We use thepyo3crate, then define a simple function that squaresx, and rustimport takes care of exporting that function as part of a Python extension calledsomecode.Templating & Pre-Processingrustimport offers several layers of customization. This is archieved through a simple pre-processor and templates (well, the only existing template at the moment ispyo3- pull requests welcome :D).What rustimport did for you in the backgroundThe first example in this Readme is the simplest possible form of using rustimport. You just tell rustimport to use thepyo3template by writingrustimport:pyo3in the first line, and define a function annotated withpyo3's#[pyfunction]macro. In the background, rustimport handled a lot of stuff for you:It set up a minimalistic folder structure for a rust crate with your source code in a temporary location.It generated a Cargo.toml file with the basic configuration for pyo3 and your extension:[package]name="somecode"version="0.1.0"edition="2021"[lib]name="somecode"crate-type=["cdylib",][dependencies.pyo3]version="0.16.2"features=["extension-module",]It generated a code block exporting your method and appended it to the end of your file:#[pymodule]fnsomecode(_py:Python,m:&PyModule)->PyResult<()>{m.add_function(wrap_pyfunction!(square,m)?)?;Ok(())}Customizing an extensionYou can do all the above yourself. rustimport will detect that and only fill in the missing parts to make your extension work.1. ExtendingCargo.tomlFor example, to add additional contents to the generatedCargo.tomlfile, use the special//:comment syntax at the top of your.rsfile:// rustimport:pyo3// Set the library's version and add a dependency://: [package]//: version = "1.2.3"//://: [dependencies]//: rand = "0.8.5"userand::Rng;#[pyfunction]fnmyfunc(){println!("{}",rand::thread_rng().gen_range(0..10))}2.Tracking additional source filesTo track additional files for changes, use the special//d:comment syntax://d: ../other-folder/somefile.rs//d: ../*.rs//d: ./my-directory/**/*.json// --snip--rustimport will now track files matching these patterns too and re-compiles your extension if any of them changes.3. Full customization for more controlIf you write a more complex extension, it's preferrable to just create a normal Rust crate:$python3-mrustimportnewmy_crate
$treemy_crate
my_crate
├──Cargo.toml
├──.rustimport
└──src└──lib.rsThe crate contains all necessary configuration to be directly be imported by rustimport and also some additional explanations on how to configure manually.Using environment variablesrustimport exposed most of it's settings via environment variables. This can be pretty handy in development but also in production environments.For example, to force recompilation, just run your script like this:RUSTIMPORT_FORCE_REBUILD=true python my_script.pyOr to instruct the compiler to optimize binaries (for example to examine the real performance boost rust gives you) run it like this:RUSTIMPORT_RELEASE_BINARIES=true python my_script.pyTake a look atsettings.pyfor all available environment variables.Using in Jupyterrustimportsupports compiling code in a Jupyter notebook. To enable this feature load therustimportextension from within the Jupyter notebook:%load_extrustimportThen, prefix a cell with the%%rustimportmarker to compile it:%%rustimportusepyo3::prelude::*;#[pyfunction]fnsquare(x:i32)->i32{x*x}%%rustimportsupports release mode and force compilation with flags:%%rustimport--release--force...Usage in production1. Building release binariesIn production deployments you usually don't want to include the Rust toolchain, all the sources and compile at runtime. Therefore, a simple cli utility for pre-compiling all source files is provided. This utility may, for example, be used in CI/CD pipelines.Usage is as simple aspython -m rustimport build --releaseThis will build a release-optimized binary for all*.rsfiles and Rust crates in the current directory (and it's subdirectories) if they are eligible to be imported (i.e. contain the// rustimportcomment in the first line or a.rustimportfile in case of a crate).Alternatively, you may specifiy one or more root directories or source files to be built:python -m rustimport build --release ./my/root/folder/ ./my/single/file.rs ./my/crate/Note: When specifying a path to a file, the header check (// rustimport) is skipped for that file.2. Toggling release mode onTo further improve startup performance for production builds, you can opt-in to skip the checksum and compiled binary existence checks during importing by either setting the environment variableRUSTIMPORT_RELEASE_MODEtotrueor setting the configuration from within Python:rustimport.settings.release_mode=TrueThis essentially just disables the import hook and uses the standard python utilities to import the pre-compiled binary.Warning:Make sure to have all binaries pre-compiled with when in release mode, as importing any missing ones will cause exceptions.In case you would, for whatever reason, like the binaries to be checked and built in production too, setrustimport.settings.compile_release_binariestoTrueto use release-optimized binaries.Frequently asked questionsWhat's actually going on?Sometimes Python just isn't fast enough. Or you have existing code in a Rust crate. So, you write a Pythonextension module, a library of compiled code. I recommendpyo3for Rust to Python bindings.I discovered that my productivity is slower when my development process goes fromEdit -> Testin just Python toEdit -> Compile -> Testin Python plus Rust. So,rustimportcombines the process of compiling and importing an extension in Python so that you can just runimport foobarand not have to worry about multiple steps. Internally,rustimportlooks for a filefoobar.rsor a Rust crate (discovered throughfoobar/Cargo.toml). Assuming one is found, the comments at it's beginning are parsed for either a template (rustimport:pyo3) or a cargo manifest, then it's compiled and loaded as an extension module.Does rustimport recompile every time a module is imported?No! Compilation should only happen the first time the module is imported. The Rust source is compared with a checksum on each import to determine if any relevant file has changed. Additional dependencies can be tracked by adding to the header comments://d: ../myothercrate/**/*.rs//d: ../myothercrate/Cargo.tomlBy default, rustimport tracks all*.rsfiles as well asCargo.tomlandCargo.lockfor crates and no additional dependencies for single-file Rust extensions.rustimport isn't doing what I want, can I get more verbose output?rustimportuses the standard Python logging tools. Thus, you can enable logging like this:importlogginglogging.basicConfig(level=logging.DEBUG)# or logging.INFO for a bit less verbosity# ... do some rustimport stuff hereIt's fast, but can it get even faster?To create release-optimized binaries, setrustimport.settings.compile_release_binaries=TrueOr set the environment variableRUSTIMPORT_RELEASE_BINARIEStotrueCompilation might be a little bit slower now due to rust's optimization mechanisms, but at runtime the extension is significantly faster in most cases.How can I force a rebuild even when the checksum matches?Set:rustimport.settings.force_rebuild=TrueOr set the environment variableRUSTIMPORT_FORCE_REBUILDtotrueAnd if this is a common occurrence, I would love to hear your use case and why the normal dependency tracking is insufficient!Can I use something else thanpyo3?Sure! Though I recommend usingpyo3due to it's simplicity, you're completely free to use any other library, for examplerust-cpython.There is an example usingrust-cpythoninexamples/doublecount.rsHow can I make compilation faster?Compilation happens incrementally by default. That is, the first compilation might take a bit, but subsequent ones are usually much faster.rustimport uses a temporary directory for caching, which is deleted after a reboot on most systems. Therefore it might be beneficial for you to set a custom cache directory to have a more permanent cache:rustimport.settings.cache_dir=os.path.realpath("./.rust-cache")Or - you guessed it - use theRUSTIMPORT_CACHE_DIRenvironment variable.If this directory doesn't exist, it will be created automatically by rustimport.Apple, huh?Yes, macOS is supported. No additional config should be necessary for pyo3 as therequired linker argsare set automatically by rustimport.Why does the import hook need "rustimport" on the first line of the .rs file?Modifying the Python import system is a global modification and thus affects all imports from any other package. As a result, whencppimportwas first implemented, other packages (e.g.scipy) suddenly started breaking because import statements internal to those packages were importing C or C++ files instead of the modules they were intended to import. To avoid this failure mode, the import hook uses an "opt in" system where C and C++ files can specify they are meant to be used with cppimport by having a comment on the first line that includes the text "cppimport".rustimport has adopted from this and follows the same pattern. Since rustimport also supports importing whole crates, an additional mechanism was necessary to make that work in the same fashion: You can either create a.rustimportfile in the crate's root folder (next toCargo.toml) or, alternatively, add a# rustimportcomment toCargo.tomls first line.As an alternative to the import hook, you can useimporimp_from_path. Therustimport.impandrustimport.imp_from_pathperforms exactly the same operation as the import hook but in a slightly more explicit way:foobar=rustimport.imp("foobar")foobar=rustimport.imp_from_path("./some/path/foobar.rs")mycrate=rustimport.imp_from_path("./mycrate/")By default, these explicit function do not require the "rustimport" keyword on the first line of the .rs source file or the according marker in the crate.Contributing and architectureSeeCONTRIBUTING.mdfor details on the internals ofrustimportand how to get involved in development.Licenserustimport uses the MIT License.