Package: rust-dll-example
How to compile DLL in Rust and import it in Python. Useful hints.Here I am providing a number of examples to compileDLLin Rust. In order to check the final DLL I am using Python, although DLL standard is compatible with almost all the widely known programming languages. The examples include plain functions, input and output arrays, structures and OOP-like approach. I also attached a benchmark. In the end, I am describing a way to create a Python package powered by Rust, that is ready to distribute onPyPI. Source code of the project is available onhttps://github.com/fomalhaut88/rust-dll-example.How to create a DLL project in RustCreate a project as a library (using the flag--lib):cargo new rust-dll-example --libAdd the following section inCargo.tomlto specify the type of the library:[lib]
crate-type = ["cdylib"]Add#[no_mangle]andexternto the exporting functions like this:#[no_mangle]pubexternfnadd(left:usize,right:usize)->usize{left+right}Build the project:cargo build --releaseAfter that the DLL file will appear by path./target/release/rust_dll_example.dllif you are using Windows or something similar on different platforms (usually.sofor Linux and.dylibfor Mac OS X).It is a good practice to cover your functions with standard Rust tests and benchmarks, so you can control the correctness and the performance.Plain functionsHere are two plain functions written in Rust://! An addition of two unsigned integer numbers.#[no_mangle]pubexternfnadd(left:usize,right:usize)->usize{left+right}//! Square of a float value.#[no_mangle]pubexternfnsqr(x:f64)->f64{x*x}There are two differences to normal functions: the keywordexternso the function is linked to DLL interface and the attribuueno_manglethat is needed to disable the standard name encoding on the compilation stage (read more about it here:https://doc.rust-lang.org/reference/abi.html#the-no_mangle-attribute).To call this functions in Python, there is a following code:importctypesdll=ctypes.CDLL("./target/release/rust_dll_example.dll")# Test addassertdll.add(2,3)==5# Test sqrdll.sqr.restype=ctypes.c_doubledll.sqr.argtypes=[ctypes.c_double]assertdll.sqr(6.0)==36.0Notice, that forsqrfunction we had to specify input and output types explicitly before the call, so Python knows how to interpret the data. It is recommended to avoid skipping types specification as it is done foraddfunction, despite of usually integer data types are set by default. Do you remember the phraseExplicit is better than implicit.fromThe Zen of Python? (The full text of it can always be outputed byimport this)Passing arraysThere are two ways to handle arrays: by reference (norammly for fixed size arrays) and by pointer (it the size is unknown or mutable from call to call), and both a supported in DLLs.//! Sum of elements of an array given by pointer and the size.#[no_mangle]pubexternfnarray_sum(size:usize,arr:*constf64)->f64{letmutres=0.0;foridxin0..size{unsafe{res+=*arr.offset(idxasisize);}}res}//! Fill the given array with a float value.#[no_mangle]pubexternfnarray_set(size:usize,arr:*mutf64,val:f64){foridxin0..size{unsafe{*arr.offset(idxasisize)=val;}}}//! Set elements of a fixed size array to zeros.#[no_mangle]pubexternfnarray3_zero(arr:&mut[f64;3]){foridxin0..arr.len(){arr[idx]=0.0;}}//! Concatenate two arrays.#[no_mangle]pubexternfnarray_concat(size1:usize,arr1:*constf64,size2:usize,arr2:*constf64)->*constf64{letmutres=Vec::with_capacity(size1+size2);res.resize(size1+size2,0.0);unsafe{ptr::copy(arr1,res.as_mut_ptr(),size1);ptr::copy(arr2,res.as_mut_ptr().add(size1),size2);}Box::new(res).as_ptr()}//! Return fixed size array filled with the given value.#[no_mangle]pubexternfnarray5_fill(val:f64)->Box<[f64;5]>{Box::new([val;5])}Accessing to the elements of an array passed by pointer is an unsafe operation for Rust, so it is reflected by theunsafesections. If we return the array as the result, we can do it by reference (as it is inarray_concat) or as a boxed fixed size array.In order by call these functions from Python, we can use following commands:# Create arrayarr_type=(ctypes.c_double*5)arr=arr_type(*[1.0,2.0,3.0,4.0,5.0])# Test array_sumdll.array_sum.restype=ctypes.c_doubledll.sqr.argtypes=[ctypes.c_uint64,arr_type]assertdll.array_sum(5,arr)==15.0# Test array_setdll.array_set.argtypes=[ctypes.c_uint64,arr_type,ctypes.c_double]dll.array_set(5,arr,ctypes.c_double(3.0))assertlist(arr)==[3.0]*5# Test array3_zerodll.array3_zero(arr)assertlist(arr)==[0.0,0.0,0.0,3.0,3.0]# Test array_concatdll.array_concat.argtypes=[ctypes.c_uint64,ctypes.c_double*2,ctypes.c_uint64,ctypes.c_double*3]dll.array_concat.restype=ctypes.POINTER(ctypes.c_double*5)arr1=(ctypes.c_double*2)(*[1.0,2.0])arr2=(ctypes.c_double*3)(*[3.0,4.0,5.0])res=dll.array_concat(2,arr1,3,arr2)assertlist(res.contents)==[1.0,2.0,3.0,4.0,5.0]# Test array5_filldll.array5_fill.argtypes=[ctypes.c_double]dll.array5_fill.restype=ctypes.POINTER(ctypes.c_double*5)arr=dll.array5_fill(2.5)assertlist(arr.contents)==[2.5]*5What is interesting there? First of all, we define array data type as a product of the type of element and the size. And after that we must convert Python list to the understandable by Rust format, because Python lists are totally not the same as C-compatible arrays. Also there is no difference from Python side regarding passing by reference or by pointer.In two last functions we return arrays as the results: as a pointer and as a fixed size array. In both cases, in Python we should setrestypetoctypes.POINTER, so after that we can extract the values from.contentsattribute.Working with strings (or bytes) is similar, because string is represented as an array of chars (oruint8), so on the Rust side they should have the types*u8or[u8; SIZE], and on the Python side it isctypes.c_char * SIZE.C-compatible structuresWe also are allowed to work with complex data types like structures as with base types, but there are some additions. Let us look at the Rust code that implements a few functions for complex numbers:#[derive(Debug, PartialEq)]#[repr(C)]pubstructComplex{pubx:f64,puby:f64,}#[no_mangle]pubexternfncomplex_len(z:Complex)->f64{(z.x*z.x+z.y*z.y).sqrt()}#[no_mangle]pubexternfncomplex_conj(z:Complex)->Complex{Complex{x:z.x,y:-z.y,}}implComplex{#[no_mangle]#[export_name="complex_real"]pubexternfnreal(&self)->f64{self.x}#[no_mangle]#[export_name="complex_image"]pubexternfnimage(&self)->f64{self.y}#[no_mangle]#[export_name="complex_mul"]pubexternfnmul(&mutself,val:f64){self.x*=val;self.y*=val;}}We can see already familiar functions withexternandno_manglekeywords and passing variables is done as it would be for standard data types. I also added an OOP-like part (implsection), so the functions inside can be considered as methods of the structure. They also haveexternandno_mandgebut there is alsoexport_nameattribute that customizes the name of the linked function to reach it outside.Notice, that the structure has the attributerepr(C)that is important, otherwise external programs barely can understand the way the data is stored in a structure instance. This is because there are several standards to manage inner data using different alignment algorithms (you can read more about it inData structure alignment).As for Python code:# Complex structclassComplex(ctypes.Structure):_fields_=[('x',ctypes.c_double),('y',ctypes.c_double),]def__repr__(self):returnf"Complex(x={self.x}, y={self.y})"def__eq__(self,other):returnself.x==other.xandself.y==other.yz=Complex(x=3.0,y=-4.0)# Test complex_lendll.complex_len.argtypes=[Complex]dll.complex_len.restype=ctypes.c_doubleassertdll.complex_len(z)==5.0# Test complex_conjdll.complex_conj.argtypes=[Complex]dll.complex_conj.restype=Complexassertdll.complex_conj(z)==Complex(x=3.0,y=4.0)# Test realdll.complex_real.restype=ctypes.c_doubleassertdll.complex_real(z)==3.0# Test imagedll.complex_image.restype=ctypes.c_doubleassertdll.complex_image(z)==-4.0# Test muldll.complex_mul.argtypes=[ctypes.c_void_p,ctypes.c_double]dll.complex_mul(ctypes.byref(z),2.0)assertz==Complex(x=6.0,y=-8.0)Fortunately,ctypessupports the opportunity to define structure data type in a very friendly way. So after that we can work withComplexas with an ordinary data type, passing it toargtypesandrestypeattributes. But there is a peculiarity if the structure instance is supposed to be mutable: in this case we have to addctypes.byrefas it is done incomplex_mulfunction. The argument type is set toctypes.c_void_pinstead ofComplex.This approach is good but it does not cover all the needs. The main lack is that only C-compatible data types are allowed in structure fields. If we are not going to share inner data of the structure outside there is a different approach shown in the next section.OOP exampleIn spite of the previous example, here we can use any data types in our structure we want. But we cannot access the fields outside, though we usually do not need it if we require OOP that supposes encapsulation.structCounter{val:usize,}implCounter{#[no_mangle]#[export_name="counter_new"]pubexternfnnew()->Box<Self>{Box::new(Self{val:0,})}#[no_mangle]#[export_name="counter_get"]pubexternfnget(&self)->usize{self.val}#[no_mangle]#[export_name="counter_increment"]pubexternfnincrement(&mutself){self.val+=1;}}There is norepr(C)attribute andnewreturns boxed instance because we need to allocate the instance in the heap. The other methods are implemented similar to what we had forComplexstructure.dll.counter_new.restype=ctypes.c_void_pdll.counter_get.argtypes=[ctypes.c_void_p]dll.counter_increment.argtypes=[ctypes.c_void_p]classCounter:_dll=dlldef__init__(self):self._counter=self._dll.counter_new()defget(self):returnself._dll.counter_get(self._counter)defincrement(self):self._dll.counter_increment(self._counter)# Create an instancecounter=Counter()# Get valueassertcounter.get()==0# Increment valuecounter.increment()# Get valueassertcounter.get()==1On the Python level, we can defineCounteras a class with the same methods, a class object keeps the Rust counter instance inside. Notice, thatself._counteris a pointer that has the typectypes.c_void_p.Benchmark exampleUsually, developers combine Python and a low-level programming language to improve the performance. This is the most frequent reason why DLL is needed. So this article will not be complete if I do not include a real world case with a benchmark. For this purpose I implemented a sort ofLevenshtein distancealgorithm. I did not much care about the performance on Rust level, so it may work slower than the solutions you can find in the net, although the performance will be surely comparable.usestd::{cmp,slice};//! Levenshtain distance algorithm#[no_mangle]#[export_name="levenshtein_distance"]pubexternfndistance(n1:usize,p1:*constu8,n2:usize,p2:*constu8)->usize{lets1=unsafe{slice::from_raw_parts(p1,n1)};lets2=unsafe{slice::from_raw_parts(p2,n2)};letlcs=lcs_solve(s1,s2);cmp::max(n1,n2)-lcs}//! Calculate Longest Common Subsequence lengthpubfnlcs_solve<T:PartialEq>(s1:&[T],s2:&[T])->usize{letn1=s1.len();letn2=s2.len();letmutrow=vec![0usize;n1];fori2in0..n2{letmutprev=0;fori1in0..n1{letprev_new=row[i1];ifs1[i1]==s2[i2]{row[i1]=prev+1;}elseifi1>0{ifrow[i1]<row[i1-1]{row[i1]=row[i1-1];}}prev=prev_new;}}row[n1-1]}Also I attached Rust benchmark:#[cfg(test)]modtests{usesuper::*;usetest::Bencher;#[bench]fnbench_distance(bencher:&mutBencher){lets1="lewenstein";lets2="levenshtein";bencher.iter(||{distance(s1.len(),s1.as_ptr(),s2.len(),s2.as_ptr());});}}Before run the benchmark do not forget to switch to theNightly toolchain:rustup default nightlyOn Python level we have following script (filerust_dll_example/__init__.py):importctypes_dll=ctypes.CDLL("./target/release/rust_dll_example.dll")_dll.levenshtein_distance.restype=ctypes.c_uint64_dll.levenshtein_distance.argtypes=[ctypes.c_uint64,ctypes.c_char_p,ctypes.c_uint64,ctypes.c_char_p]deflevenshtein_distance(s1,s2):b1=s1.encode()b2=s2.encode()return_dll.levenshtein_distance(len(b1),b1,len(b2),b2)If you want to benchmark the result on Python level, you can always do it with the help oftimeitwith the command:python -m timeit -s 'from rust_dll_example import levenshtein_distance' 'levenshtein_distance(\"lewenstein\", \"levenshtein\")'How to prepare a Python packageIf we are going to distribute the Python library with compiled DLL inside, first, it is necessary to preparesetup.pyhaving some non-standard lines. Here is an example:importosimportjsonimportsubprocessasspfromsetuptoolsimportfind_packages,setupdefbuild_src():sp.Popen(["cargo","build","--release"]).communicate()defget_version():ifos.path.exists('version'):withopen('version')asf:returnf.read()else:out,_=sp.Popen(["cargo","metadata"],stdout=sp.PIPE).communicate()metadata=json.loads(out.decode())version=metadata['packages'][0]['version']withopen('version','w')asf:f.write(version)returnversiondefget_long_description():withopen('README.md')asf:returnf.read()defget_dll_paths():return['./target/release/rust_dll_example.dll',]# Build from sourcebuild_src()# Setupsetup(name='rust-dll-example',version=get_version(),packages=find_packages(),license="MIT",description="",long_description=get_long_description(),long_description_content_type="text/markdown",install_requires=[],data_files=[('dlls',get_dll_paths()),('',['version'])],)Here we specified the path to DLL file as a data file for our package. Obviously, the old path./target/release/rust_dll_example.dllwill not work. So we need to make some changes in__init__.pyto import DLL correctly:..._dll_path=os.path.join(sys.prefix,'dlls','rust_dll_example.dll')_dll=ctypes.CDLL(_dll_path)...Oncesetup.pyis created and the path to DLL is corrected, we can build our Python package:python setup.py sdistAfter that the filedist/rust-dll-example-0.1.0.tar.gzwill appear. It can be installed withpipand distributed onPyPI. In order to upload it on PyPI, run the following command:twine upload dist/rust-dll-example-0.1.1.tar.gzAfter that the project will be available onhttps://pypi.org/project/rust-dll-example/.Notice! As far as DLL is compiled under a single platform (Windows 11 in my case), the library can be successfully installed only for the users with the same platform. Otherwise, the attached DLL file cannot be executed correctly. If you would like your library to be crossplatform, you need to build several DLLs for each platform and implement a more tricky way to catch the right DLL on Python level.