Package: scandir-rs
scandir-rsscandir_rsis a directory iteration module likeos.walk(), but with more features and higher speed. Depending on the function call
it yields a list of paths, tuple of lists grouped by their entry type orDirEntryobjects that include file type and stat information along
with the name. Usingscandir_rsis about2-17 times fasterthanos.walk()(depending on the platform, file system and file tree structure)
by parallelizing the iteration in background.If you are just interested in directory statistics you can use theCount.scandir_rscontains following classes:Countfor determining statistics of a directory.Walkfor getting names of directory entries.Scandirfor getting detailed stats of directory entries.For the API see:ClassCountdoc/count.mdClassWalkdoc/walk.mdClassScandirdoc/scandir.mdInstallationFor building this wheel from source you need the toolmaturin.Installmaturin:cargoinstallmaturinIMPORTANT: In order to build this project at least Rust version 1.61 is needed!Build wheel (not on Windows):maturinbuild--release--stripBuild wheel on Windows:maturinbuild--release--strip--no-sdistmaturinwill build the wheels for all Python versions installed on your system.Building and running tests for different Python versionsTo make it easier to build wheels for several different Python versions the scriptbuild_wheels.shhas been added.
It creates wheels for Python versions 3.7, 3.8, 3.9, 3.10 and 3.11. In addition it runspytestafter successfull creation of each wheel.Instruction how to installpyenvcan be foundhere.ExamplesGet statistics of a directory:fromscandir_rsimportCount,ReturnTypeprint(Count("/usr",return_type=ReturnType.Ext).collect())Thecollectmethod releases the GIL. So other Python threads can run in parallel.The same, but asynchronously in background using a class instance:fromscandir_rsimportCount,ReturnTypeinstance=Count("/usr",return_type=ReturnType.Ext))instance.start())# Start scanning the directory...values=instance.results()# Returns the current statistics. Can be read at any time...ifinstance.busy():# Check if the task is still running....instance.stop()# If you want to cancel the task...instance.join()# Wait for the instance to finish.and with a context manager:importtimefromscandir_rsimportCount,ReturnTypewithCount("/usr",return_type=ReturnType.Ext)asinstance:whileinstance.busy():statistics=instance.results()# Do somethingtime.sleep(0.01)print(instance.results())os.walk()example:fromscandir_rsimportWalkforroot,dirs,filesinWalk("/usr"):# Do somethingwith extended data:fromscandir_rsimportWalk,ReturnTypeforroot,dirs,files,symlinks,other,errorsinWalk("/usr",return_type=ReturnType.Ext):# Do somethingos.scandir()example:fromscandir_rsimportScandir,ReturnTypeforpath,entryinScandir("~/workspace",return_type=ReturnType.Ext):# entry is a custom DirEntry objectBenchmarksSeeexamples/benchmark.pyIn the below table the lineWalk.iterreturns comparable
results to os.walk.Linux with Ryzen 5 2400G and SSDDirectory/usrwith110171 directories862634 files47804 symlinks12275 hardlinks12 devices0 pipes32.7GB size and 34.8GB usage on diskTime [s]Method3.450os.walk (Python 3.10)6.021scantree (Python 3.10)1.186Count.collect1.416Count(ReturnType=Ext).collect1.089Walk.iter1.350Walk.collect1.336Walk(ReturnType=Ext).collect2.232Scandir.collect1.839Scandir.iter2.437Scandir(ReturnType=Ext).collectAround~3 times fasteron Linux (os.walk compared to Walk.iter).Windows 10 with Laptop Core i7-4810MQ @ 2.8GHz Laptop, MTF SSDDirectoryC:\Windowswith132604 directories349911 files44.4GB size and 45.2GB usage on diskTime [s]Method21.779os.walk (Python 3.10)13.085scantree (Python 3.10)3.257Count.collect16.605Count(ReturnType=Ext).collect4.102Walk.iter4.056Walk.collect4.190Walk(ReturnType=Ext).collect3.993Scandir.collect8.921Scandir.iter17.616Scandir(ReturnType=Ext).collectAround~5.3 times fasteron Windows 10 (os.walk compared to Walk.iter).Directorylinux-5.9with4711 directories69973 files1.08GB size and 1.23GB usage on diskTime [s]Method0.411os.walk (Python 3.10)1.203os.walk (stat)0.218scandir.Count()0.278scandir.Count(return_type=ReturnType.Ext).collect()0.227scandir_rs.Walk().collect()0.164scandir.Walk(return_type=scandir.ReturnType.Ext) (iter)0.204scandir.Walk(return_type=scandir.ReturnType.Ext) (collect)0.350scandir.Scandir(return_type=ReturnType.Base).collect()0.426scandir.Scandir(return_type=ReturnType.Ext).collect()Around~2.5 times fasteron Linux (os.walk compared to Walk.iter).Time [s]Method1.998os.walk (Python 3.10)14.875os.walk (stat)0.278scandir.Count()2.114scandir.Count(return_type=ReturnType.Ext).collect()0.464scandir_rs.Walk().collect()0.313scandir.Walk(return_type=scandir.ReturnType.Ext) (iter)0.455scandir.Walk(return_type=scandir.ReturnType.Ext) (collect)0.624scandir.Scandir(return_type=ReturnType.Base).collect()2.409scandir.Scandir(return_type=ReturnType.Ext).collect()Around~6.4 times fasteron Windows 10 (os.walk compared to Walk.iter).