Package: scandir
scandir()is a directory iteration function likeos.listdir(),
except that instead of returning a list of bare filenames, it yieldsDirEntryobjects that include file type and stat information along
with the name. Usingscandir()increases the speed ofos.walk()by 2-20 times (depending on the platform and file system) by avoiding
unnecessary calls toos.stat()in most cases.Now included in a Python near you!scandirhas been included in the Python 3.5 standard library asos.scandir(), and the related performance improvements toos.walk()have also been included. So if you’re lucky enough to be
using Python 3.5 (release date September 13, 2015) you get the benefit
immediately, otherwise justdownload this module from PyPI,
install it withpip install scandir, and then do something like
this in your code:# Use the built-in version of scandir/walk if possible, otherwise# use the scandir module versiontry:fromosimportscandir,walkexceptImportError:fromscandirimportscandir,walkPEP 471, which is the
PEP that proposes includingscandirin the Python standard library,
wasacceptedin July 2014 by Victor Stinner, the BDFL-delegate for the PEP.Thisscandirmodule is intended to work on Python 2.7+ and Python
3.4+ (and it has been tested on those versions).BackgroundPython’s built-inos.walk()is significantly slower than it needs to be,
because – in addition to callinglistdir()on each directory – it callsstat()on each file to determine whether the filename is a directory or not.
But bothFindFirstFile/FindNextFileon Windows andreaddiron Linux/OS
X already tell you whether the files returned are directories or not, so
no furtherstatsystem calls are needed. In short, you can reduce the number
of system calls from about 2N to N, where N is the total number of files and
directories in the tree.In practice, removing all those extra system calls makesos.walk()about7-50 times as fast on Windows, and about 3-10 times as fast on Linux and Mac OS
X.So we’re not talking about micro-optimizations. See more benchmarks
in the “Benchmarks” section below.Somewhat relatedly, many people have also asked for a version ofos.listdir()that yields filenames as it iterates instead of returning them
as one big list. This improves memory efficiency for iterating very large
directories.So as well as a fasterwalk(), scandir adds a newscandir()function.
They’re pretty easy to use, but see “The API” below for the full docs.BenchmarksBelow are results showing how many times as fastscandir.walk()is thanos.walk()on various systems, found by runningbenchmark.pywith no
arguments:System versionPython versionTimes as fastWindows 7 64-bit2.7.7 64-bit10.4Windows 7 64-bit SSD2.7.7 64-bit10.3Windows 7 64-bit NFS2.7.6 64-bit36.8Windows 7 64-bit SSD3.4.1 64-bit9.9Windows 7 64-bit SSD3.5.0 64-bit9.5Ubuntu 14.04 64-bit2.7.6 64-bit5.8Mac OS X 10.9.32.7.5 64-bit3.8All of the above tests were done using the fast C version of scandir
(source code in_scandir.c).Note that the gains are less than the above on smaller directories and greater
on larger directories. This is whybenchmark.pycreates a test directory
tree with a standardized size.The APIwalk()The API forscandir.walk()is exactly the same asos.walk(), so justread the Python docs.scandir()The full docs forscandir()and theDirEntryobjects it yields are
available in thePython documentation here.
But below is a brief summary as well.scandir(path=’.’) -> iterator of DirEntry objects for given pathLikelistdir,scandircalls the operating system’s directory
iteration system calls to get the names of the files in the givenpath, but it’s different fromlistdirin two ways:Instead of returning bare filename strings, it returns lightweightDirEntryobjects that hold the filename string and provide
simple methods that allow access to the additional data the
operating system may have returned.It returns a generator instead of a list, so thatscandiracts
as a true iterator instead of returning the full list immediately.scandir()yields aDirEntryobject for each file and
sub-directory inpath. Just likelistdir, the'.'and'..'pseudo-directories are skipped, and the entries are
yielded in system-dependent order. EachDirEntryobject has the
following attributes and methods:name: the entry’s filename, relative to the scandirpathargument (corresponds to the return values ofos.listdir)path: the entry’s full path name (not necessarily an absolute
path) – the equivalent ofos.path.join(scandir_path, entry.name)is_dir(*,follow_symlinks=True): similar topathlib.Path.is_dir(), but the return value is cached on theDirEntryobject; doesn’t require a system call in most cases;
don’t follow symbolic links iffollow_symlinksis Falseis_file(*,follow_symlinks=True): similar topathlib.Path.is_file(), but the return value is cached on theDirEntryobject; doesn’t require a system call in most cases;
don’t follow symbolic links iffollow_symlinksis Falseis_symlink(): similar topathlib.Path.is_symlink(), but the
return value is cached on theDirEntryobject; doesn’t require a
system call in most casesstat(*,follow_symlinks=True): likeos.stat(), but the
return value is cached on theDirEntryobject; does not require a
system call on Windows (except for symlinks); don’t follow symbolic links
(likeos.lstat()) iffollow_symlinksis Falseinode(): return the inode number of the entry; the return value
is cached on theDirEntryobjectHere’s a very simple example ofscandir()showing use of theDirEntry.nameattribute and theDirEntry.is_dir()method:defsubdirs(path):"""Yield directory names not starting with '.' under given path."""forentryinos.scandir(path):ifnotentry.name.startswith('.')andentry.is_dir():yieldentry.nameThissubdirs()function will be significantly faster with scandir
thanos.listdir()andos.path.isdir()on both Windows and POSIX
systems, especially on medium-sized or large directories.Further readingThe Python docs for scandirPEP 471, the
(now-accepted) Python Enhancement Proposal that proposed addingscandirto the standard library – a lot of details here,
including rejected ideas and previous discussionFlames, comments, bug reportsPlease send flames, comments, and questions about scandir to Ben Hoyt:http://benhoyt.com/File bug reports for the version in the Python 3.5 standard libraryhere, or file bug reports
or feature requests for this module at the GitHub project page:https://github.com/benhoyt/scandir