Package: tarwalker
SummaryTarWalkerprovides a method to easily scan files somewhat likeos.walk,
handling compressed files, recursing through directories and scanning
withintarfiles.The library is very stable, changes are rare. It well documented and
has full unit testing (100% code coverage), and is maintained.Build StatusOverviewNotes about this library:It can walk through compressed or uncompressed files andtarfiles(and optionally
directories), processing the target files one at a time.It uses a pair of callbacks to avoid opening or decompressing files
that do are not of interest:Thematcheris called first with meta data of the file, and
returns true if the file is to be used.If so, thehandleris called with the meta data, thematcherreturn value, and an open (possibly decompressed) file handle.Decompression is done on the stream, to reduce memory requirements
and to avoid wasted processing when ahandlerreturns early.If therecurseparameter is true and the walker encounters atarfileembedded within atarfile, its contents will also be
scanned the same way.There are two (2) classes that are provided.  The primary difference
is thatTarWalkerwill throw an exception if given a directory.TarWalkerhandles compressed or uncompressed files andtarfilearchives.TarDirWalkeris a subclass ofTarWalkerthat expands it to
recursively walk through directories, processing any files
encountered.InstallationInstall the package usingpip, eg:pip install –user tarwalkerpip3 install –user tarwalkerExamplesThe following is simple tool to look for a given string within files.
Files can be given as arguments or within tarballs, and must end with
either ‘.log’ (w/an optional numeric suffix) or with ‘.txt’:importreimportsysfromtarwalkerimportTarWalkerPATTERN=re.compile(r'.*\.(txt|log(\.\d+)?)$')defhandler(fileobj,filename,arch,info,match):try:forlineinfileobj:iftextinline:path=(arch+':')ifarchelse''print("Found in: "+path+filename)returnexceptIOError:passtext=sys.argv[1]walker=TarWalker(file_handler=handler,name_matcher=PATTERN.match,recurse=False)forarginsys.argv[2:]:walker.handle_path(arg)Constructors and CallbacksConstructing an instance ofTarWalkerorTarDirWalkertake the
same parameters.  Note that at most one offile_matcherorname_matcheris allowed.file_handler(required) a callable taking five (5) positional parameters:fileobj- a readablefileobject for the file contents.filepath- astrwith the filename, either as one of:the file path given tohandle_path(), orthe path of a file found beneath a directory given tohandle_path().the file path of a file within an expanded tar archive.archname- astrpath of the tar archive name, when handling a
file found within a tar archive.  It will be a colon (‘:’)
separated list if reading a recursive tar archive.fileinfo- may beNoneor an object with the following
attributes.  Seeos.statfor more
details:name- thestrname of the file,size- the size of the file in bytes,mtime- modification time, in POSIX (epoch) time,mode- the file permission bits,uid- the file owner’s User ID, andgid- the file owner’s Group IDMATCH - the value returned from thename_matcherorfile_matchercall.NOTE:files with a compression suffix will have the suffix
removed, and the file object will return decompressed contents.For example, for “foo.txt.gz”filepathwould be “foo.txt” andfileobjwould be the equivalent contents of “foo.txt”.file_matcher(optional) a callable that takes two (2) positional
parameters and returns true if the file should be opened and
passed to thefile_handlercallback:filepath- Seefilepathabove.fileinfo- Seefileinfoabove.name_matcher(optional) a callable that takes one (1) positional
parameter  and returns true if the file be opened and passed tofile_handler:filepath- Seefile_handler, above.recurse(optional) If true, the algorithm will recurse into
tarballs found within other tarballs. Furthermore, ifrecurseis a
callable it will be called before and after opening an interior
tarball, with four (4) positional parameters:start- a bool that indicates recursion into the given tarball
is starting; it is False on the second call.tarname- name of the contained (interior) tarball, seefilepathabove.archive- the name of the containing (exterior) tarball, seearchnameabove.fileinfo- Seefileinfoabove.Known IssuesIf you think you have found a defect, or wish to add an enhancement
request, please do so via theGitLab issues page:.The ARCHNAME passed to thefile_handlercallback uses ‘:’ as a
separator, which is a legal filename component, so does not
necessarily indicate a nested archive.Therecursefeature will scan an embeddedtarfile, but there is
currently no mechanism to avoid scanning atarfilefound within an
embeddedtarfile(at any level).  If needed, please submit an
enhancement request.There are lots of other compression algorithms that are not handled.