Package: iterfilesystem
IterFilesystemMultiprocess directory iteration viaos.scandir()Who’s this Lib for?You want to process a large number of files and/or a few very big files and give feedback to the user on how long it will take.Features:Progress indicator:Immediately after start: process files and indication of progress via multiprocessprocess bars viatqdmEstimated time based on file count and sizeEasy to implement extra process bar for big file processing.Skip directories and file name via fnmatch.How it works:The main process startsstatisticprocesses in background via Python multiprocess and starts directly with the work.There are two backgroundstatisticprocesses collects information for the process bars:Count up all directories and files.Accumulates the sizes of all files.Why two processes?Because collect only the count of all filesystem items viaos.scandir()is very fast. This is the fastest way to predict a processing time.Useos.DirEntry.stat()to get the file size is significantly slower: It requires another system call.OK, but why two processed?Use only the total count of allDirEntrymay result in bad estimated time Progress indication.
It depends on what the actual work is about: When processing the contents of large files, it is good to know how much total data to be processed.That’s why we used two ways: theDirEntrycount to forecast a processing time very quickly and the size to improve the predicted time.requirements:Python 3.6 or newer.tqdmfor process barspsutilsfor setting process priorityFor dev.:Pipenv. Packages and virtual environment managercontributePlease: try, fork and contribute! ;)travis-ci.org/jedie/IterFilesystemci.appveyor.com/project/jedie/IterFilesystemcodecov.io/gh/jedie/IterFilesystemcoveralls.io/r/jedie/IterFilesystemrequires.io/github/jedie/IterFilesystem/requirements/ExampleUse example CLI, e.g.:~$ git clone https://github.com/jedie/IterFilesystem.git
~$ cd IterFilesystem
~/IterFilesystem$ pipenv install
~/IterFilesystem$ pipenv shell
(IterFilesystem) ~/IterFilesystem$ print_fs_stats --help
(IterFilesystem) ~/IterFilesystem$ pip install -e .
...
Successfully installed iterfilesystem

~/IterFilesystem$ $ poetry run print_fs_stats --help
usage: print_fs_stats.py [-h] [-v] [--debug] [--path PATH]
                         [--skip_dir_patterns [SKIP_DIR_PATTERNS [SKIP_DIR_PATTERNS ...]]]
                         [--skip_file_patterns [SKIP_FILE_PATTERNS [SKIP_FILE_PATTERNS ...]]]

Scan filesystem and print some information

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         show program's version number and exit
  --debug               enable DEBUG
  --path PATH           The file path that should be scanned e.g.: "~/foobar/"
                        default is "~"
  --skip_dir_patterns [SKIP_DIR_PATTERNS [SKIP_DIR_PATTERNS ...]]
                        Directory names to exclude from scan.
  --skip_file_patterns [SKIP_FILE_PATTERNS [SKIP_FILE_PATTERNS ...]]
                        File names to ignore.example output looks like this:(IterFilesystem) ~/IterFilesystem$ $ print_fs_stats --path ~/IterFilesystem --skip_dir_patterns ".*" "*.egg-info" --skip_file_patterns ".*"
Read/process: '~/IterFilesystem'...
Skip directory patterns:
    * .*
    * *.egg-info

Skip file patterns:
    * .*

Filesystem items..:Read/process: '~/IterFilesystem'...

...

Filesystem items..: 100%|█████████████████████████████████████████|135/135 13737.14entries/s [00:00<00:00, 13737.14entries/s]
File sizes........: 100%|██████████████████████████████████████████████████████████████|843k/843k [00:00<00:00, 88.5MBytes/s]
Average progress..: 100%|███████████████████████████████████████████████████████████████████████████████████████|00:00<00:00
Current File......:, /home/jens/repos/IterFilesystem/Pipfile


Processed 135 filesystem items in 0.02 sec
SHA515 hash calculated over all file content: 10f9475b21977f5aea1d4657a0e09ad153a594ab30abc2383bf107dbc60c430928596e368ebefab3e78ede61dcc101cb638a845348fe908786cb8754393439ef
File count: 109
Total file size: 843.5 KB
6 directories skipped.
6 files skipped.History**dev** - compare v1.4.3…masterTBC16.03.2020 - v1.4.3Use logging and remove “verbose mode”Nicer “Average progess” barBugfix “Current File” bar: remove comma16.02.2020 - v1.4.2iterate over sorted dir entriesupdate CI pipelines02.02.2020 - v1.4.1Bugfixhuman_filesize02.02.2020 - v1.4.0stats_helper.abortexists always usefull to get information if KeyboardInterrupt was useduse poetry and modernize project setup20.10.2019 - v1.3.1Bugfix if scan directory is completely empty13.10.2019 - v1.3.0Set ionice and nice priority via psutils13.10.2019 - v1.2.0Refactor APIcleanup statistics and process barhandle access errors like:Permission deniedfix tests12.10.2019 - v1.1.0don’t create separate process for worker: Just do the work in main processdir/file filter uses nowfnmatch12.10.2019 - v1.0.0refactoring:don’t usepersist-queueswitch from threading to multiprocessingenhance progress display with multipletqdmprocess bars15.09.2019 - v0.2.0store persist queue in temp directoryDon’t catchprocess_path_itemerrors, this should be made in child class15.09.2019 - v0.1.0add some project meta files and testssetup CIfix tests15.09.2019 - v0.0.1first Release on PyPiLinkshttps://pypi.python.org/pypi/IterFilesystem/https://github.com/jedie/IterFilesystem/Donatingpaypal.me/JensDiemerFlattr This!SendBitcoinsto1823RZ5Md1Q2X5aSXRC5LRPcYdveCiVX6FNote: this file is generated from README.creole2020-03-1618:09:30 with"python-creole"