Package: deplicate
deplicate=========Table of contents------------------  `Description`_-  `Features`_-  `Installation`_-  `PIP Install`_-  `Tarball Install`_-  `Usage`_-  `Quick Start`_-  `Advanced Usage`_-  `API Reference`_-  `Exceptions`_-  `Classes`_-  `Functions`_Description-----------``deplicate`` is an high-performance duplicate file finder written inPure Python with low memory impact and several advanced filters.Find out all the duplicate files in one or more directories, you canalso scan directly a bunch of files. Latest releases let you to removethe spotted duplicates and/or apply a custom action over them.Features---------  [x] N-tree layout for low memory consumption-  [x] Multi-threaded (partially)-  [x] Raw drive access to maximize I/O performance (Unix only)-  [x] xxHash algorithm for fast file identification-  [x] File size and signature checking for quick duplicate exclusion-  [x] Extended file attributes scanning-  [x] Multi-filtering-  [x] Full error handling-  [x] Unicode decoding-  [x] Safe from path walking loop-  [ ] SSD detection-  [x] Duplicates purging-  [x] Support for moving dulicates to trash/recycle bin-  [x] Custom action handling over deletion-  [x] **Command Line Interface**(https://github.com/deplicate/deplicate-cli)-  [x] Unified structured result-  [x] Support posix\_fadvise-  [ ] Graphical User Interface(https://github.com/deplicate/deplicate-gui)-  [ ] Incremental file chunk checking-  [ ] Hard-link scanning-  [ ] Duplicate directories recognition-  [ ] Multi-processing-  [x] Fully documented-  [ ] PyPy support-  [ ] [STRIKEOUT:Exif data scanning]Installation------------**Note:** This will install just ``deplicate``, without its CLIand GUI.CLI *(Command Line Interface)*:https://github.com/deplicate/deplicate-cli.GUI *(Graphical User Interface)*:https://github.com/deplicate/deplicate-gui.The easiest way to install ``deplicate`` on your system is the `PIPInstall way`_, but, if you want, you can try to install it from thesources as described in the `Tarball Install section`_.PIP Install~~~~~~~~~~~If the command ``pip`` is not found in your system, install the latest``pip`` distribution: download `get-pip.py`_ and run it using the`Python Interpreter`_.Then, type in your command shell **with *administrator/root*privileges**:::pip install deplicateIn Unix-based systems, you may have to type``sudo pip install deplicate``.If the above command fails, consider installing with the option`--user`_:::pip install --user deplicateTarball Install~~~~~~~~~~~~~~~0. Make sure you have installed the `Python Interpreter`_ with thepackage ``setuptools`` **(>=20.8.1)**.1. Get the latest tarball of the source code in format `ZIP`_ or `TAR`_.2. Extract the downloaded archive.3. From the extracted path, execute the command``python setup.py install``.Usage-----In your script import the module ``duplicate``.::import duplicateCall its function ``find`` to search the duplicate files in the givenpath:::duplicate.find('/path')Or call the function ``purge`` if you want to remove them in addition:::duplicate.purge('/path')Youll get a ``duplicate.ResultInfo`` object as result, with thefollowing properties: - ``dups``  Tuples of paths of duplicate files. -``deldups``  Tuple of paths of purged duplicate files. - ``duperrors``Tuple of paths of files not filtered due errors. - ``scanerrors``Tuple of paths of files not scanned due errors. - ``delerrors``  Tupleof paths of files not purged due errors.**Note:** By default, directory paths are scanned recursively.**Note:** By default, files smaller than **100 KiB** or bigger than**100 GiB** are not scanned.**Note:** File paths are returned in canonical form.**Note:** Tuples of duplicate files are sorted in descending orderaccording input priority, file modification time and name length.Quick Start~~~~~~~~~~~Scan for duplicates a single directory:::import duplicateduplicate.find('/path/to/dir')Scan for duplicates two files (at least):::import duplicateduplicate.find('/path/to/file1', '/path/to/file2')Scan for duplicates a single directory and move them to thetrash/recycle bin:::import duplicateduplicate.purge('/path/to/dir')Scan for duplicates a single directory and delete them:::import duplicateduplicate.purge('/path/to/dir', trash=False)Scan more directories together:::import duplicateduplicate.find('/path/to/dir1', '/path/to/dir2', '/path/to/dir3')Scan from iterable:::import duplicateiterable = ['/path/to/dir1', '/path/to/dir2', '/path/to/dir3']duplicate.find.from_iterable(iterable)Scan ignoring the minimum file size threshold:::import duplicateduplicate.find('/path/to/dir', minsize=0)Advanced Usage~~~~~~~~~~~~~~Scan without recursing directories:::import duplicateduplicate.find('/path/to/file1', '/path/to/file2', '/path/to/dir1',recursive=False)**Note:** In *not-recursive mode*, like the case above, directorypaths are simply ignored.Scan checking file names and hidden files:::import duplicateduplicate.find.from_iterable('/path/to/file1', '/path/to/dir1',comparename=True, scanhidden=True)Scan excluding files ending with extension ``.doc``:::import duplicateduplicate.find('/path/to/dir', exclude="*.doc")Scan including file links:::import duplicateduplicate.find('/path/to/file1', '/path/to/file2', '/path/to/file3',scanlinks=True)Scan for duplicates, handling errors with a custom action (printing):::import duplicatedef error_callback(exc, filename):print(filename)duplicate.find('/path/to/dir', onerror=error_callback)Scan for duplicates and apply a custom action (printing), instead ofpurging:::import duplicatedef purge_callback(filename):print(filename)raise duplicate.SkipExceptionduplicate.purge('/path/to/dir', ondel=purge_callback)Scan for duplicates, apply a custom action (printing) and move them tothe trash/recycle bin:::import duplicatedef purge_callback(filename):print(filename)duplicate.purge('/path/to/dir', ondel=purge_callback)Scan for duplicates, handling errors with a custom action (printing),and apply a custom action (moving to path), instead of purging:::import shutilimport duplicatedef error_callback(exc, filename):print(filename)def purge_callback(filename):shutil.move(filename, '/path/to/custom-dir')raise duplicate.SkipExceptionduplicate.purge('/path/to/dir',ondel=purge_callback, onerror=error_callback)API Reference-------------Exceptions~~~~~~~~~~-  duplicate.\ ``SkipException``\ (*\*args*, *\*\*kwargs*)-  **Description**: Raised to skip file scanning, filtering or purging.-  **Return**: Self instance.-  **Parameters**: Same as built-in ``Exception``.-  **Proprieties**: Same as built-in ``Exception``.-  **Methods**: Same as built-in ``Exception``.Classes~~~~~~~-  duplicate.\ ``Cache``\ (*maxlen*\ =\ ``DEFAULT_MAXLEN``)-  **Description**: Internal shared cache class.-  **Return**: Self instance.-  **Parameters**:-  ``maxlen``  Maximum number of entries stored.-  **Proprieties**:-  ``DEFAULT_MAXLEN``-  **Description**: Default maximum number of entries stored.-  **Value**: ``128``.-  **Methods**:--  ``clear``\ (*self*)-  **Description**: Clear the cache if not acquired by any object.-  **Return**: ``True`` if went cleared, otherwise ``False``.-  **Parameters**: None.-  duplicate.\ ``Deplicate``\ (*paths*,*minsize*\ =\ ``DEFAULT_MINSIZE``, *maxsize*\ =\ ``DEFAULT_MAXSIZE``,*include*\ =\ ``None``, *exclude*\ =\ ``None``,*comparename*\ =\ ``False``, *comparemtime*\ =\ ``False``,*comparemode*\ =\ ``False``, *recursive*\ =\ ``True``,*followlinks*\ =\ ``False``, *scanlinks*\ =\ ``False``,*scanempties*\ =\ ``False``, *scansystem*\ =\ ``True``,*scanarchived*\ =\ ``True``, *scanhidden*\ =\ ``True``)-  **Description**: Duplicate main class.-  **Return**: Self instance.-  **Parameters**:-  ``paths``  Iterable of directory and/or file paths.-  ``minsize``  *(optional)* Minimum size in bytes of files toinclude in scanning.-  ``maxsize``  *(optional)* Maximum size in bytes of files toinclude in scanning.-  ``include``  *(optional)* Wildcard pattern of files to include inscanning.-  ``exclude``  *(optional)* Wildcard pattern of files to excludefrom scanning.-  ``comparename``  *(optional)* Check file name.-  ``comparemtime``  *(optional)* Check file modification time.-  ``compareperms``  *(optional)* Check file mode (permissions).-  ``recursive``  *(optional)* Scan directory recursively.-  ``followlinks``  *(optional)* Follow symbolic links pointing todirectory.-  ``scanlinks``  *(optional)* Scan symbolic links pointing to file(hard-links included).-  ``scanempties``  *(optional)* Scan empty files.-  ``scansystems``  *(optional)* Scan OS files.-  ``scanarchived``  *(optional)* Scan archived files.-  ``scanhidden``  *(optional)* Scan hidden files.-  **Proprieties**:-  ``DEFAULT_MINSIZE``-  **Description**: Minimum size of files to include in scanning (inbytes).-  **Value**: ``102400``.-  ``DEFAULT_MAXSIZE``-  **Description**: Maximum size of files to include in scanning (inbytes).-  **Value**: ``107374182400``.-  ``result``-  **Description**: Result of ``find`` or ``purge`` invocation (bydefault is ``None``).-  **Value**: ``duplicate.ResultInfo``.-  **Methods**:-  ``find``\ (*self*, *onerror*\ =\ ``None``, *notify*\ =\ ``None``)-  **Description**: Find duplicate files.-  **Return**: None.-  **Parameters**:-  ``onerror``  *(optional)* Callback function called with twoarguments, ``exception`` and ``filename``, when an error occursduring file scanning or filtering.-  ``notify``  *(internal)* Notifier callback.-  ``purge``\ (*self*, *trash*\ =\ ``True``, *ondel*\ =\ ``None``,*onerror*\ =\ ``None``, *notify*\ =\ ``None``)-  **Description**: Find and purge duplicate files.-  **Return**: None.-  **Parameters**:-  ``trash``  *(optional)* Move duplicate files to trash/recyclebin, instead of deleting.-  ``ondel``  *(optional)* Callback function called with onearguments, ``filename``, before purging a duplicate file.-  ``onerror``  *(optional)* Callback function called with twoarguments, ``exception`` and ``filename``, when an error occursduring file scanning, filtering or purging.-  ``notify``  *(internal)* Notifier callback.-  duplicate.\ ``ResultInfo``\ (*dupinfo*, *delduplist*, *scnerrlist*,*delerrors*)-  **Description**: Duplicate result class.-  **Return**: ``collections.namedtuple``\ (``'ResultInfo'``,``'dups deldups duperrors scanerrors delerrors'``).-  **Parameters**:-  ``dupinfo``  *(internal)* Instance of``duplicate.structs.DupInfo``.-  ``delduplist``  *(internal)* Iterable of purged files (deleted ortrashed).-  ``scnerrlist``  *(internal)* Iterable of files not scanned (dueerrors).-  ``delerrors``  *(internal)* Iterable of files not purged (dueerrors).-  **Proprieties**: Same as ``collections.namedtuple``.-  **Methods**: Same as ``collections.namedtuple``.Functions~~~~~~~~~-  duplicate.\ ``find``\ (*\*paths*,*minsize*\ =\ ``duplicate.Deplicate.DEFAULT_MINSIZE``,*maxsize*\ =\ ``duplicate.Deplicate.DEFAULT_MAXSIZE``,*include*\ =\ ``None``, *exclude*\ =\ ``None``,*comparename*\ =\ ``False``, *comparemtime*\ =\ ``False``,*comparemode*\ =\ ``False``, *recursive*\ =\ ``True``,*followlinks*\ =\ ``False``, *scanlinks*\ =\ ``False``,*scanempties*\ =\ ``False``, *scansystem*\ =\ ``True``,*scanarchived*\ =\ ``True``, *scanhidden*\ =\ ``True``,*onerror*\ =\ ``None``, *notify*\ =\ ``None``)-  **Description**: Find duplicate files.-  **Return**: ``duplicate.ResultInfo``.-  **Parameters**:-  ``paths``  Iterable of directory and/or file paths.-  ``minsize``  *(optional)* Minimum size in bytes of files toinclude in scanning.-  ``maxsize``  *(optional)* Maximum size in bytes of files toinclude in scanning.-  ``include``  *(optional)* Wildcard pattern of files to include inscanning.-  ``exclude``  *(optional)* Wildcard pattern of files to excludefrom scanning.-  ``comparename``  *(optional)* Check file name.-  ``comparemtime``  *(optional)* Check file modification time.-  ``compareperms``  *(optional)* Check file mode (permissions).-  ``recursive``  *(optional)* Scan directory recursively.-  ``followlinks``  *(optional)* Follow symbolic links pointing todirectory.-  ``scanlinks``  *(optional)* Scan symbolic links pointing to file(hard-links included).-  ``scanempties``  *(optional)* Scan empty files.-  ``scansystems``  *(optional)* Scan OS files.-  ``scanarchived``  *(optional)* Scan archived files.-  ``scanhidden``  *(optional)* Scan hidden files.-  ``onerror``  *(optional)* Callback function called with twoarguments, ``exception`` and ``filename``, when an error occursduring file scanning or filtering.-  ``notify``  \_(internal)\_ *(optional)* Notifier callback.-  duplicate.\ ``purge``\ (*\*paths*,*minsize*\ =\ ``duplicate.Deplicate.DEFAULT_MINSIZE``,*maxsize*\ =\ ``duplicate.Deplicate.DEFAULT_MAXSIZE``,*include*\ =\ ``None``, *exclude*\ =\ ``None``,*comparename*\ =\ ``False``, *comparemtime*\ =\ ``False``,*comparemode*\ =\ ``False``, *recursive*\ =\ ``True``,*followlinks*\ =\ ``False``, *scanlinks*\ =\ ``False``,*scanempties*\ =\ ``False``, *scansystem*\ =\ ``True``,*scanarchived*\ =\ ``True``, *scanhidden*\ =\ ``True``,*trash*\ =\ ``True``, *ondel*\ =\ ``None``, *onerror*\ =\ ``None``,*notify*\ =\ ``None``)-  **Description**: Find and purge duplicate files.-  **Return**: ``duplicate.ResultInfo``.-  **Parameters**:-  ``paths``  Iterable of directory and/or file paths.-  ``minsize``  *(optional)* Minimum size in bytes of files toinclude in scanning.-  ``maxsize``  *(optional)* Maximum size in bytes of files toinclude in scanning.-  ``include``  *(optional)* Wildcard pattern of files to include inscanning.-  ``exclude``  *(optional)* Wildcard pattern of files to excludefrom scanning.-  ``comparename``  *(optional)* Check file name.-  ``comparemtime``  *(optional)* Check file modification time.-  ``compareperms``  *(optional)* Check file mode (permissions).-  ``recursive``  *(optional)* Scan directory recursively.-  ``followlinks``  *(optional)* Follow symbolic links pointing todirectory.-  ``scanlinks``  *(optional)* Scan symbolic links pointing to file(hard-links included).-  ``scanempties``  *(optional)* Scan empty files.-  ``scansystems``  *(optional)* Scan OS files.-  ``scanarchived``  *(optional)* Scan archived files.-  ``scanhidden``  *(optional)* Scan hidden files.-  ``trash``  *(optional)* Move duplicate files to trash/recyclebin, instead of deleting.-  ``ondel``  *(optional)* Callback function called with onearguments, ``filename``, before purging a duplicate file.-  ``onerror``  *(optional)* Callback function called with twoarguments, ``exception`` and ``filename``, when an error occursduring file scanning, filtering or purging.-  ``notify``  *(internal)* *(optional)* Notifier callback... _Description: #description.. _Features: #features.. _Installation: #installation.. _PIP Install: #pip-install.. _Tarball Install: #tarball-install.. _Usage: #usage.. _Quick Start: #quick-start.. _Advanced Usage: #advanced-usage.. _API Reference: #api-reference.. _Exceptions: #exceptions.. _Classes: #classes.. _Functions: #functions.. _PIP Install way: #pip-install.. _Tarball Install section: #tarball-install.. _get-pip.py: https://bootstrap.pypa.io/get-pip.py.. _Python Interpreter: https://www.python.org.. _--user: https://pip.pypa.io/en/latest/user_guide/#user-installs.. _ZIP: https://github.com/deplicate/deplicate/archive/master.zip.. _TAR: https://github.com/deplicate/deplicate/archive/master.tar.gz