Package: dataintegrityfingerprint
Data Integrity Fingerprint (DIF)A reference implementation in PythonCommand line interface (CLI) applicationGraphical user interface (GUI) applicationProgramming library (Python package)byOliver Lindemann&Florian KrauseTable of contentsIntroductionInstallationUsageCommand line interface (CLI) application usageGraphical user interface (GUI) application usageProgramming library (Python package) usageSupport and contributionIntroductionThis software calculates theData Integrity Fingerprint (DIF)of multi-file datasets. It can be used via the command line, via a graphical user interface, or as a Python library for embedding in other software. In either case, the user has the choice of calculating the DIF based on a variety of (cryptographic) algorithms using serial (single CPU core) or parallel (multiple CPU cores) computing. In addition, a checksums file with fingerprints of individual files in a dataset can be created. These files can also serve as the basis for calculating the DIF and, in addition, can be compared against a dataset in order to reveal content differences in case a DIF could not be verified.Note:We strongly recommend to use SHA-256 or one of the other cryptographic algorithms for calculating the DIF. The non-cryptographic algorithms are significantly faster, but also significantly less secure (i.e. collisions are much more likely, breaking the uniqueness of a DIF, and opening a door for potential manipulation). They might hence only be an option for very large datasets in scenarios where a potential manipulation by a third party is not part of the threat model. The graphical user interface does not allow for selecting non-cryptographic algorithms.InstallationThe quickest way to use the application is to install it withpipx:pipx install dataintegrityfingerprintTo also make use of the programming library, a classical pip installation is of course also possible:python -m pip install dataintegrityfingerprintUsageCommand line interface (CLI) application usageAfter successful installation, the command line interface is available asdataintegrityfingerprint:dataintegrityfingerprint [-h] [-f] [-a ALGORITHM] [-C] [-D] [-G] [-L] [-s]
                         [-d CHECKSUMSFILE] [-n] [-p] [--non-cryptographic]
                         [PATH]
                         
positional arguments:
  PATH                  the path to the data directory

options:
  -h, --help            show this help message and exit
  -f, --from-checksums-file
                        Calculate dif from checksums file. PATH is a checksums
                        file
  -a ALGORITHM, --algorithm ALGORITHM
                        the hash algorithm to be used (default=SHA-256)
  -C, --checksums       print checksums only
  -D, --dif-only        print dif only
  -G, --gui             open graphical user interface
  -L, --list-available-algorithms
                        print available algorithms
  -s, --save-checksums-file
                        save checksums to file
  -d CHECKSUMSFILE, --diff-checksums-file CHECKSUMSFILE
                        Calculate differences of checksums to CHECKSUMSFILE
  -n, --no-multi-processing
                        switch of multi processing
  -p, --progress        show progressbar
  --non-cryptographic   allow non cryptographic algorithms (Not suggested,
                        please read documentation carefully!)Graphical user interface (GUI) application usageAfter successful installation, the graphical user interface is available asdataintegrityfingerprint-gui:Button "Browse..."- Opens a file browser for selecting a data directory.
The selected data directory will be shown at the top of the interface.Button "Generate DIF"- Generates the DIF for the selected data
directory. The DIF will be shown at the bottom of the interface. In addition,
the main area in the middle of the interface will show the checksums
(fingerprints) of individual files.Button "Copy"- Copies the DIF into the clipboard for pasting into other
applications.Menu item "File --> Open checksums"- Opens a checksums file. The DIF of
that checksums file will be shown at the bottom of the interface. In
addition, the main area in the middle of the interface will show the
checksums (fingerprints) of individual files.Menu item "File --> Save checksums"- Saves the checksums (fingerprints)
of individual files to a file.Menu item "File --> Quit"- Quits the application.Menu item "Edit --> Diff checksums"- Opens a checksums file and shows
differences of checksums (fingerprints) of individual files to those
currently shown in the main area in the middle of the interface.Menu item "Options --> Hash algorithm"- Selects the cryptographic hash
algorithm used as basis for DIF calculation.Menu item "Progress updating"- Enables/disables progress updating via a
progress bar.Menu item "Options --> Multi-core processing"- Enables/disables parallel
computing (usage of multiple CPU cores).Programming library (Python package) usageAfter successful installation, the Python package is available asdataintegrityfingerprint:importdataintegrityfingerprintA DIF can then be created in the following way:dif=dataintegrityfingerprint.DataIntegrityFingerprint("/path/to/dataset")print(dif)# get the DIFprint(dif.checksums)# get the list of checksums of individual filesAPI documentationThe main functionality for usage in other code is made available via the classDataIntegrityFingerprint.DataIntegrityFingerprintCreate a DataIntegrityFingerprint object.DataIntegrityFingerprint(data,
                         from_checksums_file=False,
                         hash_algorithm='SHA-256',
                         multiprocessing=True,
                         allow_non_cryptographic_algorithms=False)
 
    Parameters
    ----------
    data : str
        the path to the data
    from_checksums_file : bool
        data argument is a checksums file
    hash_algorithm : str
        the hash algorithm (optional, default: sha256)
    multiprocessing : bool
        using multi CPU cores (optional, default: True)
        speeds up creating of checksums for large data files
    allow_non_cryptographic_algorithms : bool
        set True only, if you need non cryptographic algorithms (see
        notes!)
    
    Note
    ----
    We do not suggest to use non-cryptographic algorithms.
    Non-cryptographic algorithms are, while much faster, not secure (e.g.
    can be tempered with). Only use these algorithms to check for technical
    file damage and in cases security is not of critical concern.TheDataIntegrityFingerprintclass includes a set of global variables which
affect all instances.CHECKSUM_FILENAME_SEPARATOR = '  'Global variable.Default value ='␣␣'(i.e., two U+0020 whitespace characters)CRYPTOGRAPHIC_ALGORITHMSGlobal variable.Default value =['MD5', 'SHA-1', 'SHA-224', 'SHA-256', 'SHA-384', 'SHA-512', 'SHA3-224', 'SHA3-256', 'SHA3-384', 'SHA3-512']NON_CRYPTOGRAPHIC_ALGORITHMSGlobal variable.Default value =['ADLER-32', 'CRC-32']Once initiated, aDataIntegrityFingerprintobject provides several methods and
attributes.dif_checksumsCalculate differences of checksums to checksums file.diff_checksums(filename)
    
    Parameters
    ----------
    filename : str
        the name of the checksums file
    
    Returns
    -------
    diff : str
        the difference of checksums to the checksums file
        (minus means checksums is missing something from checksums file,
        plus means checksums has something in addition to checksums file)generateGenerate hash list to get Data Integrity Fingerprint.generate(progress=None)
    
    Parameters
    ----------
    progress: function, optional
        a callback function for a progress reporting that takes the
        following parameters:
            count  -- the current count
            total  -- the total count
            status -- a string describing the statusget_filesGet all files to hash.get_files(self)
   
   Returns
   -------
   files : list
       the list of files to hashsave_checksumsSave the checksums to a file.save_checksums(filename=None)
   
   Parameters
   ----------
   filename : str, optional
       the name of the file to save checksums to
   
   Returns
   -------
   success : bool
       whether saving was successfulAn initiatedDataIntegrityFingerprintobject also provides a set of
read-only properties.allow_non_cryptographic_algorithmsRead-only propertychecksumsRead-only property.dataRead-only property.difRead-only property.file_countRead-only property.file_hash_listRead-only property.hash_algorithmRead-only property.multiprocessingRead-only property.Support and contributionFor any questions, please use thediscussionsection from the code repository.
If you wish to contribute or report an issue, please use theissue trackerandpull requests.