Package: ascmhl
ASC Media Hash List (ASC MHL)The software in this repository is the reference implementation for the ASC Media Hash List (ASC MHL) format
specification by the Advanced Data Management Subcommittee of the ASC Motion Imaging Technology Council (MITC) at
theAmerican Society of Cinematographers(ASC).Resources:📄ASC MHL Specification page(attheasc.com)In case you are looking for the original specification of MHL, please take a look athttps://mediahashlist.org.Ensuring file integrity when backing up and verifying files during production and post-production is of utmost
importance. The ASC MHL is used to create a chain of custody by tracking each and every copy made between the
media’s initial download on set, all the way through to final archival.The ASC MHL uses common checksum methods for hashing files and folders, specifies what information is gathered, where
the checksum is placed, and documents these hashes together with essential file metadata in an XML format that is
human-readable.This repository holds all information about the document format, a reference implementation, and tools.ASC MHL Format SpecificationThe ASC MHL consists of adefinition of naming conventions for the ascmhl folder and the file names of its contentXML schema for the ASC MHL filesdefinition for the chain fileThe schema definition can be found in the./xsdfolder.mhllibReference ImplementationThe implementation of a reference library aims to be used in applications and tools dealing with ASC MHL files. The
library takes responsibility of dealing with complex use cases of nesting and assembling of information.The reference library coversreading ascmhl folders and their contentsparsing and writing of ASC MHL XML filesparsing and writing ASC MHL chain and collection filesdealing with nested mhl foldersASC MHL supports the hash formatsxxHash (64-bit, and latest XXH3 with 64-bit and 128-bit)MD5SHA1, SHA256C4The source code formhllibcan be found in the./ascmhlfolder.TheascmhlToolTheascmhltool is a command line tool based onmhllibthat allows to perform typical activities for the use
cases of ASC MHL.Thecreatecommand: Create a new generation for a folder or file(s)Thediffcommand: Diff an entire folder structureTheflattencommand: Flatten an MHL history into one external manifestTheinfocommand: Prints information from the ASC MHL historyTypical scenarios, sample CLI output, and generated ASC MHL files can be found in theREADME.mdfile in theexamples/scenariosfolder
of the git repository.The documentation can also be found athttps://ascmhl.readthedocs.io/Theascmhl-debugToolTheascmhl-debugtool is a command line tool with additional operations and commands that might come in handy during
implementation or testing.Theverifycommand: Verify a folder, single file(s), or a directory hash (without writing a
new generation)Thexsd-schema-checkcommand: Checks a .mhl file against the xsd schema definitionThehashcommand: Create and print a hash value for a fileGetting startedThemhllibas well as theascmhlandascmhl-degugtools require a few dependencies that need to be installed
first.For installing system dependencies on macOSHomebrewis recommended.System requirementsMake sure you have Python 3 installed:$brewinstallpython3
$brewpostinstallpython3Installing / updatingascmhlas a userPlease run the following command to install (or upgrade to) the latest development version ofascmhl:$pip3install--upgradeascmhlTo verify that it has been correctly installed run:$ascmhl--helpInstallingascmhlas a developerDownload the source code and install dependencies using aVirtual Environment:$gitclonehttps://github.com/ascmitc/mhl.git
$cdmhl
$python3-mvenvenv
$sourceenv/bin/activate
$pip3install--editable.This will install the wrapper scriptsascmhlandascmhl-debugto be available on your$PATH. Inside the
virtualenv, this wrapper will be installed asenv/bin/ascmhl. Regular users might have it in/Library/Frameworks/Python.framework/Versions/3.9/bin/ascmhlor/usr/local/bin. For Windows users, pip will
create anascmhl.exeand anascmhl-debug.exe.More information on installing Python commandline tools usingentry_pointscan be found here:https://setuptools.readthedocs.io/en/latest/userguide/entry_point.htmlhttps://packaging.python.org/specifications/entry-points/#use-for-scriptshttps://click.palletsprojects.com/en/master/setuptools/Adding the-e / --editableflag installs a linked version to yoursite-packagesdirectory to allow editing the
source files in your working directory as usual.Common Scenarios forascmhlTheascmhltool can be used toverify and create new MHL generations for given files and folders (commandcreate),print differences between the records in the MHL history and given files and folders (commanddiff),create one "flattened" manifest file from a history (commandflatten), andprint information about an MHL history (commandinfo).Working with file hierarchies (with completeness check)The most common commands when using theascmhlin data management scenarios are thecreateand thecheckcommands in their default behavior (without subcommand options).Creating a new generation for a folder / drive with thecreatecommand traverses through a folder hierarchy, hashes
all found files and compares the hashes against the records in theascmhlfolder (if present). The command creates
a new generation (or an initial one) for the content of an entire folder at the given folder level. It can be used to
document all files in a folder or drive with all verified or newly created file hashes of the moment thecreatecommand ran.Thediffcommand also traverses through the content of a folder / drive.  Thediffcommand thus behaves like theverifycommand, but thediffcommand does not hash any files (e.g. doesn't do file verification) and thus is much
faster in execution. It can be used to print all files that are existent in the file system and are not registered in
theascmhlfolder yet, and all files that are registered in theascmhlfolder but that are missing in the file
system.Checking a folder / drive with theverifycommand (of theascmhl-debugtool) traverses through the content of a folder,
hashes all found files and compares the hashes against the records in theascmhlfolder. Theverifycommand behaves
like acreatecommand (both without additional options), but doesn't write new generations. It can be used to verify
the content of a received drive with existing ascmhl information.Working with single files (without completeness check)In some scenarios working with an entire folder structure is not adequate, and finer control of the processes files
is needed. For those scenarios thecreatecommand is used with additional subcommand options.Adding single files in a new generation with thecreate -sf("single files, no completeness check") command allows
to add single files to an existing folder structure and create new generations only with records of these files.Theinfo -sf("single file") command prints the known history of a single file with details about all generations.Hashing and verifying single files against hash information stored in theascmhlfolder with theverify -sf("single files") command (of theascmhl-debugtool) allows to "check" single files without the need for a (probably
much longer running) check of the integrity of the entire folder structure.Commands ofascmhlImplementation status 2022-09:Implemented:create,flatten(partially),diff,info(partially)ThecreatecommandThecreatecommand hashes all files given with the different options and creates a new generation in the mhl-history
with records for all hashed files. The command compares the hashes against the hashes stored in previous generations
if available.createdefault behavior (for file hierarchy, with completeness check)Thecreatecommand traverses through a folder hierarchy (such as a folder with media files, a camera card, or an
entire drive). The command hashes all files (not ignored by the given ignore patterns given with the-ior-iioptions) and the hashes are compared against records in theascmhlfolder. It records all hashed files in the new
generation. Directory hashes are computed and also recorded in the new generation.The command detects, prints error, and exits with a non-0 exit code if it finds files that are registered in theascmhlfolder but that are missing in the file system.Files that are existent in the file system but are not registered in theascmhlfolder yet, are registered as new
entries in the newly created generation(s).Thecreatecommand takes the root path of the file hierarchy as the parameter:$ ascmhl create [-i ignore pattern|-ii /path/to/ignore-file.txt] [creator-info options] /path/to/folder/Creator-info options:--location: Location value of the<creatorinfo>element.--comment: Comment value of the<creatorinfo>element.--author_name: Name value of the<author>element in the<creatorinfo>element.--author_email: Email value of the<author>element in the<creatorinfo>element (--author_namemust also be
set for this option).--author_phone: Phone value of the<author>element in the<creatorinfo>element (--author_namemust also be
set for this option).--author_role: Role value of the<author>element in the<creatorinfo>element (--author_namemust also be
set for this option).It works on folders with or without anascmhlfolder within the given folder hierarchy, and creates a newascmhlfolder at the given folder level if none is present before.ascmhlfolders further down the file hierarchy are read, handled, and referenced in top-levelascmhlfolders.
Existingascmhlfolders further down the folder structure will also get a new generation added.Implementation:read (recursive) mhl history (mhllib)
traverse folder
 	hash each file
	if `ascmhl` folder exists, compare hash (mhllib)
	on error (including mismatching hashes):
		print error
	 	continue
 	add files to new generation if not present yet
compare found files in file system with records in ascmhl folder and \
   warn if files are missing that are recorded in the ascmhl folder
create new generation(s) (mhllib)createwith-sfoption(s) (for single file(s), no completeness check)Thecreatecommand with-sfoption is run with the root path of the file hierarchy as well as one or multiple
paths to the individual files to be recorded as the parameters.This command can be used for instance when adding single files to an already mhl-managed file hierarchy.$ ascmhl create /path/to/root/folder -sf /path/to/single/file1 [-sf /path/to/single/file2 ..]A new generation is created in allascmhlfolders below the given root path (e.g. in a nested mhl-history). If no
mhl-history is present yet, an error is thrown.No other files than the ones specified as-sfoptions are handled by this command.Implementation:read (recursive) mhl-history (mhllib) starting from root path
for each file from input
	check if file is not recorded in `ascmhl` folder yet
	hash file
	add record for file to new generation (mhllib)
		add a new generation if necessary in appropriate `ascmhl` folder (mhllib)TheflattencommandTheflattencommand takes the root path of the file hierarchy and the destination path for the flattened manifest as
the parameter:$ ascmhl flatten [-i ignore pattern|-ii /path/to/ignore-file.txt] [creator-info options] /path/to/folder/ /destination/path/Creator-info options:--location: Location value of the<creatorinfo>element.--comment: Comment value of the<creatorinfo>element.--author_name: Name value of the<author>element in the<creatorinfo>element.--author_email: Email value of the<author>element in the<creatorinfo>element (--author_namemust also be
set for this option).--author_phone: Phone value of the<author>element in the<creatorinfo>element (--author_namemust also be
set for this option).--author_role: Role value of the<author>element in the<creatorinfo>element (--author_namemust also be
set for this option).TBD% ascmhl flatten --help
Usage: ascmhl flatten [OPTIONS] ROOT_PATH DESTINATION_PATH

  Flatten an MHL history into one external manifest

  The flatten command iterates through the mhl-history, collects all known files and
  their hashes in multiple hash formats and writes them to a new mhl file outside of the
  iterated history.

Options:
  -v, --verbose              Verbose output
  -n, --no_directory_hashes  Skip creation of directory hashes, only reference
                             directories without hash
  -i, --ignore TEXT          A single file pattern to ignore.
  -ii, --ignore_spec PATH    A file containing multiple file patterns to
                             ignore.
  --author_name TEXT         Name value for the <author> element in the
                             <creatorinfo> element
  --author_email TEXT        Email value for the <author> element in the
                             <creatorinfo> element
  --author_phone TEXT        Phone value for the <author> element in the
                             <creatorinfo> element
  --author_role TEXT         Role value for the <author> element in the
                             <creatorinfo> element
  --location TEXT            Value for the <location> element in the
                             <creatorinfo> element
  --comment TEXT             Value for the <comment> element in the
                             <creatorinfo> element
  --help                     Show this message and exit.ThediffcommandThediffcommand is very similar to theverifycommand in the default behavior, only that it doesn't create hashes
and doesn't verify them. It can be used to quickly check if a folder structure has new files that have not been
recorded yet, or if files are missing.The command detects, prints errors, and exits with a non-0 exit code forall files that existent in the file system but not registered in theascmhlfolder yet, andall files that are registered in theascmhlfolder but that are missing in the file system.It is run with the root path of the file hierarchy as the parameter.$ ascmhl diff /path/to/folder/If noascmhlfolder is found on the root level, an error is thrown.ascmhlfolders are read recursively.Implementation:error if no mhl folder found on root level
read (recursive) mhl history (mhllib)
traverse folder
	on missing file:
		print error
	 	continue
compare found files in file system with records in ascmhl folder \
  and warn if files are missing that are recorded in the ascmhl folder
end with exit !=0 if at least one of the files has failed, a file was \
  missing, or new files have been foundTheinfocommandinfodefault behaviorTheascmhlfolder contains well readable XML files, but the number of recorded files, generations, hash entries,
verification info and so forth adds up to an amount of information that cannot be quickly understood. Theinfocommand helps to get a quick overview of the contents of the stored information in anascmhlfolder.Theinfocommand printsa list of generations (with the-voption also with creator info and process info)[not implemented yet]a summary (with the-ssubcommand option) of the information in an ascmhl folder, such asnumber of recorded files, and a list of the generations with their creator info, and/or[not implemented yet]a list (with the-loption) of all file (and folder) records stored in an ascmhl folder,together with relative file paths, file size, and known file hashes.It is run with the path to a specificascmhlfolder as the parameter.$ ascmhl info [-s|-l] [-v] /path/to/ascmhl/Implementation:error if no mhl folder found on root level
read (recursive) mhl history (mhllib)
if summary option:
	print summary
if list option:
	for each file record
		print file info, hashes, etc.infowith the-sfsubcommand optionTheinfocommand with the-sfsubcommand option outputs information about the full and detailed history
information about one file.$ ascmhl info -sf /path/to/file [-sf /path/to/other/file] [/root/path]The command outputs each generation where the file has been handled, including date, hash, and activity (and creator
info and absolute path with the-voption). The history information is read from the "next" ASC MHL history found in
the path, of at the given root path.Implementation:find mhl-history information in the path above (mhllib)
	error of no `ascmhl` folder is found
print detailed info for fileinfowith the-dhsubcommand option[not implemented yet]Theinfocommand with the-dhsubcommand option printsthe directory hash of a folder computed from stored file hashes of anascmhlfolder (with the-dhoption).The directory hash can be used to quickly verify if the state of a folder structure is still the same compared to the
last generation created with acreatecommand (manually compare with the hash in the<root>tag in the ASC MHL
file).It is run with the path to a specificascmhlfolder and the path to the desired folder for the computed directory
hash.$ ascmhl info -dh /path/to/ascmhl/ /path/to/sub/folderImplementation:error if no mhl folder found on root level
read (recursive) mhl history (mhllib)
calculate directory hash from file hashes
print directory hashCommands ofascmhl-debugTheverifycommandverifydefault behavior (for file hierarchy, with completeness check)Theverifycommand traverses through the content of a folder, hashes all found files  (filtered by the ignore
patterns from theascmhlfolder) and compares the hashes against the records in theascmhlfolder.The command detects, prints errors, and exits with a non-0 exit code forall files that are existent in the file system but are not registered in theascmhlfolder yet, andall files that are registered in theascmhlfolder but that are missing in the file system.It is run with the root path of the file hierarchy as the parameter.$ ascmhl verify /path/to/folder/If noascmhlfolder is found on the root level, an error is thrown.ascmhlfolders further down the file hierarchy are also read, and its recorded hashes are used for verification.Implementation:error if no mhl folder found on root level
read (recursive) mhl history (mhllib)
traverse folder
	hash each file (filtered by ignore patterns from mhl folder)
	compare hash (mhllib)
	on error (including mismatching hashes):
		print error
	 	continue
compare found files in file system with records in ascmhl folder and \
  warn if files are missing that are recorded in the ascmhl folder
end with exit !=0 if at least one of the files has failed, a file was \
  missing, or new files have been foundverifywith-sfoption (for single file, no completeness check)Theverifycommand can be used to verify a single file. It is run with the path to a single file as the parameter.The path can bethe relative path to the file starting from the root folder of the history, orthe absolute path to the file.$ ascmhl verify -sf /absolute/path/to/single/file
$ ascmhl verify -sf realtive/path/to/single/fileThe command looks for anascmhlfolder in the folders above the given files. If no mhl-history is present yet, an
error is thrown.Implementation:find mhl-history information in the path above (mhllib)
	error of no `ascmhl` folder is found
read (recursive) mhl-history (mhllib)
for file from input
	hash file
	compare hash (mhllib)
if file is not found in mhl-history, throw error
on error (including mismatching hashes):
	don't break
	print error
	end with exit !=0 if the verification has failedverifywith-dhsubcommand option (for directory hash)Theverifycommand with the-dhsubcommand (or--directory_hash) option creates the directory hash by hashing
the contained files of the given directory path (filtered by the ignore patterns from theascmhlfolder) and compares
it with the to-be-expected directory hash calculated from the file hashes (same calculation as theinfocommand with
the-dhsubcommand option).$ ascmhl verify -dh [-co [-ro]] /path/to/folderThe-cooption (or--calculate_only) only calculates and prints the directory hashes and doesn't verify them
against an existing history.
This option also works when no history is present. The-rooption (or--root_only) only calculates and prints the
root directory hash. This option is only in effect with the-cooption.Implementation:find mhl-history information in the path above (mhllib)
	error of no `ascmhl` folder is found
read (recursive) mhl history (mhllib)
calculate to-be-expected directory hash from file hashes
traverse folder
 	hash each file
calculate actual directory hash
compare to-be-expected directory hash with actual directory hash
on error (including mismatching hash):
	print error
	end with exit !=0verifywith-plsubcommand option (for packing lists)Theverifycommand with the-plsubcommand (or--packing_list) option can be used to verify a folder structure
with a given packing list.It is run with the path to the packing list manifest file as the parameter.$ ascmhl verify -pl /path/to/packing-list.mhlTBDThexsd-schema-checkcommandThexsd-schema-checkcommand validates a given ASC MHL Manifest file against the XML XSD. This command can be used
to ensure the creation of syntactically valid ASC MHL files, for example during  implementation of tools creating
ASC MHL files.Note: Thexsd-schema-checkcommand must be run from a directory with axsdsubfolder where the ASC MHL xsd files
are located (for example it can be run from the root folder of the ASC MHL git repository). Alternatively you can pass
the local path to the XSD file (availablehere)
with the-xsdor--xsd_fileoption.$ ascmhl xsd-schema-check /path/to/ascmhl/XXXXX.mhlxsd-schema-checkwith the-dfsubcommand optionThexsd-schema-checkcommand with the-dfsubcommand option can validates a ASC MHL Directory file instead of a
manifest file.It is run with the path to a ASC MHL Directory file.$ ascmhl xsd-schema-check -df /path/to/ascmhl/ascmhl_chain.xmlThehashcommandThehashcommand hashes an individual file with the given hash algorithm (via-hor--hash_format) and prints the hash value.$ ascmhl-debug hash --help
Usage: ascmhl-debug hash [OPTIONS] FILE_PATH

  Create and print a hash value for a file

Options:
  -h, --hash_format [md5|sha1|xxh128|xxh3|xxh64|c4]
                                  Algorithm  [required]
  --help                          Show this message and exit.Known issuesThe current state of the implementation is intended to give a good overview what can be done with ASC MHL. Nonetheless
this is not yet a complete implementation of the ASC MHL specification:Currently not all initially specified commands are implemented yet (see sections above)Renaming of files is currently not implemented (neither as command, nor proper handling in histories and packinglists)The chain file is currently not verified yetAlso see theGitHub issuespage for more.