Package: ebmlite
ebmliteREADMEebmliteis a lightweight, "pure Python" library for parsing EBML (Extensible Binary Markup Language) data. It is designed to crawl through EBML files quickly and efficiently, and that's about it.ebmlitecan also do basic EBML encoding, but more advanced EBML manipulation (e.g. with a properDOM) are beyond its scope, and are better left to other libraries.ebmliteis currently a work-in-progress. It is usable (we use it extensively), but does not (yet) implement the full EBML specification.Parts ofebmlitewere modeled afterpython-ebml, which we had previously been using, but is not a directly derivative work.ebmlitecan importpython-ebmlschemata XML (to a limited degree), but that is the extent of its cross-compatibility.EBML Overview (the short version)EBML(Extensible Binary Markup Language) is a hierarchical tagged binary format, originally created for theMatroskaproject. The hierarchical structure of EBML bears some conceptual/functional similarity to XML, although the actual structure differs significantly.In the raw, EBML elements consist of a numeric ID, the size of the element, and a payload. It is space-efficient; the lengths of the ID and size descriptors are variable, using prefix bits to indicate their lengths, a system similar to UTF-8. The mapping of IDs to names and payload data types is done via an external schema.See theofficial specificationfor more information.EBML SchemataAn EBML file is largely meaningless without a schema that defines its elements. The schema maps element IDs to names and data types; it also describes the structure (e.g. what elements can be children of other elements) and provides additional metadata.Note: ebmlite currently uses the structure for decoding only, and does not stringently enforce it.ebmliteschemata are defined in XML. From these XML files, aSchemainstance is created; within theSchemaareElementsubclasses for each element defined in the XML. Since the interpretation of an EBML file is almost entirely dependent on a schema, importing of EBML files is done through aSchemainstance.fromebmliteimportloadSchemaschema=loadSchema('mide_ide.xml')doc=schema.load('test_file.ebml')ebmliteuses its own Schema definition syntax; it can also import python-ebml schemata. It does not (currently) use theofficial schema format.Here is an example of anebmliteschema, showing a simplified version of the definition of the standard EBML header elements:<?xml version="1.0" encoding="utf-8"?><Schema><MasterElementname="EBML"id="0x1A45DFA3"mandatory="1"multiple="0"><UIntegerElementname="EBMLVersion"id="0x4286"multiple="0"mandatory="1"/><UIntegerElementname="EBMLReadVersion"id="0x42F7"multiple="0"mandatory="1"/><UIntegerElementname="EBMLMaxIDLength"id="0x42F2"multiple="0"mandatory="1"/><UIntegerElementname="EBMLMaxSizeLength"id="0x42F3"multiple="0"mandatory="1"/><StringElementname="DocType"id="0x4282"multiple="0"mandatory="1"/><UIntegerElementname="DocTypeVersion"id="0x4287"multiple="0"mandatory="1"/><BinaryElementname="Void"global="1"id="0xEC"multiple="1"/><BinaryElementname="CRC-32"global="1"id="0xBF"multiple="0"/><MasterElementname="SignatureSlot"global="1"id="0x1B538667"multiple="1"><UIntegerElementname="SignatureAlgo"id="0x7E8A"multiple="0"/><UIntegerElementname="SignatureHash"id="0x7E9A"multiple="0"/><BinaryElementname="SignaturePublicKey"id="0x7EA5"multiple="0"/><BinaryElementname="Signature"id="0x7EB5"multiple="0"/><MasterElementname="SignatureElements"id="0x7E5B"multiple="0"><MasterElementname="SignatureElementList"id="0x7E7B"multiple="1"><BinaryElementname="SignedElement"id="0x6532"multiple="1"/></MasterElement></MasterElement></MasterElement></MasterElement><!-- More definitions would follow... --></Schema>Each element defined in the schema is a subclass of one of 8 Element base classes:MasterElement:An element containing other elements.IntegerElement:Contains a signed integer value of variable length.UIntegerElement:Contains an unsigned integer value of variable length.FloatElement:Contains a 32 or 64 bit floating point value.StringElement:Contains printable US-ASCII characters (0x20 to 0x7E).UnicodeElement:Contains UTF-8 string data.DateElement:Contains a timestamp, stored as nanoseconds since 2001-01-01T00:00:00 UTC as a 64 bit integer.ebmliteautomatically translates this into a Pythondatetime.datetimeobject.BinaryElement:Contains binary data.Element definitions have several attributes:name(string): The Element subclass' name.id(integer): The Element subclass' EBML ID.global(bool, optional): If "true" (e.g.1orTrue), the element may
appear in any location in an EBML file, not just where it appears in the
schema. This is equivalent to adepthof-1in apython-ebmlschemalength(integer, optional): A fixed size to use when encoding the element, overriding the EBML variable length encoding. Use to create byte-aligned structures.multiple(bool, optional, default=1): Indicates that the element can appear more than once within the same parent.Currently partially enforced for encoding.mandatory(bool, optional, default=0): Indicates that the elementmustbe present.Not currently enforced.precache(bool, optional, default varies by type): Indicates that the element's value should be read and cached when the element is parsed, rather than 'lazy-loaded' when explicitly accessed. Can be used to reduce the number of seeks when working with an EBML file after it has been imported. Simple numeric element types have this enabled by default; master, binary, and string/Unicode elements do not.There are two additional, special-case Element subclasses which are not subclassed:UnknownElement:Instantiated for elements with IDs that do not appear in the schema. Its payload is treated as binary data. The UnknownElement itself does not appear in the Schema. Unlike other Element subclasses, its ID can vary from instance to instance.VoidElement:"Void" (ID0xEC) is a standard EBML element, typically used for padding. If the Schema defines the Void element, it is replaced by this special-case element. The contents of its payload are ignored.The structure of the schema's XML defines the structure of the EBML document; children of a MasterElement in the schema are valid child element types in the EBML. An Element type can appear multiple times in a schema; i.e. if its type can appear as a child of different parent types. Only the first definition requires bothnameandidattributes. Successive definitions can be abbreviated to just thenameand/orid; they will inherit all the other attributes of the first definition. Successive definitions mustnothave contradictory attributes, however.<Schema><MasterElementname="Parent1"id="0x5210"><!-- first definition of child: has all attributes --><IntegerElementname="SharedChild"id="0x5211"precache="1"length="8"/></MasterElement><!-- Proper reuse of a child element --><MasterElementname="Parent2"id="0x5220"><!-- second definition of child: only name (preferred) or ID required --><IntegerElementname="SharedChild"/></MasterElement><MasterElementname="Parent3"id="0x5230"><!-- third definition of child: only name (preferred) or ID required --><IntegerElementid="0x5211"/></MasterElement><!-- A bad reuse! This will raise an exception when the schema is parsed. --><MasterElementname="Parent3"id="0x5230"><!-- BAD REDEFINITION: attribute(s) contradict initial definition! --><IntegerElementname="SharedChild"id="0xBAD1D"/></MasterElement></Schema>Note:As seen in the example above,ebmliteallows an EBML document to have multiple elements at its root level. Several other EBML libraries do this as well, but this is apparently counter to the official spec. Officially, an EBML document should have only a single root element, similar to an XML file.Using Schema FilesSchema File Location (ebmlite.SCHEMA_PATH)ebmlite.SCHEMA_PATHis a list that stores a set of paths which will be searched for schema files, similar tosys.pathworks for modules. If a schema filename with no path is used (e.g.ebmlite.loadSchema("matroska.xml")),
it is searched for inSCHEMA_PATH's paths. Users may modifySCHEMA_PATHas needed.The default schemata XML files are in the package'sschematasubdirectory.Module-relative PathsSince multiple packages are currently usingebmlite, schemata may be imported using module names. Modules
can be specified in paths by using braces (curly brackets) around their names (e.g.,"{idelib}/schemata/mide_ide.xml").
Module-relative names may be used when loading schemata, can be included inebmlite.SCHEMA_PATH, and can be
used with the command-line utilities (in quotes).New to version 3.3.TheEBMLITE_SCHEMA_PATHEnvironment VariableAn operating system environment variable may be defined as a global means of specifying schema paths, in and out
of Python.EBMLITE_SCHEMA_PATHfunctions like thePATHenvironment variable in Windows.EBMLITE_SCHEMA_PATHcontains one or more paths, which will be added toebmlite.SCHEMA_PATH; multiple paths are delimited by;in
Windows,:in *NIX operating systems (Linux, macOS, etc.).EBMLITE_SCHEMA_PATHis largely intended for use
with theebmlitecommand-line utilities.New to version 3.3.ebmliteSchemaTheSchemaclass is a factory used to encode and decode EBML files.  When it's initialized, it scans through the schema file and creates a new class for each element present in the file; then, when encoding or decoding files, it references these classes in order to encapsulate everything safely.DocumentsDocumentsare subclasses of MasterElements, which act as an interface to EBML files and act as the root node of the EBML tree.  EachSchemaalso creates aDocumentsubclass to use, and the baseDocumentclass will not function without class variables defined by theSchema.UtilitiesThe functions provided by util.py will expose the majority of functionality needed to users, without the need to interface too deeply with this library.  The following functions are provided:util.toXml(el, [parent=None,] [offsets=True,] [sizes=True,] [types=True,] [ids=True]):Recursively converts EBML elements into xml elements.Argumentel: an EBML element or document.Optional argumentparent: The resulting XML element's parent element, if any.Optional argumentoffsets: IfTrue, create anoffsetattributes for
each generated XML element, containing the corresponding EBML element's
offset.Optional argumentsizes: IfTrue, createsizeattributes containing the
corresponding EBML element's size.Optional argumenttypes: IfTrue, createtypeattributes containing the
name of the corresponding EBML element type.Optional argumentids: IfTrue, createidattributes containing the
corresponding EBML element's EBML ID.Returns the root of an XML tree created using the xml.etree.ElementTree
built-in class.util.xmlElement2ebml(xmlEl, ebmlFile, schema, [sizeLength=4,] [unknown=True]):Recursively converts XML elements tonight into EBML elements.ArgumentxmlEl: The XML element. Its tag must match an element defined in theschema.ArgumentebmlFile: An open file-like stream, to which the EBML data will be
written.Argumentschema: Anebmlite.core.Schemainstance to use when writing the
EBML document.Optional argumentsizeLength:Optional argumentunknown: IfTrue, unknown element names will be allowed,
provided their XML elements include anidattribute with the EBML
ID (in hexadecimal).Returns the length of the encoded element, including header and children.RaisesNameError: raised if an xml element is not present in the schema and
unknown is False, OR if the xml element does not have an ID.util.xml2ebml(xmlFile, ebmlFile, schema, [sizeLength=4,] [headers=True,] [unknown=True]):
ArgumentxmlFile: The XML source. Can be a filename, an open file-like
stream, or a parsed XML document.ArgumentebmlFile: The EBML file to write. Can be a filename or an open
file-like stream.Argumentschema: The EBML schema to use. Can be a filename or an instance of
aSchema.Optional argumentsizeLength: The default length of each element's size
descriptor. Must be large enough to store the largest 'master' element.
If an XML element has asizeLengthattribute, it will override
this.Optional argumentheaders: IfTrue, generate the standardEBMLEBML
element if the XML document does not contain one.Optional argumentunknown: IfTrue, unknown element names will be allowed,
provided their XML elements include anidattribute with the EBML
ID (in hexadecimal).Returns the size of the ebml file in bytes.Raises NameError: raises if an xml element is not present in the schema.util.loadXml(xmlFile, schema, [ebmlFile=None]):Helpful utility to load an EBML document from an XML file.ArgumentxmlFile: The XML source. Can be a filename, an open file-like
stream, or a parsed XML document.Argumentschema: The EBML schema to use. Can be a filename or an instance of
aSchema.Optional ArgumentebmlFile: The name of the temporary EBML file to write, or:memory:to use RAM (likesqlite3). Defaults to an
automatically-generated temporary file.Returns the root node of the specified EBML fileutil.pprint:Test function to recursively crawl an EBML document or element and print its
structure, with child elements shown indented.Argumentel: An instance of aDocumentorElementsubclass.Argumentvalues: IfTrue, show elements' values.Optional Argumentout: A file-like stream to which to write.Optional argumentindent: The string containing the character(s) used for each
indentation.Command Line UtilitiesWhenebmliteis installed as a Python library, the utilities can be called from the command line.
From the command line, documentation can be viewed using one of the following:python -m ebmlite.tools.ebml2xml -h
python -m ebmlite.tools.xml2ebml -h
python -m ebmlite.tools.view_ebml -hThe commands available are:ebml2xmlpython -m ebmlite.tools.ebml2xml <EBML file> <schema> -o <file.XML>ebml2xmlwill translate an EBML file into XML. For example:python -m ebmlite.tools.ebml2xml DAQ11093_000001.ide mide_ide.xml -o DAQ11093_000001.xmlwill translate the EBML fileDAQ11093_000001.ide(an enDAQ data recorder file) into XML,
and write the result intoDAQ11093_000001.xml. The schemamide_ide.xmlis built in to
the EBMLite library.xml2ebmlpython -m ebmlite.tools.xml2ebml <file.XML> <schema> -o <EBML file>xml2ebmlwill translate XML back in to EBML. For examplepython -m ebmlite.tools.xml2ebml DAQ11093_000001.xml mide_ide.xml -o DAQ11093_000001b.ideWill turnDAQ11093_000001.xmlback into an IDE file.view_ebmlpython -m ebmlite.tools.view_ebml <EBML file> <schema>view_ebmlwill show summary element data about an EBML file, including element ID and typelist_schematapython -m ebmlite.tools.list_schematalist_schematawill list allebmliteschemata XML files in the directories specified inebmlite.SCHEMA_PATH(and theEBMLITE_SCHEMA_PATHOS environment variable, if defined). The resulting list displays the base filename
of the schema, followed by the file's full path, as well as the full paths of any schemata in other
directories/modules that share the base name. If the schema's base name is used without a path, the first file
will be loaded.New to version 3.3.To DoComplete documentation and example code.Seetodoitems in the Python files (i.e.core.py).