Package: tlv8
Type-Length-Value8 (TLV8) for pythonType-Length-Value (TLV) are used to encode arbitrary data. In this case the type and length are represented by 1 byte each. Hence the name TLV8.A TLV8 entry consists of the following parts:thetype: this 8 bit field denotes the type of information that is represented by the data.thelength: this 8 bit field denotes the length of the data (this does not include the 2 bytes for type and length. For data longer than 255 bytes, there is a defined procedure available.thevalue: theselengthbytes represent the value of this TLV. The different types of data is represented differently:bytes: this is raw binary data and will be used as is, no further interpretation takes placetlv8: this is a specialized case ofbytesvalues. Using this instead of pure bytes enables nesting of data and creating a hierarchy.integer: integers are stored in little-endian byte order and are encoded with the minimal number of bytes possible (1, 2, 4 or 8)float: floats are stored as little-endian ieee754 numbersstring: strings are always UTF-8 encoded anddo notcontain the terminating NULL byteTLV8 entries whose content is longer than 255 bytes are split up into fragments. The type is repeated is repeated in each fragment, only the last fragment may contain less than 255 bytes. Fragments of one TLV8 entry must be continuous.Multiple TLV8 entries can be combined to create larger structures. Entries of different types can placed one after another. Entries of the same type must be separated by a TLV8 entry of a different type (and probably zero length).TLV8 entries of unknown or unwanted type are to be silently ignored.Examplessimple TLV8sEncoding of some atomic examples:an empty TLV of type 42:[42, None]will be encoded asb'\x2a\x00'.a TLV of type 2 with 2 bytes0x12, 0x34:[2, b'\x12\x34']will be encoded asb'\x02\x02\x12\x34'a TLV of type 3 that contains the TLV from above:[3, [2, b'\x12\x34']]will be encoded asb'\x03\x04\x02\x02\x12\x34'a TLV of type 4 that contains 1024:[4, 1024]will be encoded asb'\x04\0x02\x00\x04'a TLV of type 5 that contains 3.141:[4, 3.141]will be encoded asb'\x04\x04\x0a\xd7\x23\x41'a TLV of type 23 with stringHello 🌍:[23, 'Hello 🌍']will be encoded asb'\x17\x0a\x48\x65\x6c\x6c\x6f\x20\xf0\x9f\x8c\x8d'fragmented TLV8sEncoding of a fragmented TLV8 entry:an TLV of type 6 that contains 256 bytes from 0 to 255:[6, b'\x00\x01...\xfe\xff']will be encoded asb'\x06\xff\x00...\xfe\x06\x01\xff'combined TLV8sEncoding of two TLV8 Entries that follow each other in the input list:the combination of 2 TLV8 entries ([1, 123]and[2, 'Hello']) will be encoded asb'\x01\x01\x7b\x02\x05\x48\x65\x6c\x6c\x6f'sequences of TLV8s of same type:a sequence of 3 TLV8 entries of type 1 ([1, 1],[1, 2]and[1, 1]) will be encoded asb'\x01\x01\x01\xff\x00\x01\x01\x02\xff\x00\x01\x01\x03'Using in codeThere are two main use cases of this module.Create a bytes representationHere we want to have a comfortable way to create a data structure in python and to encode this structure into a bytes value.encode a simple listFor example, create a representation containing the following structure:Type: 1, Value: 23Type: 2, Value: 2345This can be code like that:importtlv8structure=[tlv8.Entry(1,23),tlv8.Entry(2,2345)]bytes_data=tlv8.encode(structure)print(bytes_data)And this will result in:b'\x01\x01\x17\x02\x02)\t'Nesting structuresRepresenting a line ([x: 10, y: 20] - [x: 30, y: 40]) between to points could be represented like:Type: 1, Value:Type: 3, Value: 10Type: 4, Value: 20Type: 2, Value:Type: 3, Value: 30Type: 4, Value: 40importtlv8structure=[tlv8.Entry(1,[tlv8.Entry(3,10),tlv8.Entry(4,10),]),tlv8.Entry(2,[tlv8.Entry(3,30),tlv8.Entry(4,40),])]bytes_data=tlv8.encode(structure)print(bytes_data)And this will result in:b'\x01\x06\x03\x01\n\x04\x01\n\x02\x06\x03\x01\x1e\x04\x01('Decode a bytes representationDecoding TLV8 entries from bytes data will return all bytes from all first level entries. This includes possible separator entries between entries of the same type.Decoding can be assisted by hinting with an expected structure. To represent the structure in pythondictobjects are used and nested. The keys of thedictobjects are the type ids of the TLV8 entries. If the id of an entry is not contained in the structure, it will be ignored.decode the simple listimporttlv8in_data=b'\x01\x01\x17\x02\x02)\t'expected_structure={1:tlv8.DataType.INTEGER,2:tlv8.DataType.INTEGER}result=tlv8.decode(in_data,expected_structure)print(tlv8.format_string(result))This will result in:[
  <1, 23>,
  <2, 2345>,
]decode nested dataimporttlv8in_data=b'\x01\x06\x03\x01\n\x04\x01\n\x02\x06\x03\x01\x1e\x04\x01('sub_struct={3:tlv8.DataType.INTEGER,4:tlv8.DataType.INTEGER}expected_structure={1:sub_struct,2:sub_struct}result=tlv8.decode(in_data,expected_structure)print(tlv8.format_string(result))This will result in:[
  <1, [
    <3, 10>,
    <4, 10>,
  ]>,
  <2, [
    <3, 30>,
    <4, 40>,
  ]>,
]Using IntEnum data during encoding and decodingUsing enumerations might increase readabilty of encode end decode processes.During encodingIt is possible to useenum.IntEnumfor encoding:importtlv8importenumclassKeys(enum.IntEnum):X=42# ...classValues(enum.IntEnum):Y=23# ...result=tlv8.encode([tlv8.Entry(Keys.X,Values.Y)])print(result)This will result in:b'*\x01\x17'During decodingAs during encoding,enum.IntEnumcan be used for keys and values during decoding:importtlv8importenumclassKeys(enum.IntEnum):X=42# ...classValues(enum.IntEnum):Y=23# ...result=tlv8.decode(b'*\x01\x17',{Keys.X:Values})print(tlv8.format_string(result))print(type(result[0].type_id),type(result[0].data))This will result in[
  <Keys.X, Values.Y>,
]
<enum 'Keys'> <enum 'Values'>So thetype_idand thedatafields are not simpleintinstance anymore but values of their enumerations. This alos helps during usingformat_stringto get a easier to read output.CodingThe module offers the following primary functions and classes.functionformat_stringThis function formats a list of TLV8 Entry objects as str. The hierarchy of the entries will be represented by increasing the indentation of the output.The parameters are:entries: a python list of tlv8.Entries objectsindent: the level of indentation to be used, this defaults to 0 and is increased on recursive calls for nested entries.The function returns astrinstance and raisesValueErrorinstances if the input is not a list oftlv8.Entryobjects.Example:importtlv8data=[tlv8.Entry(1,3.141),tlv8.Entry(2,[tlv8.Entry(3,'hello'),tlv8.Entry(4,'world'),]),tlv8.Entry(1,2)]print(tlv8.format_string(data))This will become:[
  <1, 3.141>,
  <2, [
    <3, hello>,
    <4, world>,
  ]>,
  <1, 2>,
]functionencodeFunction to encode a list oftlv8.Entryobjects into a sequence of bytes following the rules for creating TLVs. Theseparator_type_idis used for the separating entries between two entries of the same type.The parameters are:entries: a list oftlv8.Entryobjectsseparator_type_id: the 8-bit type id of the separator to be used. The default is (as defined in table 5-6, page 51 of HomeKit Accessory Protocol Specification Non-Commercial Version Release R2) 0xff.The function returns an instance ofbytes. This is empty if nothing was encoded. The function raisesValueErrorif the input parameter is not a list oftlv8.Entryobjects or a data value is not encodable. AValueErrorwill also be raised if theseparator_type_idis used astype_idin one of the entries as well.Example:importtlv8data=[tlv8.Entry(1,3.141),tlv8.Entry(2,[tlv8.Entry(3,'hello'),tlv8.Entry(4,'world')]),tlv8.Entry(1,2)]print(tlv8.encode(data))This will result in:b'\x01\x04%\x06I@\x02\x0e\x03\x05hello\x04\x05world\x01\x01\x02'functiondecodeFunction to decode abytesorbytearrayinstance into a list oftlv8.Entryinstances. This reverses the process done by theencodefunction.The parameters are:data: abytesorbytearrayinstance to be parsedexpected: a dict of type ids onto expectedtlv8.DataTypevalues. If the expected entry is again atlv8.Entrythat should be parsed, use another dict to describe the hiearchical structure. This defaults toNonewhich means not filtering will be performed but also no interpretation of the entries is done. This means they will be returned asbytessequence.strict_mode: This defaults toFalse. If set toTrue, this will raise additionalValueErrorinstances if there are possible missing separators between entries of the same type.The function returns alistinstance and raisesValueErrorinstances if the input is either not abytesobject or an invalid tlv8 structure.Example:importtlv8data=b'\x01\x04%\x06I@\x02\x0e\x03\x05hello\x04\x05world\x03\x01\x02'structure={1:tlv8.DataType.FLOAT,2:{3:tlv8.DataType.STRING,4:tlv8.DataType.STRING},3:tlv8.DataType.INTEGER}print(tlv8.decode(data,structure))This will result in:[
  <1, 3.1410000324249268>,
  <2, [
    <3, hello>,
    <4, world>,
  ]>,
  <3, 2>,
]functiondeep_decodeThis function works like thedecodefunction but tries to do it recursively. That means it decodes the first level of
a TLV8 structure first, then looks at each entry and tries to decode that as well. This is mostly meant for debugging
purposes in combination withformat_string.Example:importtlv8data=b'\x01\x01\x23\x02\x03\x04\x01\x42\x01\x01\x23'print(tlv8.deep_decode(data))This will result in:[
  <1, b'#'>,
  <2, [
    <4, b'B'>,
  ]>,
  <1, b'#'>,
]Notice:This function might misinterpret data as TLV8 data. For exampleimporttlv8data=tlv8.encode([tlv8.Entry(1,16843330),tlv8.Entry(2,b'\x01')])# here data is b'\x01\x04B\x02\x01\x01\x02\x01\x01'print(tlv8.deep_decode(data))This will result in a misinterpretation of the entry with ID 1:[
  <1, [
    <66, b'\x01\x01'>,
  ]>,
  <2, b'\x01'>,
]classDataTypeThis enumeration is used to represent the data type of atlv8.Entry.Enumeration EntryTLV8 typePython typeBYTESbytesbytes, alsobytearrayfor encodingTLV8tlv8custom classtlv8.Entryfor encoding anddictfor the expected structure during decodingINTEGERintegerintFLOATfloatfloatSTRINGstringstrAUTODETECTn/athis is used declare that a data type is not preset but will be determined by the python type of the dataclassEntryThis class represents a single entry in a TLV8 data set. The class overrides the methods__eq__,__str__and__repr__to fit the needs of the application.constructorThe constructor takes the following parameters:type_id: the type id of the entry. Must be between 0 and 255 (8-bit type id).data: the data to be stored in this entry.data_type: the data type of the entry. Defaults toDataType.AUTODETECT.length: if set, this overrides the automatic length detection. This used for integer, when there is special need to set higher byte count than the value would need.The constructor raises aValueErrorif thetype_idis not within the 8-bit range.encode() -> bytesThis function is called to encode the data stored in thisEntry. The data type of the data will be used to decide how to encode the data. It uses thetlv8.encode()function to encode nested lists oftlv8.Entryobjects.format_string() -> strThis function formats the data stored in this entry as readable string. It is mostly called bytlv8.format_string().classEntryListThis class represents a list of entries. The class overrides the methods__repr__,__eq__,__len__,__getitem__and__iter__to fit the needs of the application.constructorThe constructor takes the following parameters:data: if set, thislistoftlv8.Entryinstances is used to initialize theEntryList.The constructor raised aValueErrorif the data is either not alistor not a list oftlv8.Entryinstances.append(entry)Append thetlv8.Entryto theEntryList. It performs type checks, so onlytlv8.Entryinstances can be appended.assert_has(type_id, message)Looks for atlv8.Entryinstance withtype_idin the first level of theEntryList. If none is found, it raises anAssertionErrorwith the givenmessage. This does not iterate recursivly, because the same type id may have different meanings on different levels (and different contexts).encode(self, separator_type_id)Encodes theEntryListusing the given separator type id. This relies ontlv8.encode().by_id(type_id)Filters theEntryListand returns onlyEntryinstance whosetype_idmatch the given one. If noEntryinstances
were found it returns an empty list.first_by_id(type_id)Search theEntryListfor the firstEntrywith the giventype_id. If no suchEntrywas found, it returnsNone.