Package: canoser
A python implementation of the canonical serialization for the Libra network.Canonical serialization guarantees byte consistency when serializing an in-memory
data structure. It is useful for situations where two parties want to efficiently compare
data structures they independently maintain. It happens in consensus where
independent validators need to agree on the state they independently compute. A cryptographic
hash of the serialized data structure is what ultimately gets compared. In order for
this to work, the serialization of the same data structures must be identical when computed
by independent validators potentially running different implementations
of the same spec in different languages.InstallationRequire python 3.6 or above installed.$python3-mpipinstallcanoserUsageFirst define a data structure with Canoser, that is, write a class that inherits from "canoser.Struct", and then define the fields owned by the structure through the "_fields" array. This structure naturally has the ability to canonical serialize and deserialize types. For example, the following AccountResource defines a data structure of the same name in the Libra code：#python code，define canoser data structurefromcanoserimportStruct,Uint8,Uint64classAccountResource(Struct):_fields=[('authentication_key',[Uint8]),('balance',Uint64),('delegated_withdrawal_capability',bool),('received_events',EventHandle),('sent_events',EventHandle),('sequence_number',Uint64)]Here is the code that defines this data structure and serialization in Libra code:// rust code in Libra// define the data structurepubstructAccountResource{balance:u64,sequence_number:u64,authentication_key:ByteArray,sent_events:EventHandle,received_events:EventHandle,delegated_withdrawal_capability:bool,}// serializationimplCanonicalSerializeforAccountResource{fnserialize(&self,serializer:&mutimplCanonicalSerializer)->Result<()>{serializer.encode_struct(&self.authentication_key)?.encode_u64(self.balance)?.encode_bool(self.delegated_withdrawal_capability)?.encode_struct(self.received_events)?.encode_struct(self.sent_events)?.encode_u64(self.sequence_number)?;Ok(())}}In the rust language used by Libra, it is necessary to manually write code to serialize/deserialize the data structure, and the order of the fields in the data structure and the order of serialization are not necessarily the same.In Canoser, after defining the data structure, you don't need to write code to implement serialization and deserialization. Note that the order of the data structures in Canoser is defined in the order in which they are serialized in Libra.Supported field typesfield typeoptionl sub typedescriptioncanoser.Uint8Unsigned 8-bit integercanoser.Uint16Unsigned 16-bit integercanoser.Uint32Unsigned 32-bit integercanoser.Uint64Unsigned 64-bit integercanoser.Uint128Unsigned 128-bit integercanoser.Int8Signed 8-bit integercanoser.Int16Signed 16-bit integercanoser.Int32Signed 32-bit integercanoser.Int64Signed 64-bit integercanoser.Int128Signed 128-bit integerboolBooleanstrStringbytesBinary String[]supportedArray Type{}supportedMap Type()supportedTuple Typecanoser.StructAnother structure nested (cannot be recycled)RustEnumEnum typeRustOptionalOptional typeAbout Array TypeThe default data type (if not defined) in the array is Uint8. The following two definitions are equivalent:classArr1(Struct):_fields=[(addr,[])]classArr2(Struct):_fields=[(addr,[Uint8])]Arrays can also define lengths to represent fixed length data. For example, the address in Libra is 256 bits, which is 32 bytes, so it can be defined as follows:classAddress(Struct):_fields=[(addr,[Uint8,32])]When the fixed length data is serialized, you can skip the length information written to the output. For example, following code will generate 32 bytes without writing the length of the addr during serialization.classAddress(Struct):_fields=[(addr,[Uint8,32,False])]About map typeThe default data type (if not defined) in the map is an array of Uint8 in Libra, both of key and value.
But the python language dosn't support the array data type to be the key of a dict, so we change the key type from array of Uint8 to bytes in python, the type of value is unchanged.
The following two definitions are equivalent:classMap1(Struct):_fields=[(addr,{})]classMap2(Struct):_fields=[(addr,{bytes:[Uint8]})]About enum typeIn python and C, enum is just enumerated constants. But in Libra(Rust), a enum has a type constant and a optional Value. A rust enumeration is typically represented as:enum SomeDataType {
  type0(u32),
  type1(u64),
  type2
}To define a enum with Canoser, first write a class that inherits from "canoser.RustEnum", and then define the types owned by the enum through the "_enums" array.
For example, the following TransactionArgument defines a data structure of the same name in the Libra code. The argument of a transaction can be one of the four types: Uint64 or Address or String or IntArray.：classTransactionArgument(RustEnum):_enums=[('U64',Uint64),('Address',[Uint8,ADDRESS_LENGTH]),('String',str),('ByteArray',[Uint8])]You can instantiate an enum obj and call its method and properties like this:arg2=TransactionArgument('String','abc')assertarg2.index==2assertarg2.value=='abc'assertarg2.String==Trueassertarg2.U64==FalseEvery RustEnum object has anindexproperty and avalueproperty. After instantiated, theindexcan't be modified. You can only modify thevalueof an enum with correct data type.For example, the following code is valid:arg2=TransactionArgument('String','abc')arg2.value=='Bcd'For example, the following code is invalid:arg2.index=0#raise an exceptionarg2.value=[3]#raise an exceptionThe RustEnum can have a enum without value type, which represented byNone.classEnum1(RustEnum):_enums=[('opt1',[Uint8]),('opt2',None)]e2=Enum1('opt2',None)#ore2=Enum1('opt2')You can also instantiate a RustEnum object by index and value.e1=Enum1.new(0,[5])# which is equivalent toe1=Enum1('opt1',[5])About optional typeAn optional type in libra is a nullable data either exists in its full representation or does not. For example,optional_data: Option(uint8); // Rust/Libra
uint8 *optional_data; // CIt has similar semantics meaning with the following enum type:enum Option<uint8> {
    Some(uint8),
    None,
}To define a optional with Canoser, first write a class that inherits from "canoser.RustOptional", and then define the types owned by RustOptional through the "_type" field. For example,classOptionUInt(RustOptional):_type=Uint8null=OptionUInt(None)obj=OptionUInt(8)assertobj.value==8Here's a complete example:classOptionStr(RustOptional):_type=strclassOptionTest(Struct):_fields=[('message',OptionStr)]def__init__(self,msg=None):ifmsgisnotNone:self.message=OptionStr(msg)else:self.message=OptionStr(None)test=OptionTest('test_str')asserttest.message.value=='test_str'The RustOptional type in canoser is similar totyping.Optionalin python. Note that this is not the same concept as an optional argument, which is one that has a default.Nested data structureThe following is a complex example with three data structures:classAddr(Struct):_fields=[('addr',[Uint8,32])]classBar(Struct):_fields=[('a',Uint64),('b',[Uint8]),('c',Addr),('d',Uint32),]classFoo(Struct):_fields=[('a',Uint64),('b',[Uint8]),('c',Bar),('d',bool),('e',{}),]This example refers to the test code from canonical serialization in libra.Serialization and deserializationAfter defining canoser.Struct, you don't need to implement serialization and deserialization code yourself, you can directly call the default implementation of the base class. Take the AccountResource structure as an example:# serialize an objectobj=AccountResource(authentication_key=...,...)bbytes=obj.serialize()# deserialize an object from bytesobj=AccountResource.deserialize(bbytes)Json pretty printFor any canoserStruct, you can call theto_jsonmethod to get a formatted json string:# serialize an objectprint(obj.to_json())Get field value from objectFor all fields defined by the "_fields", the value of this field of an object can be obtained via field_name. such as:obj.authentication_keyRust Type AliasFor simple type alias in rust, such as:// in rustpubtypeRound=u64;We can define the alias like this:# in pythonRound=Uint64Rust Tuple StructStruct like Address and ByteArray has no fields:pubstructAddress([u8;ADDRESS_LENGTH]);pubstructByteArray(Vec<u8>);These struct calledtuple structinRustprogramming language. Tuple struct is liketypedefother than struct inClike programming languages.You can just define them as a direct type, no struct. Just code like this:classTransactionArgument(RustEnum):_enums=[...('Address',[Uint8,ADDRESS_LENGTH]),...]Or you can define anAddressclass which inherit fromcanoser.DelegateTand has adelegate_typefield with type[Uint8, ADDRESS_LENGTH]:classAddress(DelegateT):delegate_type=[Uint8,ADDRESS_LENGTH]classTransactionArgument(RustEnum):_enums=[...('Address',Address),...]Do not instantiate acanoser.DelegateTtype in assaignment, for example:transactionArgument.address=[...]#oktransactionArgument.address=Address([...])#errorNoticeMust define canoser struct by serialized fields and sequence, not the definition in the rust struct.For example, the SignedTransaction in Libra is defined as following code:pubstructSignedTransaction{raw_txn:RawTransaction,public_key:Ed25519PublicKey,signature:Ed25519Signature,transaction_length:usize,}But fieldtransaction_lengthdoesn't write to the output.implCanonicalSerializeforSignedTransaction{fnserialize(&self,serializer:&mutimplCanonicalSerializer)->Result<()>{serializer.encode_struct(&self.raw_txn)?.encode_bytes(&self.public_key.to_bytes())?.encode_bytes(&self.signature.to_bytes())?;Ok(())}}So we define SignedTransaction in canoser as following code:classSignedTransaction(canoser.Struct):_fields=[('raw_txn',RawTransaction),('public_key',[Uint8,ED25519_PUBLIC_KEY_LENGTH]),('signature',[Uint8,ED25519_SIGNATURE_LENGTH])]Here is another example. The definition sequence and serialize sequence is opposite inWriteOppubenumWriteOp{Value(Vec<u8>),Deletion,}enumWriteOpType{Deletion=0,Value=1,}implCanonicalSerializeforWriteOp{fnserialize(&self,serializer:&mutimplCanonicalSerializer)->Result<()>{matchself{WriteOp::Deletion=>serializer.encode_u32(WriteOpType::Deletionasu32)?,WriteOp::Value(value)=>{serializer.encode_u32(WriteOpType::Valueasu32)?;serializer.encode_vec(value)?}};Ok(())}}So, we defineWriteOpas follow:classWriteOp(RustEnum):_enums=[('Deletion',None),('Value',[Uint8])]Related ProjectsMoveOnLibra OpenAPI: make writing libra wallet & move program easierA Ruby implementation of the LCS(Libra Canonical Serialization)A Python implementation of client APIs and command-line tools for the Libra networkLicenseThe package is available as open source under the terms of theMIT License.