Package: pyfury
Fury is a blazing-fast multi-language serialization framework powered byjit(just-in-time compilation)andzero-copy, providing up to 170x performance and ultimate ease of use.https://furyio.orgFeaturesMultiple languages: Java/Python/C++/Golang/JavaScript/Rust/Scala/TypeScript.Zero-copy: Cross-language out-of-band serialization inspired
bypickle5and off-heap read/write.High performance: A highly-extensible JIT framework to generate serializer code at runtime in an async multi-thread way to speed serialization, providing 20-170x speed up by:reduce memory access by inlining variables in generated code.reduce virtual method invocation by inline call in generated code.reduce conditional branching.reduce hash lookup.Multiple binary protocols: Object graph, row format, and so on.In addition to cross-language serialization, Fury also features at:Drop-in replace Java serialization frameworks such as JDK/Kryo/Hessian without modifying any code, but 100x faster.
It can greatly improve the efficiency of high-performance RPC calls, data transfer, and object persistence.100% compatiblewith JDK serialization API with much faster implementation: supporting JDKwriteObject/readObject/writeReplace/readResolve/readObjectNoData/ExternalizableAPI.SupportsJava 8~21, Java 17+recordis supported too.SupportAOT compilation serializationforGraalVM native image, and no reflection/serialization json config are needed.Supports shared and circular reference object serialization for golang.Supportsscala serializationSupports automatic object serialization for golang.ProtocolsDifferent scenarios have different serialization requirements. Fury designed and implemented
multiple binary protocols for those requirements:Cross-language object graph protocol:Cross-language serialize any object automatically, no need for IDL definition, schema compilation and object to/from protocol
conversion.Support shared reference and circular reference, no duplicate data or recursion error.Support object polymorphism.Native java/python object graph protocol: Highly-optimized based on type system of the language.Row format protocol: A cache-friendly binary random access format, supports skipping serialization and partial serialization,
and can convert to column-format automatically.New protocols can be easily added based on fury existing buffer, encoding, meta, codegen and other capabilities. All of those share the same codebase, and the optimization for one protocol
can be reused by another protocol.BenchmarksDifferent serialization frameworks are suitable for different scenarios, and benchmark results here are for reference only.If you need to benchmark for your specific scenario, make sure all serialization frameworks are appropriately configured for that scenario.Dynamic serialization frameworks support polymorphism and references, but they often come with a higher cost compared to static serialization frameworks, unless they utilize JIT techniques like Fury does.
Because Fury generates code at runtime, it is recommended towarm upthe system before collecting benchmark statistics.Java SerializationTitle containing "compatible" represent schema compatible mode: support type forward/backward compatibility.Title without "compatible" represent schema consistent mode: class schema must be the same between serialization and deserialization.Structis a class with100 primitive fields,MediaContentis a class fromjvm-serializers,Sampleis a class fromkryo benchmark.Seebenchmarksfor more benchmarks about type forward/backward compatibility, off-heap support, zero-copy serialization.InstallationJavaNightly snapshot:<repositories><repository><id>sonatype</id><url>https://s01.oss.sonatype.org/content/repositories/snapshots</url><releases><enabled>false</enabled></releases><snapshots><enabled>true</enabled></snapshots></repository></repositories><dependency><groupId>org.furyio</groupId><artifactId>fury-core</artifactId><version>0.5.0-SNAPSHOT</version></dependency><!-- row/arrow format support --><!-- <dependency><groupId>org.furyio</groupId><artifactId>fury-format</artifactId><version>0.5.0-SNAPSHOT</version></dependency> -->Release version:<dependency><groupId>org.furyio</groupId><artifactId>fury-core</artifactId><version>0.4.1</version></dependency><!-- row/arrow format support --><!-- <dependency><groupId>org.furyio</groupId><artifactId>fury-format</artifactId><version>0.4.1</version></dependency> -->ScalalibraryDependencies += "org.furyio" % "fury-core" % "0.4.1"Python# Release version will be provided in the future.pipinstallpyfury--preJavaScriptnpminstall@furyjs/furyGolanggogetgithub.com/alipay/fury/go/furyQuickstartHere we give a quick start about how to use fury, seeuser guidefor more details aboutjava,cross language, androw format.Fury java object graph serializationIf you don't have cross-language requirements, using this mode will
have better performance.importio.fury.*;importio.fury.config.*;importjava.util.*;publicclassExample{publicstaticvoidmain(String[]args){SomeClassobject=newSomeClass();// Note that Fury instances should be reused between// multiple serializations of different objects.{Furyfury=Fury.builder().withLanguage(Language.JAVA)// Allow to deserialize objects unknown types, more flexible// but may be insecure if the classes contains malicious code..requireClassRegistration(false).build();// Registering types can reduce class name serialization overhead, but not mandatory.// If class registration enabled, all custom types must be registered.fury.register(SomeClass.class);byte[]bytes=fury.serialize(object);System.out.println(fury.deserialize(bytes));}{ThreadSafeFuryfury=Fury.builder().withLanguage(Language.JAVA)// Allow to deserialize objects unknown types, more flexible// but may be insecure if the classes contains malicious code..requireClassRegistration(false).buildThreadSafeFury();byte[]bytes=fury.serialize(object);System.out.println(fury.deserialize(bytes));}{ThreadSafeFuryfury=newThreadLocalFury(classLoader->{Furyf=Fury.builder().withLanguage(Language.JAVA).withClassLoader(classLoader).build();f.register(SomeClass.class);returnf;});byte[]bytes=fury.serialize(object);System.out.println(fury.deserialize(bytes));}}}Cross-language object graph serializationJavaimportio.fury.*;importio.fury.config.*;importjava.util.*;publicclassReferenceExample{publicstaticclassSomeClass{SomeClassf1;Map<String,String>f2;Map<String,String>f3;}publicstaticObjectcreateObject(){SomeClassobj=newSomeClass();obj.f1=obj;obj.f2=ofHashMap("k1","v1","k2","v2");obj.f3=obj.f2;returnobj;}// mvn exec:java -Dexec.mainClass="io.fury.examples.ReferenceExample"publicstaticvoidmain(String[]args){Furyfury=Fury.builder().withLanguage(Language.XLANG).withRefTracking(true).build();fury.register(SomeClass.class,"example.SomeClass");byte[]bytes=fury.serialize(createObject());// bytes can be data serialized by other languages.System.out.println(fury.deserialize(bytes));}}PythonfromtypingimportDictimportpyfuryclassSomeClass:f1:"SomeClass"f2:Dict[str,str]f3:Dict[str,str]fury=pyfury.Fury(ref_tracking=True)fury.register_class(SomeClass,type_tag="example.SomeClass")obj=SomeClass()obj.f2={"k1":"v1","k2":"v2"}obj.f1,obj.f3=obj,obj.f2data=fury.serialize(obj)# bytes can be data serialized by other languages.print(fury.deserialize(data))Golangpackagemainimportfurygo"github.com/alipay/fury/go/fury"import"fmt"funcmain(){typeSomeClassstruct{F1*SomeClassF2map[string]stringF3map[string]string}fury:=furygo.NewFury(true)iferr:=fury.RegisterTagType("example.SomeClass",SomeClass{});err!=nil{panic(err)}value:=&SomeClass{F2:map[string]string{"k1":"v1","k2":"v2"}}value.F3=value.F2value.F1=valuebytes,err:=fury.Marshal(value)iferr!=nil{}varnewValueinterface{}// bytes can be data serialized by other languages.iferr:=fury.Unmarshal(bytes,&newValue);err!=nil{panic(err)}fmt.Println(newValue)}Row formatJavapublicclassBar{Stringf1;List<Long>f2;}publicclassFoo{intf1;List<Integer>f2;Map<String,Integer>f3;List<Bar>f4;}RowEncoder<Foo>encoder=Encoders.bean(Foo.class);Foofoo=newFoo();foo.f1=10;foo.f2=IntStream.range(0,1000000).boxed().collect(Collectors.toList());foo.f3=IntStream.range(0,1000000).boxed().collect(Collectors.toMap(i->"k"+i,i->i));List<Bar>bars=newArrayList<>(1000000);for(inti=0;i<1000000;i++){Barbar=newBar();bar.f1="s"+i;bar.f2=LongStream.range(0,10).boxed().collect(Collectors.toList());bars.add(bar);}foo.f4=bars;// Can be zero-copy read by pythonBinaryRowbinaryRow=encoder.toRow(foo);// can be data from pythonFoonewFoo=encoder.fromRow(binaryRow);// zero-copy read List<Integer> f2BinaryArraybinaryArray2=binaryRow.getArray(1);// zero-copy read List<Bar> f4BinaryArraybinaryArray4=binaryRow.getArray(3);// zero-copy read 11th element of `readList<Bar> f4`BinaryRowbarStruct=binaryArray4.getStruct(10);// zero-copy read 6th of f2 of 11th element of `readList<Bar> f4`barStruct.getArray(1).getLong(5);RowEncoder<Bar>barEncoder=Encoders.bean(Bar.class);// deserialize part of data.BarnewBar=barEncoder.fromRow(barStruct);BarnewBar2=barEncoder.fromRow(binaryArray4.getStruct(20));Python@dataclassclassBar:f1:strf2:List[pa.int64]@dataclassclassFoo:f1:pa.int32f2:List[pa.int32]f3:Dict[str,pa.int32]f4:List[Bar]encoder=pyfury.encoder(Foo)foo=Foo(f1=10,f2=list(range(1000_000)),f3={f"k{i}":iforiinrange(1000_000)},f4=[Bar(f1=f"s{i}",f2=list(range(10)))foriinrange(1000_000)])binary:bytes=encoder.to_row(foo).to_bytes()foo_row=pyfury.RowData(encoder.schema,binary)print(foo_row.f2[100000],foo_row.f4[100000].f1,foo_row.f4[200000].f2[5])CompatibilitySchema CompatibilityFury java object graph serialization support class schema forward/backward compatibility. The serialization peer and deserialization peer can add/delete fields independently.We plan to add support cross-language serialization aftermeta compressionis finished.Binary CompatibilityWe are still improving our protocols, binary compatibility is not ensured between fury major releases for now.
it's ensured between minor versions only. Pleaseversioningyour data by fury major version if you will upgrade fury in the future, seehow to upgrade furyfor further details.Binary compatibility will be ensured when fury 1.0 is released.SecurityStatic serialization is secure. But dynamic serialization such as fury java/python native serialization supports deserializing unregistered types, which provides more dynamics and flexibility, but also introduce security risks.For example, the deserialization may invokeinitconstructor orequals/hashCodemethod, if the method body contains malicious code, the system will be at risk.Fury provides a class registration option that is enabled by default for such protocols, allowing only deserialization of trusted registered types or built-in types.Do not disable class registration unless you can ensure your environment is secure.If this option is disabled, you are responsible for serialization security. You can configureio.fury.resolver.ClassCheckerbyClassResolver#setClassCheckerto control which classes are allowed for serialization.How to BuildPlease read theBUILDguide for instructions on how to build.How to ContributePlease read theCONTRIBUTINGguide for instructions on how to contribute.For ecosystem projects, please seehttps://github.com/fury-projectLicenseLicensed under theApache License, Version 2.0