Package: dataclasses-tensor
Dataclasses TensorThe library provides a simple API for encoding and decoding Pythondataclassesto and from tensors (PyTorch tensors or NumPy arrays) based ontypingannotations.Heavily inspired bydataclasses-jsonpackage.Installpipinstalldataclasses-tensorQuickstartTensor representation for a game state in Chess:fromdataclassesimportdataclass,fieldfromenumimportEnumfromtypingimportOptional,Listfromdataclasses_tensorimportdataclass_tensor,configclassPlayer(Enum):WHITE=0BLACK=1classPieceType(Enum):PAWN=0BISHOP=1KNIGHT=2ROOK=3QUEEN=4KING=5@dataclassclassPiece:piece_type:PieceTypeowner:Player@dataclass_tensor@dataclassclassChess:num_moves:floatnext_move:Playerboard:List[Optional[Piece]]=field(metadata=config(shape=(64,)))Working with tensors:>>>state=Chess(100.,next_move=Player.WHITE,board=[Piece(PieceType.KING,Player.BLACK)])>>>t1=state.to_numpy()array([100.,1.,0.,0.,0.,0.,0.,0.,0.,1.,0.,1.,1.,0.,0.,0.,0.,0.,0.,0.,0.,1.,...>>>t1.shape(579,)>>>Chess.from_numpy(t1)Chess(num_moves=100.,next_move=<Player.WHITE:0>,board=[Piece(piece_type=<PieceType.KING:5>,owner=<Player.BLACK:1>),...])TypesData ClassesThe library uses type annotations to determine appropriate encoding layout. Data class member variables serialized sequentially. See supported types listed below.Primitives (int, float, bool)The library supports numerical primitives (int,float) andbool. Strings and byte arrays are not supported.Warning: be careful with tensordtypeas an implicit type conversion could potentially lead to losing information (for example, writingfloatintoint32tensor and reading it back won't produce expected result).EnumsPythonEnumsare encoded using one-hot encoding.>>>fromdataclasses_tensorimportdataclass_tensor>>>fromdataclassesimportdataclass>>>fromenumimportEnum>>>>>>classMatrix(Enum):...THE_MATRIX=1...RELOADED=2...REVOLUTIONS=3...>>>@dataclass_tensor...@dataclass...classWatchList:...matrix:Matrix...>>>WatchList(Matrix.RELOADED).to_numpy()array([0.,0.,1.])>>>WatchList.from_numpy(_)WatchList(matrix=<Matrix.RELOADED:2>)Optionaltyping.Optionaltype is encoded using additional dimension prior to the main datatype.>>>fromtypingimportOptional>>>>>>@dataclass_tensor...@dataclass...classMaybeWatchList:...matrix:Optional[Matrix]>>>>>>MaybeWatchList(Matrix.RELOADED).to_numpy()array([0.,0.,1.,0.])>>>MaybeWatchList.from_numpy([0.,0.,1.,0.])MaybeWatchList(matrix=<Matrix.RELOADED:2>)>>>MaybeWatchList.from_numpy([1.,0.,0.,0.])MaybeWatchList(matrix=None)The layout described forOptional[Enum]is consistent with havingNoneas additional option into enumeration.ArraysArrays, defined either usingtyping.Listor[](supported in Python3.9+), require size to be statically provided. See example:>>>fromtypingimportList>>>fromdataclasses_tensorimportconfig>>>@dataclass_tensor...@dataclass...classMultipleWatchList:...matrices:List[Matrix]=field(metadata=config(shape=(2,)))>>>>>>MultipleWatchList([Matrix.THE_MATRIX,Matrix.RELOADED]).to_numpy()array([1.,0.,0.,0.,1.,0.])>>>MultipleWatchList.from_numpy([1.,0.,0.,0.,1.,0.])MultipleWatchList(matrices=[<Matrix.THE_MATRIX:1>,<Matrix.RELOADED:2>])Nested lists are supported, note multidimensionalshapeconfiguration:>>>@dataclass_tensor...@dataclass...classMultipleWatchList:...matrices:List[List[Matrix]]=field(metadata=config(shape=(1,2)))>>>>>>MultipleWatchList([[Matrix.THE_MATRIX,Matrix.RELOADED]]).to_numpy()array([1.,0.,0.,0.,1.,0.])>>>MultipleWatchList.from_numpy([1.,0.,0.,0.,1.,0.])MultipleWatchList(matrices=[[<Matrix.THE_MATRIX:1>,<Matrix.RELOADED:2>]])IfListargument isOptional, the list is automatically padded to the right shape withNones.>>>@dataclass_tensor...@dataclass...classMaybeMultipleWatchList:...matrices:List[Optional[Matrix]]=field(metadata=config(shape=(3,)))>>>>>>MaybeMultipleWatchList([Matrix.THE_MATRIX,Matrix.RELOADED]).to_numpy()array([0.,1.,0.,0.,0.,0.,1.,0.,1.,0.,0.,0.])>>>MaybeMultipleWatchList.from_numpy([0.,1.,0.,0.,0.,0.,1.,0.,1.,0.,0.,0.])MaybeMultipleWatchList(matrices=[<Matrix.THE_MATRIX:1>,<Matrix.RELOADED:2>,None])Uniontyping.Unionis encoded by allocating one-hot tensor to determine which option from the union is given following by corresponding layouts for all options.>>>fromtypingimportUnion>>>>>>classBatman(Enum):...BEGINS=1...DARK_KNIGHT=2...DARK_KINGHT_RISES=3...>>>@dataclass_tensor...@dataclass...classWatchList:...next_movie:Union[Matrix,Batman]...>>>WatchList(Matrix.RELOADED).to_numpy()array([1.,0.,0.,1.,0.,0.,0.,0.])>>>WatchList.from_numpy(_)WatchList(next_movie=<Matrix.RELOADED:2>)>>>WatchList(Batman.DARK_KNIGHT).to_numpy()array([0.,1.,0.,0.,0.,0.,1.,0.])>>>WatchList.from_numpy(_)WatchList(next_movie=<Batman.DARK_KNIGHT:2>)Decoding is a fairly straigtforward process though encoding might be somewhat problematic: Python'stypingis not designed to provide separation-by-construction for union types. The library uses simpleisinstancechecks to test out all types provided against a given value, first match is used. The library does not traverse generics, origins, supertypes, etc. So, be diligent defining ofUnion.Recursive DefinitionsRecursive definitions, like linked lists, trees, graphs etc, arenot supported. From a usability and performance point of view, it's crucial for encoder/decoder to be able to evaluate statically output tensor size.TargetsThe library supports the following containers as tensors:NumPy ndarraywithto_numpy/from_numpyPyTorch tensorswithto_torch/from_torchThe best way to work withTensorFlow tensorsis to use NumPy ndarrays and convert result withtensorflow.convert_to_tensor(as the tensor stored in memory as a ndarray anywyas).Note, that dependencies are not installed with the library itself (TensorFlow, PyTorch or NumPy) and should be provided at runtime.PerformanceTensor layout is not cached and is computed for each operation. When performing a lot of operations with class definition staying the same, it makes sense to re-use layout. For example:>>>classMatrix(Enum):...THE_MATRIX=1...RELOADED=2...REVOLUTIONS=3...>>>@dataclass_tensor...@dataclass...classWatchList:...matrix:Matrix...>>>layout=WatchList.tensor_layout()>>>WatchList(Matrix.RELOADED).to_numpy(tensor_layout=layout)array([0.,0.,1.])>>>WatchList.from_numpy(_,tensor_layout=layout)WatchList(matrix=<Matrix.RELOADED:2>)Advanced FeaturesDtypeThe library supports float and integer (long) tensors. The data type could be specified either as a parameter to thedataclass_tensordecorator (applied to all operations) or independently as an argument toto_tensorfunction call. See examples below.dtypeargument is passed to the corresponding target library, e.g. NumPy (docs), PyTorch (docs) or TensorFlow.>>>classMatrix(Enum):...THE_MATRIX=1...RELOADED=2...REVOLUTIONS=3...>>>@dataclass_tensor...@dataclass...classWatchList:...matrix:Matrix...>>>WatchList(Matrix.RELOADED).to_numpy()array([0.,0.,1.],dtype=float32)>>>WatchList(Matrix.RELOADED).to_numpy(dtype="int32")array([0,0,1],dtype=int32)or with defaults setup in a decorator>>>classMatrix(Enum):...THE_MATRIX=1...RELOADED=2...REVOLUTIONS=3...>>>@dataclass_tensor(dtype="int32")...@dataclass...classWatchList:...matrix:Matrix...>>>WatchList(Matrix.RELOADED).to_numpy()array([0,0,1],dtype=int32)BatchTo create batch, usebatch=Trueparameter. See examples:>>>classMatrix(Enum):...THE_MATRIX=1...RELOADED=2...REVOLUTIONS=3...>>>@dataclass_tensor...@dataclass...classWatchList:...matrix:Matrix...>>>WatchList.to_numpy([...WatchList(Matrix.THE_MATRIX),...WatchList(Matrix.RELOADED),...],batch=True)array([[1.,0.,0.],[0.,1.,0.]],dtype=float32)>>>WatchList.from_numpy(_,batch=True)[WatchList(next_move=<Matrix.THE_MATRIX:0>),WatchList(next_move=<Matrix.RELOADED:1>)]batch_sizecould be used to provide length hint (to ensure good performance when working with generators):>>>WatchList.to_numpy((...WatchList(Matrix.THE_MATRIX),...WatchList(Matrix.RELOADED),...),batch_size=2)array([[1.,0.,0.],[0.,1.,0.]],dtype=float32)>>>WatchList.from_numpy(_,batch_size=2)[WatchList(next_move=<Matrix.THE_MATRIX:0>),WatchList(next_move=<Matrix.RELOADED:1>)]Custom Attribute ResolverTBDTODOCustom attribute resolver (e.g. from dict instead of class instance)Pretty-print for tensor layout objectContributingCheck for open issues or open a fresh issue to start a discussion around a feature idea or a bug.Fork the repository on Github & branch frommaintofeature-*to start making your changes.Write a test which shows that the bug was fixed or that the feature works as expected.or simply...Use it.Enjoy it.Spread the word.LicenseCopyright © 2021, Oleksii Kachaiev.dataclasses-tensoris licensed under the MIT license, available at MIT and also in the LICENSE file.