Package: turbo-broccoli
Turbo Broccoli 🥦JSON (de)serialization extensions, originally aimed atnumpyandtensorflowobjects, but now supporting a wide range of objects.Installationpipinstallturbo-broccoliUsageTo/from stringimportnumpyasnpimportturbo_broccoliastbobj={"an_array":np.array([[1,2],[3,4]],dtype="float32")}tb.to_json(obj)produces the following string (modulo indentation and the value of$.an_array.data.data):{"an_array":{"__type__":"numpy.ndarray","__version__":5,"data":{"__type__":"bytes","__version__":3,"data":"QAAAAAAAAAB7ImRhd..."}}}For deserialization, simply usetb.from_json(json_string)To/from fileSimply replaceturbo_broccoli.to_jsonandturbo_broccoli.from_jsonwithturbo_broccoli.save_jsonandturbo_broccoli.load_json:importnumpyasnpimportturbo_broccoliastbobj={"an_array":np.array([[1,2],[3,4]],dtype="float32")}tb.save_json(obj,"foo/bar/foobar.json")...obj=tb.load_json("foo/bar/foobar.json")ContextsThe behaviour ofturbo_broccoli.to_jsonandturbo_broccoli.from_jsoncan be tweaked by usingcontexts.
For example, to set a encryption/decryption key forsecret
types:importnacl.secretimportnacl.utilsimportturbo_broccoliastbkey=nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)ctx=tb.Context(nacl_shared_key=key)obj={"user":"alice","password":tb.SecretStr("dolphin")}doc=tb.to_json(obj,ctx)...obj=tb.from_json(doc,ctx)The behaviour ofturbo_broccoli.save_jsonandturbo_broccoli.load_jsoncan be tweaked in a similar manner, but for convenience, the argument of the
context are passed directly to the method:importnacl.secretimportnacl.utilsimportturbo_broccoliastbkey=nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)obj={"user":"alice","password":tb.SecretStr("dolphin")}tb.save_json(obj,"foo/bar/foobar.json",nacl_shared_key=key)Seethe
documentation.ArtifactsIf an object insideobjis too large to be embedded inside the JSON file
(e.g. a large numpy array), then anartifactfile is created:importnumpyasnpimportturbo_broccoliastbobj={"an_array":np.random.rand(1000,1000)}tb.save_json(obj,"foo/bar/foobar.json")produces the JSON file{"an_array":{"__type__":"numpy.ndarray","__version__":5,"data":{"__type__":"bytes","__version__":3,"id":"1e6dff28-5e26-44df-9e7a-75bc726ce9aa"}}}and a filefoo/bar/foobar.1e6dff28-5e26-44df-9e7a-75bc726ce9aa.tbcontaining
the array data. The artifact directory can be explicitely specified by setting
it in theserialization
contextor by setting theTB_ARTIFACT_PATHenvironment variable (see below.). The
code for loading the JSON file does not change:obj=tb.load_json("foo/bar/foobar.json")If usingturbo_broccoli.to_json,
since there is no output file path specified, the artifacts are storied in a
temporary directory instead:importnumpyasnpimportturbo_broccoliastbobj={"an_array":np.random.rand(1000,1000)}doc=tb.to_json(obj)# An artifact has been created somewhere in e.g. /tmpSince no information about this directory is stored in the output JSON string,
it is not possible to loaddocusingturbo_broccoli.from_json.
If deserialization is necessary, instantiate acontext:importnumpyasnpimportturbo_broccoliastbctx=tb.Context()obj={"an_array":np.random.rand(1000,1000)}doc=tb.to_json(obj,ctx)# An artifact has been created in ctx.artifact_path...obj=tb.from_json(doc,ctx)Environment variablesSome behaviors of Turbo Broccoli can be tweaked by setting specific environment
variables. If you want to modify these parameters programatically, do not do so
by modifyingos.environ. Rather, use aturbo_broccoli.Context.TB_ARTIFACT_PATH(default: output JSON file's parent directory): During
serialization, Turbo Broccoli may create artifacts to which the JSON object
will point to. The artifacts will be stored inTB_ARTIFACT_PATHif
specified.TB_KERAS_FORMAT(default:tf, valid values arekeras,tf, andh5):
The serialization format for keras models. Ifh5ortfis used, an
artifact following said format will be created inTB_ARTIFACT_PATH. Ifjsonis used, the model will be contained in the JSON document (anthough
the weights may be in artifacts if they are too large).TB_MAX_NBYTES(default:8000):
The maximum byte size of a python object beyond which serialization will
produce an artifact instead of storing it in the JSON document. This does not
limit the size of the overall JSON document though. 8000 bytes should be
enough for a numpy array of 1000float64s to be stored in-document.TB_NODECODE(default: empty):
Comma-separated list of types to not deserialize, for examplebytes,numpy.ndarray. Excludable types are:bokeh,bokeh.buffer,bokeh.generic,bytes,collections,collections.deque,collections.namedtuple,collections.set,dataclass,dataclass.<dataclass_name>(case sensitive),datetime,datetime.datetime,datetime.time,datetime.timedelta,generic,keras,keras.model,keras.layer,keras.loss,keras.metric,keras.optimizer,numpy,numpy.ndarray,numpy.number,numpy.dtype,numpy.random_state,pandas,pandas.dataframe,pandas.series,Warning: excludingpandas.dataframewill also excludepandas.series,pytorch,pytorch.tensor,pytorch.module,pytorch.concatdataset,pytorch.stackdataset,pytorch.subset,pytorch.tensordatasetscipy,scipy.csr_matrix,secret,sklearn,sklearn.estimator,sklearn.estimator.<estimator name>(case
sensitive, see the list of supported sklearn estimators below),tensorflow,tensorflow.sparse_tensor,tensorflow.tensor,tensorflow.variable.TB_SHARED_KEY(default: empty):
Secret key used to encrypt/decrypt secrets. The encryption usespynacl'sSecretBox.
An exception is raised when attempting to serialize a secret type while no
key is set.Guarded blocksThis is so cool. Check outturbo_broccoli.GuardedBlockHandler.Supported typesBasic typesbytesCollections:collections.deque,collections.namedtupleDataclasses:
serialization is straightforward:@dataclassclassC:a:intb:strdoc=tb.to_json({"c":C(a=1,b="Hello")})For deserialization, first register the class:ctx=tb.Context(dataclass_types=[C])tb.from_json(doc,ctx)datetime.datetime,datetime.time,datetime.timedeltaGeneric objectsserialization only. A generic object is an object that
has the__turbo_broccoli__attribute. This attribute is expected to be a list
of attributes whose values will be serialized. For example,classC:__turbo_broccoli__=["a","b"]a:intb:intc:intx=C()x.a,x.b,x.c=42,43,44tb.to_json(x)produces the following string:{"a":42,"b":43,}Registered attributes can of course have any type supported by Turbo Broccoli,
such as numpy arrays. Registered attributes can be@propertymethods.Keraskeras.Model;standard subclasses ofkeras.layers.Layer,keras.losses.Loss,keras.metrics.Metric, andkeras.optimizers.Optimizer.Numpynumpy.number,numpy.ndarraywith numerical dtype, andnumpy.dtype.Pandaspandas.DataFrameandpandas.Series, but with the following limitations:the following dtypes are not supported:complex,object,timedeltathe column / series names cannot be ints or int-strings. The following are
not acceptable:df=pd.DataFrame([[1,2],[3,4]])df=pd.DataFrame([[1,2],[3,4]],columns=["0","1"])Tensorflowtensorflow.Tensorwith numerical dtype, but nottensorflow.RaggedTensor.Pytorchtorch.Tensor,Warning: loaded tensors are automatically placed on the
CPU and gradients are lost;torch.nn.Module, don't forget to register your module type using aturbo_broccoli.Context:# SerializationclassMyModule(torch.nn.Module):...module=MyModule()# Must be instantiable without argumentsdoc=tb.to_json({"module":module})# Deserializationctx=tb.Context(pytorch_module_types=[MyModule])module=tb.from_json(doc,ctx)Warning: It is not possible to register and deserializestandard pytorch
module containersdirectly. Wrap them in your own custom module class. For following is not
acceptableimportturbo_broccoliastbimporttorchmodule=torch.nn.Sequential(torch.nn.Linear(4,2),torch.nn.ReLU(),torch.nn.Linear(2,1),torch.nn.ReLU(),)obj={"module":module}doc=tb.to_json(obj)# works, but...tb.from_json(a,ctx)# does't workbut the following works:classMyModule(torch.nn.Module):module:torch.nn.Sequential# Wrapped sequentialdef__init__(self):super().__init__()self.module=torch.nn.Sequential(torch.nn.Linear(4,2),torch.nn.ReLU(),torch.nn.Linear(2,1),torch.nn.ReLU(),)...module=MyModule()# Must be instantiable without argumentsdoc=tb.to_json({"module":module})ctx=tb.Context(pytorch_module_types=[MyModule])module=tb.from_json(doc,ctx)torch.utils.data.ConcatDataset,torch.utils.data.StackDataset,torch.utils.data.Subset,torch.utils.data.TensorDataset, as long as the
nested structure of datasets ultimately lead totorch.utils.data.TensorDatasets (e.g. a subset of a stack of subsets of
tensor datasets is supported)ScipyJustscipy.sparse.csr_matrix. ^^"Scikit-learnsklearnestimators (i.e. that inherit fromsklean.base.BaseEstimator).
Supported estimators are:AdaBoostClassifier,AdaBoostRegressor,AdditiveChi2Sampler,AffinityPropagation,AgglomerativeClustering,ARDRegression,BayesianGaussianMixture,BayesianRidge,BernoulliNB,BernoulliRBM,Binarizer,CategoricalNB,CCA,ClassifierChain,ComplementNB,DBSCAN,DecisionTreeClassifier,DecisionTreeRegressor,DictionaryLearning,ElasticNet,EllipticEnvelope,EmpiricalCovariance,ExtraTreeClassifier,ExtraTreeRegressor,ExtraTreesClassifier,ExtraTreesRegressor,FactorAnalysis,FeatureUnion,GaussianMixture,GaussianNB,GaussianRandomProjection,GraphicalLasso,HuberRegressor,IncrementalPCA,IsolationForest,Isomap,KernelCenterer,KernelDensity,KernelPCA,KernelRidge,KMeans,KNeighborsClassifier,KNeighborsRegressor,KNNImputer,LabelBinarizer,LabelEncoder,LabelPropagation,LabelSpreading,Lars,Lasso,LassoLars,LassoLarsIC,LatentDirichletAllocation,LedoitWolf,LinearDiscriminantAnalysis,LinearRegression,LinearSVC,LinearSVR,LocallyLinearEmbedding,LocalOutlierFactor,LogisticRegression,MaxAbsScaler,MDS,MeanShift,MinCovDet,MiniBatchDictionaryLearning,MiniBatchKMeans,MiniBatchSparsePCA,MinMaxScaler,MissingIndicator,MLPClassifier,MLPRegressor,MultiLabelBinarizer,MultinomialNB,MultiOutputClassifier,MultiOutputRegressor,MultiTaskElasticNet,MultiTaskLasso,NearestCentroid,NearestNeighbors,NeighborhoodComponentsAnalysis,NMF,Normalizer,NuSVC,NuSVR,Nystroem,OAS,OneClassSVM,OneVsOneClassifier,OneVsRestClassifier,OPTICS,OrthogonalMatchingPursuit,PassiveAggressiveRegressor,PCA,Pipeline,PLSCanonical,PLSRegression,PLSSVD,PolynomialCountSketch,PolynomialFeatures,PowerTransformer,QuadraticDiscriminantAnalysis,QuantileRegressor,QuantileTransformer,RadiusNeighborsClassifier,RadiusNeighborsRegressor,RandomForestClassifier,RandomForestRegressor,RANSACRegressor,RBFSampler,RegressorChain,RFE,RFECV,Ridge,RidgeClassifier,RobustScaler,SelectFromModel,SelfTrainingClassifier,SGDRegressor,ShrunkCovariance,SimpleImputer,SkewedChi2Sampler,SparsePCA,SparseRandomProjection,SpectralBiclustering,SpectralClustering,SpectralCoclustering,SpectralEmbedding,StackingClassifier,StackingRegressor,StandardScaler,SVC,SVC,SVR,SVR,TheilSenRegressor,TruncatedSVD,TSNE,VarianceThreshold,VotingClassifier,VotingRegressor. Doesn't work with:All CV classes because thescore_attribute is a dict indexed withnp.int64, whichjson.JSONEncoder._iterencode_dictrejects.Everything that is parametrized by an arbitrary object/callable/estimator:FunctionTransformer,TransformedTargetRegressor.Other classes that have non JSON-serializable attributes:ClassNon-serializable attr.Birch_CFNodeBisectingKMeansfunctionColumnTransformersliceGammaRegressorHalfGammaLossGaussianProcessClassifierProductGaussianProcessRegressorSumIsotonicRegressioninterp1dOutputCodeClassifier_ConstantPredictorPerceptronHingePoissonRegressorHalfPoissonLossSGDClassifierHingeSGDOneClassSVMHingeSplineTransformerBSplineTweedieRegressorHalfTweedieLossIdentityOther errors:FastICA: I'm not sure why...BaggingClassifier:IndexError: only integers, slices (:), ellipsis (...), numpy.newaxis (None) and integer or boolean arrays are valid indices.GradientBoostingClassifier,GradientBoostingRegressor,RandomTreesEmbedding,KBinsDiscretizer:Exception: dtype object is not covered.HistGradientBoostingClassifier: Problems with deserialization of_BinMapperobject?PassiveAggressiveClassifier: some unknown label type error...SequentialFeatureSelector: Problem with the unit test itself ^^"KNeighborsTransformer: A serialized-deserialized instance seems tofit_transforman array to a sparse matrix whereas the original object
returns an array?RadiusNeighborsTransformer: Inverse problem fromKNeighborsTransformer.BokehBokehfiguresandmodels.SecretsBasic Python types can be wrapped in their corresponding secret type according
to the following tablePython typeSecret typedictturbo_broccoli.SecretDictfloatturbo_broccoli.SecretFloatintturbo_broccoli.SecretIntlistturbo_broccoli.SecretListstrturbo_broccoli.SecretStrThe secret value can be recovered with theget_secret_valuemethod. At
serialization, the this value will be encrypted. For example,## See https://pynacl.readthedocs.io/en/latest/secret/#keyimportnacl.secretimportnacl.utilsimportturbo_broccoliastbkey=nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)ctx=tb.Context(nacl_shared_key=key)obj={"user":"alice","password":tb.SecretStr("dolphin")}tb.to_json(obj,ctx)produces the following string (modulo indentation and modulo the encrypted
content):{"user":"alice","password":{"__type__":"secret","__version__":2,"data":{"__type__":"bytes","__version__":3,"data":"gbRXF3hq9Q9hIQ9Xz+WdGKYP5meJ4eTmlFt0r0Ov3PV64065plk6RqsFUcynSOqHzA=="}}}Deserialization decrypts the secrets, but they stay wrapped inside the secret
types above. If the wrong key is provided, an exception is raised. If no key is
provided, the secret values are replaced by aturbo_broccoli.LockedSecret. Internally, Turbo Broccoli usespynacl'sSecretBox.Warning: In the case ofSecretDictandSecretList, the values contained
within must be JSON-serializablewithoutTurbo Broccoli. The following is
not acceptable:importnacl.secretimportnacl.utilsimportnumpyasnpimportturbo_broccoliastbkey=nacl.utils.random(nacl.secret.SecretBox.KEY_SIZE)ctx=tb.Context(nacl_shared_key=key)obj={"data":tb.SecretList([np.array([1,2,3])])}tb.to_json(obj,ctx)See also theTB_SHARED_KEYenvironment variable below.ContributingDependenciespython3.10or newer;requirements.txtfor runtime dependencies;requirements.dev.txtfor development dependencies.make(optional);Simply runvirtualenvvenv-ppython3.10
../venv/bin/activate
pipinstall--upgradepip
pipinstall-rrequirements.txt
pipinstall-rrequirements.dev.txtDocumentationSimply runmakedocsThis will generate the HTML doc of the project, and the index file should be atdocs/index.html. To have it directly in your browser, runmakedocs-browserCode qualityDon't forget to runmaketo format the code followingblack,
typecheck it usingmypy, and check it against coding
standards usingpylint.Unit testsRunmaketestto havepytestrun the unit tests intests/.CreditsThis project takes inspiration fromCrimson-Crow/json-numpy.