Package: recode
recodeMake codecs for fixed size structured chunks serialization and deserialization of
sequences, tabular data, and time-series.To install:pip install recodeDocs hereMake codecs for fixed size structured chunks serialization and deserialization of
sequences, tabular data, and time-series.The easiest and bigest bang for your buck ismk_codec>>>fromrecodeimportmk_codec>>>encoder,decoder=mk_codec()encoderwill encode a list (or any iterable) of numbers into bytes>>>b=encoder([0,-3,3.14])>>>bb'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\xc0\x1f\x85\xebQ\xb8\x1e\t@'decoderwill decode those bytes to get you back your numbers>>>decoder(b)[0.0,-3.0,3.14]There's only really one argument you need to know about inmk_codec.
The first argument, calledchk_format, which is a string of characters from
the "Format" column of the pythonformat charactersThe one we've just been through is in fact>>>encoder,decoder=mk_codec('d')That is, it will expect that your data is a list of numbers, and they'll be encoded
with the 'd' format character, that is 8-bytes doubles.
That default is good because it gives you a lot of room, but if you knew that you
would only be dealing with 2-byte integers (as in most WAV audio waveforms),
you would have chosenh:>>>encoder,decoder=mk_codec('h')What about those channels?
Well, some times you need to encode/decode multi-channel streams, such as:>>>multi_channel_stream=[[3,-1],[4,-1],[5,-9]]Say, for example, if you were dealing with stereo waveform
(with the standard PCM_16 format), you'd do it this way:>>>encoder,decoder=mk_codec('hh')>>>pcm_bytes=encoder(multi_channel_stream)>>>pcm_bytesb'\x03\x00\xff\xff\x04\x00\xff\xff\x05\x00\xf7\xff'>>>decoder(pcm_bytes)[(3,-1),(4,-1),(5,-9)]Then_channelsandchk_size_bytesarguments are there if you want to assert
that your number of channels and chunk size are what you expect.
Again, these are just for verification, because we know how easy it is to
misspecify thechk_format, and how hard it can be to notice that we did.It is advised to use these in any production code, for the sanity of everyone!>>>mk_codec('hhh',n_channels=2)Traceback(mostrecentcalllast):...AssertionError:Yousaidthere'd be 2 channels, but I inferred 3>>>mk_codec('hhh',chk_size_bytes=3)Traceback(mostrecentcalllast):...AssertionError:Thegivenchk_size_bytes3didnotmatchtheinferred(fromchk_format)6Finally, so far we've done it this way:>>>encoder,decoder=mk_codec('hHifd')But see that what's actually returned is a NAMED tuple, which means that you can
can also get one object that will have.encodeand.decodeproperties:>>>codec=mk_codec('hHifd')>>>to_encode=[[1,2,3,4,5],[6,7,8,9,10]]>>>encoded=codec.encode(to_encode)>>>decoded=codec.decode(encoded)>>>decoded[(1,2,3,4.0,5.0),(6,7,8,9.0,10.0)]And you can checkout the properties of your encoder and decoder (they
should be the same)>>>codec.encode.chk_format'hHifd'>>>codec.encode.n_channels5>>>codec.encode.chk_size_bytes24Further functionality and under-the-hood peepsThis section shows various examples of recode and it how it can be used with:Single channel numerical streamsMulti-channel numerical streamsDataFramesIteratorsfromrecodeimport(ChunkedEncoder,ChunkedDecoder,MetaEncoder,MetaDecoder,IterativeDecoder,StructCodecSpecs,specs_from_frames,frame_to_meta,meta_to_frame)Quick run throughFirst define the frame you want to encodeframe=[1,2,3]Next define the specifications for that encodingspecs=StructCodecSpecs(chk_format='h')specsStructCodecSpecs(chk_format='h', n_channels=1, chk_size_bytes=2)Next define an encoder and encode your frame to bytesencoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)b=encoder(frame)bb'\x01\x00\x02\x00\x03\x00'Once you need your original frame again, define a decoder and decode your framedecoder=ChunkedDecoder(specs.chk_to_frame)decoded_frames=decoder(b)decoded_frames[1, 2, 3]Step-by-step explanationDefine your StructCodecSpecsStructCodecSpecs is used to define the specs for making codecs for fixed size structured chunks serialization and deserialization of sequences, tabular data, and time-series. This definition is based on format strings of thepython struct module.There are two ways to define StructCodecSpecs, the first being to explicitly
define it using the StructCodecSpecs class.
StructCodecSpecs takes three arguments:chk_format,n_channels,
andchk_size_bytes.Onlychk_formatis required,
asn_channelsandchk_size_bytescan be determined based onchk_format.
Ifn_channelsorchk_size_bytesare given, they will be used to
assert that the values inferred fromchk_formatmatch.If this is not the case, then do not provide an argument forn_channelsand instead pass a string with the format character matching the data type for
each channel in the frame tochk_format.
For example if the first channel
contains integers and the second contains floats, thenchk_format = 'hd'.frame=[1,2,3]specs=StructCodecSpecs(chk_format='h')specsStructCodecSpecs(chk_format='h', n_channels=1, chk_size_bytes=2)The second way to define StructCodecSpecs is to usespecs_from_frameswhich will implictly define StructCodecSpecs based on the frame that is going to be encoded/decoded. This function will return a tuple containing a reconstituted version of the iterator given as an input, and the defined StructCodecSpecs. The first element of the tuple can be ignored if the frame passed is not an iterator._,specs=specs_from_frames(frame)specsStructCodecSpecs(chk_format='h', n_channels=1, chk_size_bytes=2)If frame is an iterator, then redefine frame as the first argument of the tuple so the first element of frame is not lost for encoding.frame=iter([[1,2],[3,4]])frame,specs=specs_from_frames(frame)specs,list(frame)(StructCodecSpecs(chk_format='hh', n_channels=2, chk_size_bytes=4),
 [[1, 2], [3, 4]])Define your EncoderYour Encoder will allow you to, you guessed it, encode your frames! There are two Encoders currently defined in recode:ChunkedEncoderandMetaEncoder.ChunkedEncodershould be your goto encoder for sequences, whileMetaEncoderworks best for tabular data (currently must be in the format of list of dicts).frame=[1,2,3]_,specs=specs_from_frames(frame)encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)b=encoder(frame)bb'\x01\x00\x02\x00\x03\x00'When using aMetaEncoder, an extra argument named frame_to_meta is required, which can be easily imported from recode!frame=[{'foo':1,'bar':1},{'foo':2,'bar':2},{'foo':3,'bar':4}]_,specs=specs_from_frames(frame)encoder=MetaEncoder(frame_to_chk=specs.frame_to_chk,frame_to_meta=frame_to_meta)b=encoder(frame)bb'\x07\x00foo.bar\x01\x00\x01\x00\x02\x00\x02\x00\x03\x00\x04\x00'Define your DecoderNext up your Decoder will allow you to decode your encoded bytes. There are three Encoders currently defined in recode:ChunkedDecoder,IterativeDecoder, andMetaDecoder.EitherChunkedDecoderorIterativeDecoderwill work well for sequences, with the only difference being thatIterativeDecoderreturns an iterator of decoded chunks whileChunkedDecoderreturns the whole list of decoded chunks.MetaDecoderworks best for tabular data encoded withMetaEncoder.frame=[1,2,3]_,specs=specs_from_frames(frame)encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(specs.chk_to_frame)b=encoder(frame)decoder(b)[1, 2, 3]As is shown in the following example, anIterativeDecoderwill return an unpack_iterator.frame=[[1,1],[2,2]]_,specs=specs_from_frames(frame)encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=IterativeDecoder(chk_to_frame=specs.chk_to_frame)b=encoder(frame)iter_frames=decoder(b)print(type(iter_frames))next(iter_frames),next(iter_frames)<class 'unpack_iterator'>
((1, 1), (2, 2))When using aMetaDecoder, an extra argument named meta_to_frame is required, which can be easily imported from recode!frame=[{'foo':1,'bar':1},{'foo':2,'bar':2},{'foo':3,'bar':4}]_,specs=specs_from_frames(frame)encoder=MetaEncoder(frame_to_chk=specs.frame_to_chk,frame_to_meta=frame_to_meta)decoder=MetaDecoder(chk_to_frame=specs.chk_to_frame,meta_to_frame=meta_to_frame)b=encoder(frame)decoded_frames=decoder(b)decoded_frames[{'foo': 1, 'bar': 1}, {'foo': 2, 'bar': 2}, {'foo': 3, 'bar': 4}]Encoding a waveformCreate a synthetic waveform usinghumfromhum.gen.sine_miximportfreq_based_stationary_wfimportmatplotlib.pyplotaspltimportnumpyasnpDFLT_N_SAMPLES=21*2048DFLT_SR=44100wf_mix=freq_based_stationary_wf(freqs=(200,400,600,800),weights=None,n_samples=DFLT_N_SAMPLES,sr=DFLT_SR)plt.plot(wf_mix[:300]);wf_mixarray([0.        , 0.07114157, 0.14170624, ..., 0.49480205, 0.54853009,
       0.5979443 ])Encode the wfspecs=StructCodecSpecs('d')encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(chk_to_frame=specs.chk_to_frame)b=encoder(wf_mix)b[:100]b'\x00\x00\x00\x00\x00\x00\x00\x00\x1b:\x11\x8dU6\xb2?\xb8!\xa2\x15n#\xc2?\xe6I\xedz\x15\x06\xcb?L;\xe6\xfah\xd8\xd1?}\x8c\x9b\x9a\xf5\x08\xd6?D\xcc\xf1\x897\x0c\xda?\xe3\xc5\xbeb/\xda\xdd?DE\xb6K\xb6\xb5\xe0?\x82|\xfcz\x90\\\xe2?\xae3:$\x99\xde\xe3?\x1e\x95\xbe\xef$9\xe5?\x81\xd7\xdc\xec'Decode and compare to wf_mixdecoded_wf_mix=decoder(b)plt.plot(decoded_wf_mix[:300]);np.all(decoded_wf_mix==wf_mix)TrueEncoding a pandas dataframeimportpandasaspdCreate/import your dataframedf=pd.DataFrame(data=[[1,2,3],[4,5,6]],columns=['foo','bar','set'])dffoobarset01231456Prep the dataframe for encodingframe=df.to_dict('records')frame[{'foo': 1, 'bar': 2, 'set': 3}, {'foo': 4, 'bar': 5, 'set': 6}]Encode the list of dicts_,specs=specs_from_frames(frame)encoder=MetaEncoder(frame_to_chk=specs.frame_to_chk,frame_to_meta=frame_to_meta)decoder=MetaDecoder(chk_to_frame=specs.chk_to_frame,meta_to_frame=meta_to_frame)b=encoder(frame)bb'\x0b\x00foo.bar.set\x01\x00\x02\x00\x03\x00\x04\x00\x05\x00\x06\x00'decoded_frame=decoder(b)decoded_frame[{'foo': 1, 'bar': 2, 'set': 3}, {'foo': 4, 'bar': 5, 'set': 6}]decoded_df=pd.DataFrame(decoded_frame)print(np.all(decoded_df==df))decoded_dfTruefoobarset01231456Miscellaneous informationByte order, Size, and AlignmentThis table provides the characters associated with Byte order, Size, and Alignment. If one of these characters is not given as the first character of the format string, then@will be assumed. More information about Byte order, Size, and Alignment can be foundhere.CharacterByte orderSizeAlignment@nativenativenative=nativestandardnone<little-endianstandardnone>big-endianstandardnone!network (=big-endian)standardnoneFull ExamplesSingle channel numerical streamfromrecodeimportStructCodecSpecs,ChunkedEncoder,ChunkedDecoderspecs=StructCodecSpecs(chk_format='h')encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(chk_to_frame=specs.chk_to_frame)frames=[1,2,3]b=encoder(frames)assertb==b'\x01\x00\x02\x00\x03\x00'decoded_frames=decoder(b)assertdecoded_frames==framesMulti-channel numerical streamfromrecodeimportStructCodecSpecs,ChunkedEncoder,ChunkedDecoderspecs=StructCodecSpecs(chk_format='@hh',n_channels=2)encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(chk_to_frame=specs.chk_to_frame)frames=[(1,2),(3,4),(5,6)]b=encoder(frames)assertb==b'\x01\x00\x02\x00\x03\x00\x04\x00\x05\x00\x06\x00'decoded_frames=decoder(b)assertdecoded_frames==framesIterative decoderfromrecodeimportStructCodecSpecs,ChunkedEncoder,IterativeDecoderspecs=StructCodecSpecs(chk_format='hdhd')encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=IterativeDecoder(chk_to_frame=specs.chk_to_frame)frames=[(1,1.1,1,1.1),(2,2.2,2,2.2),(3,3.3,3,3.3)]b=encoder(frames)iter_frames=decoder(b)assertnext(iter_frames)==frames[0]next(iter_frames)(2, 2.2, 2, 2.2)DataFrame (as list of dicts) using MetaEncoder/MetaDecoderfromrecodeimportStructCodecSpecs,MetaEncoder,MetaDecoder,frame_to_meta,meta_to_framedata=[{'foo':1.1,'bar':2.2},{'foo':513.23,'bar':456.1},{'foo':32.0,'bar':6.7}]specs=StructCodecSpecs(chk_format='dd',n_channels=2)encoder=MetaEncoder(frame_to_chk=specs.frame_to_chk,frame_to_meta=frame_to_meta)decoder=MetaDecoder(chk_to_frame=specs.chk_to_frame,meta_to_frame=meta_to_frame)b=encoder(data)assertdecoder(b)==dataImplicitly define codec specs based on framesfromrecodeimportspecs_from_frames,ChunkedEncoder,ChunkedDecoderframes=[1,2,3]_,specs=specs_from_frames(frames)encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(chk_to_frame=specs.chk_to_frame)b=encoder(frames)assertb==b'\x01\x00\x02\x00\x03\x00'decoded_frames=decoder(b)decoded_frames[1, 2, 3]Implicit definition of iteratorfromrecodeimportspecs_from_frames,ChunkedEncoder,IterativeDecoderframes=iter([[1.1,2.2],[3.3,4.4]])frames,specs=specs_from_frames(frames)encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(chk_to_frame=specs.chk_to_frame)b=encoder(frames)decoded_frames=list(decoder(b))assertdecoded_frames==[(1.1,2.2),(3.3,4.4)]MoreExample of recode functionality to read and write audio filesIn the below example we can see that the functionality of reading and writing audio to bytes is replicated inrecode.
An example of a waveform audio file is first created usingmk_wf.
Then, that example wave form is converted to bytes usingBytesIOandsoundfile.write.
Then, that same example wave form is converted to bytes usingrecodefunctionality.
Finally it can be seen through the assertions thatsoundfile's read/write andrecode's encode/decode provide
the same functionality for audio files (aside from the header insoundfileas a result of the .wav format).importsoundfileassffromioimportBytesIOfromenumimportEnumimportnumpyasnpfromrecodeimportChunkedDecoder,ChunkToFrame,ChunkedEncoder,StructCodecSpecsclassKind(Enum):random='random'increasing='increasing'defmk_wf(n_samples=2048,kind:Kind=Kind.random,**kwargs):ifkind==Kind.random:dtype_str=kwargs.get('num_type','int16')ifdtype_str.startswith('int'):low=kwargs.get('low',int(-2**15))high=kwargs.get('high',int(2**15-1))wf=np.random.randint(low=low,high=high,size=n_samples,dtype=dtype_str)else:raiseTypeError("Don't know how to handle this case")else:raiseTypeError("Don't know how to handle this case")returnwfwf=mk_wf(25)sr=10000file=BytesIO()sf.write(file,wf,sr,format='WAV')file.seek(0)b=file.read()specs=StructCodecSpecs('h')encoder=ChunkedEncoder(frame_to_chk=specs.frame_to_chk)decoder=ChunkedDecoder(chk_size_bytes=specs.chk_size_bytes,chk_to_frame=specs.chk_to_frame,n_channels=specs.n_channels)d=encoder(wf)assertd==b[44:]sf_read=sf.read(BytesIO(b),dtype='int16')recode_read=decoder(d)assertnp.all(sf_read[0]==recode_read)assertnp.all(recode_read==wf)assertnp.all(sf_read[0]==wf)