Package: keras-generators
keras-generatorsMulti-dimensional/Multi-input/Multi-output Data preprocessing and Batch Generators for Tensorflow modelsInstallationYou can find the library on PyPikeras_generatorspipinstallkeras_generatorsThe reasons this library existsThis library solves several basic problems in area of data preprocessing (scaling and encoding) and batch generation for
Tensorflow models, for which there are no solution ( at the moment) in Tensorflow or other open-source libraries. There
are several functionalities in opensource libraries which solves some of the below problems, but only partially, and
there's no way to combine them into a single solution without performing some custom adaptation or extending their
functionalities. The libraries arescikit-learn,TensorFlowandtensorflow-datasetsandtensorflow-transform.Problem 1:Batch generation from multi-variate Timeseries data with train/test/val split and z-score scalingImagine you have a timeseries data (like weather temperature, stock market prices) and you need to train a neural model
to predict the next value in the sequence based on the sequential input.
Basic operations you have to do:- split the data into train/test/val setsYou can
useTimeseriesGeneratorfrom Tensorflow which is able to ingest multi-variate timeseries data and produce batches of inputs and targets. It's
able to take time series parameters such as stride, length of history, etc., and produce batches for
training/validation. But, it's not able to generate multi-steptargetdata, and work on train/test splits. You'll need
to split the data manually by taking care of lookback and target data lengths, with all the possible edge cases of
stride and sampling parameters.- generatetargetdataYou'll have to generatetargetdata by yourself - there's no functionality in any of above library to extract it from
the ingested timeseries, and this might not be trivial, especially if you want to generate multi-step targets (like
predicting 3 data-points ahead).- split the data into train/test/val-you'll have to do it sets by yourself - and this is not trivial as just splitiing by index won't work, as you have
lookback to take care of, and you'll have to split the target data perfectly aligned with the input data.
Thetf.data.Datasetis able to do that by
usingwindow,skipandtakeandbatchmethods. It's able to align the targets (
if you'd have those somehow generated), but it's not able to perform re-shuffling of data at the end of the epoch, as
well as scaling/normalizing of the data. Although there
existstft.scale_to_z_score, it doesn't
let you save the coefficients and scale the data for the inference, thus rendering it unusable for production use.- scaling/normalizing the data + encodingyou'll have to perform encoding and scaling of the data by yourself, and take care of saving coefficients and
parameters of encoding for the inference. There
areStandardScalerandencoderswhich are able to perform de-normalizing of the target predictions for inference (inverse_transform), as well as save
coefficients for normalizing the new input data for inference, but it will have to be saved separately for every input
and output separately, i.e. if you have multi-input, or multi-output network, for every I/O layer you'll have to save
the scalers separately and apply them before inference.- re-shuffling after every epochyou'll need to perform re-shuffling of the data at the end of the epoch by yourself, so you need to create custom
batch generator class to be used by TF trainer.Problem 2Multi-input / multi-output data preprocessing and batch generationTensorflow models can have multiple inputs and multiple outputs. For example, a model can have 2 inputs (X1, X2) and 2
outputs (Y1, Y2). In this case, the data preprocessing and batch generation should be done in a way that the data is
scaled and encoded for each input and output separately, as well as split into train/test keeping the input and target
data aligned. This is not possible with the current Keras API or existing libraries.Solution to above problemskeras_generators.generators.TimeseriesDataSourceis able to:split the data train/test/val split usingTimeseriesDataSource.splitand splitters fromkeras_generators.splittersmoduleperform automatic encoding/scaling usingkeras_generators.encoders.DataEncoderinstances as parametersdecode/denormalize the predicted datagenerate multi-step target data usingTimeseriesTargetsParamsas parameter.keras_generators.generators.DataSetis able to:aligned split of multi-input/multi-output data into train/test/val sets, and in the same time prform
fit-encoding/scaling of train data and use the fit scalers and encoders for validation and test data.keras_generators.generators.XBatchGeneratorandkeras_generators.generators.XBatchGeneratoris able to:generate batches of data for inference (XBatchGenerator) and training (XYBatchGenerator) for
multi-input/multi-output modelsperform re-shuffling of the data at the end of the epochAll the above classes are used in a pipeline, and you can find an example of their usage in the example
modelhere.ExampleGenerate multi-input/multi-step output Neural Network model.Multiple Input: multi-variate timeseries + tabular dataOutput: Multi-step timeseries target (predicting 3 data-points ahead), with stride 2, on second timeseries input(target_idx=1).# input_df - input Dataframe with multi-variate timeseries dataprice_input_ds=TimeseriesDataSource(name="input",tensors=input_df.values,length=60,target_params=TimeseriesTargetsParams(delay=0,pred_len=3,stride=2,target_idx=1))# Z-score scale data (input & output)encoded_input_ds=price_input_ds.encode([ScaleEncoder(StandardScaler())])# get targets with delay of 1 and prediction length of 3targets_ds=TargetTimeseriesDataSource.from_timeseries_datasource(encoded_input_ds,name="target")# tabular_df - Dataframe with tabular data. We scale data here with MinMax scalertabular_ds=TensorDataSource(name="tabular",tensors=tabular_df.values).encode([ScaleEncoder(MinMaxScaler())])# Get train/val/test generators for Kerasdataset=DataSet(input_sources={encoded_input_ds.name:encoded_input_ds,"tabular":tabular_ds},target_sources={targets_ds.name:targets_ds})train_ds,val_ds,test_ds=dataset.split(splitter=OrderedSplitter(train=0.6,val=0.2))train_gen=XYBatchGenerator(train_ds.input_sources,train_ds.target_sources,batch_size=32)val_gen=XYBatchGenerator(val_ds.input_sources,val_ds.target_sources,batch_size=1024)test_gen=XYBatchGenerator(test_ds.input_sources,test_ds.target_sources,batch_size=1024)# Train modelhistory=model.fit(train_gen,epochs=20,validation_data=val_gen,)# Inference and de-normalize/decode the predictionsy_pred=model.predict(test_gen.get_X_generator())res_ds=TensorDataSource(name="prediction",tensors=y_pred,encoders=targets_ds.get_encoders())unscaled_y_pred=res_ds.decode()[:]