Package: pymodconn
pymodconn: A Python package for developing modular sequence to sequence control-oriented deep neural networksA control-oriented deep neural network (CONN) is a deep neural network designed to predict the future behaviour and response of complex dynamic systems, taking into accountKnown past dataandKnown future data.Known past dataincludes historical information such as weather data, time information, system dynamics, and control inputs, which are used to train the CONN to understand the relationships between these variables and their effects on the system.Known future dataencompasses forecasted weather data, time information, schedules, and most importantly targeted control inputs, representing the expected conditions and desired actions that the system will be subject to in the future.The CONN leverages this information to predict future, i.e.,Unknown future data, referring to the system dynamics or states that are not yet known. This prediction enables the control system to anticipate changes in the environment and adapt its actions, accordingly, ensuring optimal performance and robustness in spite of uncertainties.ContentsInstallationRequirementsUsageConfiguration file detailsFull implementation example - Model developement and training using KerasInstallationpymodconncan be installed usingpip, a tool for installing Python packages. To do it, run the following command:pip install pymodconnRequirementspymodconnrequires Python >= 3.9.16 or later to run. Note that you should have also the following packages installed in your system:jsonschema==4.17.3numpy==1.24.3PyYAML==6.0ruamel.base==1.0.0tensorflow==2.12.0UsageStep 1: Loading the configurations as a dictionaryThe users can design their CONN using a text-based configuration file. The details of this configuration file is discussed in next section (Configuration file details). The configuration file can be loaded as a dictionary for use usingget_configs(). User can furthur change contents of configuration dictionary before using it to develop Keras modelfrompymodconn.configs.configs_initimportget_configsMODEL_CONFIG_FILENAME='default_config.yaml'configs=get_configs(MODEL_CONFIG_FILENAME)configs['n_past']=25#Number of past observations timesteps to be considered for predictionconfigs['n_future']=10#Number of future predictions timesteps to be madeconfigs['known_past_features']=2#Number of features in the past observation window dataconfigs['known_future_features']=3#Number of features in the future prediction window dataconfigs['unknown_future_features']=2#Number of features in the future prediction window dataStep 2: Develop Keras model using configurations dictionaryTo differentiate model details the user can choose to use a unique indentation. For example this unique identifier string (ident) is created by appending the current time string to a predefined text. This identifier is used to differentiate between multiple runs or cases and for generating unique filenames when saving the model and prediction results.Finally to develop the Keras model, theModel_Gen(configs, ident)class is instantiated with the loaded configurations and the current run's identifier. Thebuild_model()method is then called to create the Keras model object.The model returned frombuild_model()can be used similarly to a Keras-based model. The user can runkeras.model.fit(),keras.model.train_on_batch(),keras.model.evaluate(), orkeras.model.save()as needed. Afull implementation exampleis shown later.frompymodconnimportModel_Gen# 'ident' is a string used to ensure unique file and prediction case namesident='test_'current_run_dt=ident+str(dt.datetime.now().strftime('%d.%m-%H.%M.%S'))# Initialize and build the model usingmodel_class=Model_Gen(configs,ident)model_class.build_model()model_class.model.fit(....)model_class.model.evaluate(....)model_class.model.save(....)Configuration file detailsAs discussed in previous section, users can design their CONN using a text-based .yaml file. YAML ("YAML Ain't Markup Language") is a human-readable data serialization format that is commonly used for configuration files due to its simplicity and readability. It facilitates implementation with various network architectures, promotes modularity and enables the reuse of predefined models, thus improving efficiency and reducing coding errors. The approach simplifies hyperparameter tuning, a critical aspect of neural network performance, by isolating these parameters in a configuration file.This file can be generated by providing config_filename = None in the get_configs(config_filename) function or obtained from the package's GitHub repositorydefault_config.yaml.The first option creates a template of the configuration file in the current directory of the Python program, which users can modify with the desired parameters. With the latter approach, users can download a template configuration file, along with an assortment of config files corresponding to various predefined model architectures. The users can use config files for predefined model architectures as starting point.To avoid errors and ensure correct operation, it is recommended that users do not alter the key names or the data types of the values in the key-value pairs within the configuration file. Maintaining this structure is essential for the Python package to function correctly. Following section gives more details of YAML configuration file lin-by-line.ALL BASIC SETTINGS INDEPENDENT OF MODEL ARCHITECTUREif_model_image:1if_model_summary:0if_seed:0seed:500model_type_prob:probloss_prob:nonparametricloss:mean_squared_errorquantiles:[0.01,0.05,0.1,0.5,0.9,0.95,0.99]metrics:[mape,acc]optimizer:AdamAdam:lr:0.001b1:0.9b2:0.999epsi:1.0e-08SGD:lr:0.001momentum:0.9save_models_dir:saved_models/batch_size:32n_past:672n_future:96known_past_features:31unknown_future_features:1known_future_features:15all_layers_neurons:128all_layers_dropout:0.4if_model_image:When set to 1, the package saves a visual representation of your model architecture as an image file. A 0 disables this feature.if_model_summary:Setting this to 1 enables the package to print a summary of your model architecture. Set it to 0 to disable this feature.if_seed:If this value is 1, a specific seed for random number generation is used, ensuring the reproducibility of your model. Set it to 0 to disable this.seed:If 'if_seed' is set to 1, this value specifies the seed for random number generation. Change it as needed.model_type_prob:Determines the type of model outputs. 'prob' signifies that probabilistic (interval) predictions will be generated.loss_prob:This field allows you to set the type of loss function for probabilistic prediction. 'nonparametric' and 'parametric' are the options, derived from DeepTCN.loss:This parameter sets the loss function for your model. Here, 'mean_squared_error' is used.quantiles:If using a quantile loss function for probabilistic forecasts, specify the quantiles you wish to estimate here.metrics:Here you specify the metrics that will be used to evaluate the model's performance.optimizer:This parameter allows you to set the optimizer for your model. In this example, the 'Adam' optimizer is used.Adam:This section lets you specify the hyperparameters for the Adam optimizer: learning rate (lr), beta 1 (b1), beta 2 (b2), and epsilon (epsi).SGD:Here you can specify the hyperparameters for the SGD optimizer if used: learning rate (lr) and momentum.save_models_dir:This is the directory where your trained models will be saved.batch_size:This parameter sets the number of samples to work through before the model’s internal parameters are updated.n_past, n_future, known_past_features, unknown_future_features, known_future_features:These parameters specify the number of past and future time steps, and the number of known past, known future and unknown future features, respectively.all_layers_neurons:This field specifies the number of neurons for all layers.all_layers_dropout:This parameter sets the dropout rate for all layers, a regularization technique to prevent overfitting.The CONN structure includes an Encoder block processing known past data, and one or more Decoder blocks handling known future data. The Encoder block generates a sequence of encoded states reflecting the temporal structure of past data, and a final state summarizing this data. Each Decoder block receives these outputs; the encoded states serve as a temporal context, while the final encoder state initializes the Decoder block.ENCODER SETTINGSThis section of config file is used to list settings used to structure theEncoderof the model.# DETAILED MODEL SETTINGS# ENCODER SETTINGSencoder:TCN_input:IF_TCN:1IF_NONE_GLUADDNORM_ADDNORM_TCN:1#0: None, 1: GLUADDNORM, 2: ADDNORMkernel_size:3nb_stacks:2dilations:[1,2,4,8,16,32]RNN_block_input:# settings for RNN block outside the block classIF_RNN:1IF_NONE_GLUADDNORM_ADDNORM_block:1#0: None, 1: GLUADDNORM, 2: ADDNORMIF_GRN_block:1#0: no GRN, 1: GRN# settings for RNN units inside the blocksrnn_depth:3rnn_type:GRU# GRU, LSTM, SimpleRNNIF_birectionalRNN:1# 0: no bidirectional, 1: bidirectionalIF_NONE_GLUADDNORM_ADDNORM_deep:1#0: None, 1: GLUADDNORM, 2: ADDNORMself_MHA_block:# settings for self MHA block outside the block classMHA_depth:3# settings for self MHA units inside the blocksIF_MHA:1IF_GRN_block:1#0: no GRN, 1: GRNMHA_head:8IF_NONE_GLUADDNORM_ADDNORM_deep:1#0: None, 1: GLUADDNORM, 2: ADDNORMencoder:TCN_input:IF_TCN:When set to 1, a Temporal Convolutional Network (TCN) is used in the input. A 0 disables this feature.IF_NONE_GLUADDNORM_ADDNORM_TCN:Determines the type of residual connection for the TCN layer. Use 0 for Nothing, 1 for an Add and Normalize residual connection with a GLU gate, and 2 for an Add and Normalize residual connection without a GLU gate.kernel_size:Specifies the size of the kernel used in the TCN layer.nb_stacks:Specifies the number of TCN stacks to be used.dilations:Lists the dilation factors for the TCN layers.RNN_block_input:IF_RNN:When set to 1, a Recurrent Neural Network (RNN) is used in the input. A 0 disables this feature.IF_NONE_GLUADDNORM_ADDNORM_block:Determines the type of residual connection for the RNN block. Use 0 for Nothing, 1 for an Add and Normalize residual connection with a GLU gate, and 2 for an Add and Normalize residual connection without a GLU gate.IF_GRN_block:Sets whether Gated Recurrent Unit (GRN) is used in the block. Set it to 0 for no GRN, and 1 for GRN.rnn_depth:Specifies the depth of the RNN units.rnn_type:Sets the type of RNN units used. Options are GRU, LSTM, and SimpleRNN.IF_birectionalRNN:Specifies whether the RNN units are bidirectional. Set it to 0 for no bidirectionality, and 1 for bidirectional.IF_NONE_GLUADDNORM_ADDNORM_deep:Determines the type of residual connection for the deeper layers. Use 0 for Nothing, 1 for an Add and Normalize residual connection with a GLU gate, and 2 for an Add and Normalize residual connection without a GLU gate.self_MHA_block:MHA_depth:Specifies the depth of the multi-head attention (MHA) units.IF_MHA:When set to 1, a Multi-Head Attention (MHA) unit is used in the input. A 0 disables this feature.IF_GRN_block:Sets whether Gated Recurrent Unit (GRN) is used in the block. Set it to 0 for no GRN, and 1 for GRN.MHA_head:Specifies the number of heads in the MHA unit.IF_NONE_GLUADDNORM_ADDNORM_deep:Determines the type of residual connection for the deeper layers. Use 0 for Nothing, 1 for an Add and Normalize residual connection with a GLU gate, and 2 for an Add and Normalize residual connection without a GLU gate.DECODER SETTINGSThis section of config file is used to list settings used to structure theDecoderof the model.# DECODER SETTINGSdecoder:# TCN block_decoder input settingsTCN_input:IF_TCN:1IF_NONE_GLUADDNORM_ADDNORM_TCN:1#0: None, 1: GLUADDNORM, 2: ADDNORMkernel_size:3nb_stacks:2dilations:[1,2,4,8,16,32]# # RNN block_decoder input settingsRNN_block_input:# settings for RNN block outside the block classIF_RNN:1IF_NONE_GLUADDNORM_ADDNORM_block:1#0: None, 1: GLUADDNORM, 2: ADDNORMIF_GRN_block:0#0: no GRN, 1: GRN# settings for RNN units inside the blocksrnn_depth:3rnn_type:GRU# GRU, LSTM, SimpleRNNIF_birectionalRNN:1# 0: no bidirectional, 1: bidirectionalIF_NONE_GLUADDNORM_ADDNORM_deep:1#0: None, 1: GLUADDNORM, 2: ADDNORMCIT_option:1#1: Concatenation, 2: Luong or Bahdanau, 3: Self+cross MHA# Contextual Information Transfer - option 1 settingsIF_NONE_GLUADDNORM_ADDNORM_CIT_1:1#0: None, 1: GLUADDNORM, 2: ADDNORMoption_1_depth:3# Contextual Information Transfer - option 2 settingsIF_NONE_GLUADDNORM_ADDNORM_CIT_2:1#0: None, 1: GLUADDNORM, 2: ADDNORMattn_type:2# 1 is Luong attention, 2 is Bahdanau attentionoption_2_depth:3# Contextual Information Transfer - option 3 settingsIF_SELF_CROSS_MHA:1SELF_CROSS_MHA_depth:3self_MHA_block:# settings for self MHA units inside the blocksIF_MHA:1IF_GRN_block:1#0: no GRN, 1: GRNMHA_head:8IF_NONE_GLUADDNORM_ADDNORM_deep:1#0: None, 1: GLUADDNORM, 2: ADDNORMcross_MHA_block:# settings for cross MHA units inside the blocksIF_MHA:1IF_GRN_block:1#0: no GRN, 1: GRNMHA_head:8IF_NONE_GLUADDNORM_ADDNORM_deep:1#0: None, 1: GLUADDNORM, 2: ADDNORM# TCN block_decoder output settingsTCN_output:IF_TCN:1IF_NONE_GLUADDNORM_ADDNORM_TCN:1#0: None, 1: GLUADDNORM, 2: ADDNORMkernel_size:3nb_stacks:2dilations:[1,2,4,8,16,32]# RNN block_decoder output settingsRNN_block_output:# settings for RNN block outside the block classIF_RNN:1IF_NONE_GLUADDNORM_ADDNORM_block:1#0: None, 1: GLUADDNORM, 2: ADDNORMIF_GRN_block:1#0: no GRN, 1: GRN# settings for RNN units inside the blocksrnn_depth:3rnn_type:GRU# GRU, LSTM, SimpleRNNIF_birectionalRNN:1# 0: no bidirectional, 1: bidirectionalIF_NONE_GLUADDNORM_ADDNORM_deep:1#0: None, 1: GLUADDNORM, 2: ADDNORMMERGE_STATES_METHOD:4#-----about merging states from Encoder_states(A)# and Decoder_input_states(B) for Decoder_output init_states# there are 8 options:# 1: None# 2: A - Dense layer# 3: B - Dense layer# 4: A+B - Concat -> Dense layer# 5: A+B - Add -> Dense layer# 6: A+B - Add_Norm -> Dense layer# 7: A+B - Add# 8: A+B - Add_NormDECODER SETTINGSdecoder:TCN_input:(Refer to the previously provided TCN settings description)RNN_block_input:(Refer to the previously provided RNN settings description)CIT_option:Defines the method for Contextual Information Transfer. Options include: 1 for Concatenation, 2 for Luong or Bahdanau attention, and 3 for Self+cross MHA.IF_NONE_GLUADDNORM_ADDNORM_CIT_1:Determines the type of residual connection for the first CIT option. Use 0 for Nothing, 1 for an Add and Normalize residual connection with a GLU gate, and 2 for an Add and Normalize residual connection without a GLU gate.option_1_depth:Specifies the depth of the first CIT option.IF_NONE_GLUADDNORM_ADDNORM_CIT_2:Determines the type of residual connection for the second CIT option. Use 0 for Nothing, 1 for an Add and Normalize residual connection with a GLU gate, and 2 for an Add and Normalize residual connection without a GLU gate.attn_type:Specifies the type of attention mechanism to use. Option 1 is for Luong attention, and Option 2 is for Bahdanau attention.option_2_depth:Specifies the depth of the second CIT option.IF_SELF_CROSS_MHA:When set to 1, enables Self+cross MHA for the third CIT option. A 0 disables this feature.SELF_CROSS_MHA_depth:Specifies the depth of the Self+cross MHA for the third CIT option.self_MHA_block:(Refer to the previously provided self MHA block settings description)cross_MHA_block:(Refer to the previously provided self MHA block settings description. For this setting, it applies to cross MHA units.)TCN_output:(Refer to the previously provided TCN settings description)RNN_block_output:(Refer to the previously provided RNN settings description)MERGE_STATES_METHOD:Specifies the method for merging states from Encoder_states(A) and Decoder_input_states(B) for Decoder_output init_states. Options include:1: None2: A - Dense layer3: B - Dense layer4: A+B - Concat -> Dense layer5: A+B - Add -> Dense layer6: A+B - Add_Norm -> Dense layer7: A+B - Add8: A+B - Add_NormTo avoid errors and ensure correct operation, it is recommended that users do not alter the key names or the data types of the values in the key-value pairs within the configuration file. Maintaining this structure is essential for the Python package to function correctly.Full implementation example - Model developement and training using KerasWithin the same directory as the default_config.yaml file, three additional configuration files are accessible. These files represent CONNs built on three recognized time series prediction mechanisms: RNNs -CONN_based_on_biLSTMs_with_Bahdanau_attention.yaml, MHA -CONN_based_on_MHA.yaml, and TCN -CONN_based_on_TCN_with_Bahdanau_attention.yaml.
An implementation example is presented below which develops and tests the three different example model configuration.importmultiprocessingimportnumpyasnpimportsysfrompymodconnimportModel_Genfrompymodconn.configs.configs_initimportget_configsimportdatetimeasdtclassMultiprocessingWindow():"""Wrapper class to spawn a multiprocessing window for training and testingto avoid the issue of 'Tensorflow not releasing GPU memory'"""def__init__(self,function_to_run,its_args):self.function_to_run=function_to_runself.its_args=its_argsdef__call__(self):multiprocessing.set_start_method('spawn',force=True)print("Multiprocessing window spawned")p=multiprocessing.Process(target=self.function_to_run,args=self.its_args)p.start()p.join()deftrain_test(MODEL_CONFIG_FILENAME):# Load the configurations for the modelconfigs=get_configs(MODEL_CONFIG_FILENAME)configs['if_model_image']=0configs['n_past']=25#Number of past observations timesteps to be considered for predictionconfigs['n_future']=10#Number of future predictions timesteps to be madeconfigs['known_past_features']=2#Number of features in the past observation window dataconfigs['known_future_features']=3#Number of features in the future prediction window dataconfigs['unknown_future_features']=2#Number of features in the future prediction window data to be predicted# Generate random time series data for training and evaluation (sequence-to-sequence)num_samples=1000x_known_past=np.random.random((num_samples,configs['n_past'],configs['known_past_features']))x_known_future=np.random.random((num_samples,configs['n_future'],configs['known_future_features']))y_unknown_future=np.random.random((num_samples,configs['n_future'],configs['unknown_future_features']))# Split the data into training and testing sets using a basic Python functiontrain_test_split_percentage=0.8split_index=int(train_test_split_percentage*num_samples)x_train_known_past,x_test_known_past=x_known_past[:split_index],x_known_past[split_index:]x_train_known_future,x_test_known_future=x_known_future[:split_index],x_known_future[split_index:]y_train_unknown_future,y_test_unknown_future=y_unknown_future[:split_index],y_unknown_future[split_index:]# 'ident' is a string used to ensure unique file and prediction case namesident='test_'current_run_dt=ident+str(dt.datetime.now().strftime('%d.%m-%H.%M.%S'))# Initialize and build the modelmodel_class=Model_Gen(configs,ident)model_class.build_model()# Note: Model compilation happens inside Model_Gen.build_model() and is dependent on the user's choice.# Note: User can also compile model again using different optimizer and loss function# Train the modelmodel_class.model.fit([x_train_known_past,x_train_known_future],y_train_unknown_future,batch_size=128,epochs=3,validation_split=0.2)# Evaluate the modeltest_loss,test_accuracy=model_class.model.evaluate([x_test_known_past,x_test_known_future],y_test_unknown_future)print(f'Test loss:{test_loss}, Test accuracy:{test_accuracy}')# Save the model with a unique filename based on ‘current_run_dt'. Can use both keras.model.save_weights() or keras.model.save()model_class.model.save(f'{current_run_dt}_random_time_series_model.h5')if__name__=='__main__':MultiprocessingWindow(train_test,(['CONN_based_on_bi_directional_LSTMs_with_Bahdanau_attention.yaml']))()MultiprocessingWindow(train_test,(['CONN_based_on_multi_head_attention.yaml']))()MultiprocessingWindow(train_test,(['CONN_based_on_temporal_convolutional_network_with_Bahdanau_attention.yaml']))()Figure below illustrates the schematics of the Keras models generated from these three configuration files.LicenseMIT LicenseFree Software, Hell Yeah!