Package: iaflow
IAfLowThis library help to create models with identifiers, checkpoints, logs and metadata automatically, in order to make the training process more efficient and traceable.RequirementsTensorflow 2.5 or higher.notify-function1.5.0 or higher.UsageThis library has a maser class calledIAFlow, that has functions to management model creation and training, the most important are:add_model: to add a new model to the internal structure of maker models to build later to training.train: to train a model with a internal callbacks to save metrics with Tensorboard, save the model with checkpoints and send notification to discord channel, email or telegram (all this is optional).show_models: to show the models that are in the internal structure of maker models.delete_model: to delete a model from the internal structure of maker models.There are more methods to complete a CRUD for models and datasets. You can check it once you create a instance ofIAFlow. For now, you could check a brief description of most important methods below.ConstructorfromiaflowimportIAFlowdefcustom_builder(input_shape):model=tf.keras.models.Sequential([tf.keras.layers.Input(input_shape),tf.keras.layers.Dense(units=512,activation='relu'),tf.keras.layers.Dense(units=512,activation='relu'),tf.keras.layers.Dense(units=1,activation='sigmoid')])returnmodelparams_notifier={# Parameters for notifier, see documentation https://pypi.org/project/notify-function/#description'title':'Training update','webhook_url':os.environ.get('WEBHOOK_URL'),'frequency_epoch':20# This will send a notification every 20 epochs, by default it is every epoch}ia_maker=IAFlow(models_folder='./models',# Folder to save the modelsparams_notifier=params_notifier,# Notifier to send notification to discord channel, email or telegram (all this is optional)builder_function=custom_builder# Function to build the model, you can change this when call `train` method)add_datasetThis function add a new dataset to the internal structure of maker models to build later to training. The dataset could be an instance oftf.data.Dataset, list of Tensors of whatever that could be passes to thefitmethod oftf.keras.Model.ia_maker.add_dataset(name='dataset_1',# Name of the datasetepochs=10,# Number of epochs to train the model with this datasetbatch_size=32,# Batch size to train the model with this datasetshuffle_buffer=512,# Buffer size to shuffle the dataset when training with this datasettrain_ds=train_ds,# Dataset to train the model, you can change this when call `train` methodval_ds=val_ds# Dataset to validate the model, you can change this when call `train` method)So far, the shuffle buffer is only applied to datasets that are an instance oftf.data.Dataset.add_modelmethodThe build method is used to create a model in a defined structure of folder. As below:`models_folder`
└─ `model_name`
   └─ `run_id`
      ├─ logs                               
      │  ├─ train
      │  ├─ validation
      ├─ `model_name`_`model_params`_checkpoint.h5
      └─ `model_name`_params.jsonmodel_1_data=ia_maker.add_model(model_name='model_1',# Name of the modelrun_id='run_1',# Run id of the model, must be unique by model namemodel_params={'input_shape':(2,1)},# Parameters for builder functionload_model_params={},# Parameters for model loading, see documentation of tf.keras.models.load_modelcompile_params={# Parameters for model compilation, see documentation tf.keras.Models.compile'optimizer':'adam','loss':'mse','metrics':['accuracy']},)Note: If your model use custom SubClass for custom Layers you must send the parameterload_model_paramswith the parametercustom_objectswith the custom class. see documentation of tf.keras.models.load_modelReturnThis method return a dictionary with information of the modeltrainmethodThe train method is used to train a model. This doen't return anything. Inside the method the model will be built, compiled and finally begin to train.Parametersia_maker.train(model_1_data,batch=32,epochs=5)If you loaded a trained model you should send a dictionary with keymodel_nameandrun_idthat belong to the model desired to load and the initial epoch, also you can send the same dictionary returned when you calledadd_modelmethod.If you want usenotify-functionlib you must send the parameterparams_notifier. To see how to use and what do this library you can see the documentation ofnotify-function. The message will have the epoch number and the values of each metric that you defined for your model.If you usenotify-functionand you specify email maybe this will add a delay to the training process if your time per step is so fast. To avoid this you can use another methods more faster like discord webhooks or telegram message or instead use the keyfrequency_epochon theparams_notifierreduce the rate of notifications.show_modelsmethodThis function show all models that you have added to the IAFlow by a print.delete_modelmethodThis function delete a model from the IAFlow. You must send a dictionary with keymodel_nameandrun_idthat belong to the model desired to delete. If you send the parameterdelete_folderit will also delete the folder of the model.FAQsIf you have any question find a bug or feedback, please contact me with a email toenmanuelmag@cardor.devMade with ❤️ byEnmanuel Magallanes