Package: redmind
RedMindThis is a python library made to help you build machine learning models.Developed by Diego Velez 2022There are some known issues with softmax and adam optimizerInstallationpip3installredmindQuickstart (XOR sample)importmatplotlib.pyplotaspltimportredmind.optimizersasoptimfromredmind.layersimportDense,Sigmoid,ReLUfromredmind.networkimportNeuralNetworkfromredmind.lossimportBinaryCrossEntropyLossfromredmind.trainerimportTrainerimporttorch# Prepare dataxor=torch.tensor([[0,0],[0,1],[1,0],[1,1]],dtype=torch.float32)y=torch.tensor([0,1,1,0],dtype=torch.float32).reshape(1,4)# xor = torch.tensor([[0,1]], dtype=torch.float32)# y = torch.tensor([1], dtype=torch.float32).reshape(1,1)x_test=xor.T# Build NNn_weights_1=3# 3 neurons in the first layern_weights_2=1# 1 neuron in the second layer (output)nn=NeuralNetwork(layers=[Dense(n_weights_1,x_test.shape[0],seed=1),ReLU(),Dense(n_weights_2,n_weights_1,seed=1),Sigmoid()])learning_rate=1e-1epochs=100loss_fn=BinaryCrossEntropyLoss()optimizer=optim.RMSprop(nn.layers_parameters(),learning_rate=learning_rate)# Initialize trainertrainer=Trainer(network=nn,loss_function=loss_fn,optimizer=optimizer)# Traintrainer.train(X=x_test,Y=y,epochs=epochs,batch_size=1)# Predictprediction_vector=nn.predict(torch.tensor([[1.],[0.]]))ifprediction_vector>0.5:print(1)else:print(0)Go tosamplesfolder for more samplesYou can also opt to not use theTrainerclass and manually train the network, here is how to do itManual Train (XOR sample)importmatplotlib.pyplotaspltimportredmind.optimizersasoptimfromredmind.layersimportDense,Sigmoid,ReLUfromredmind.networkimportNeuralNetworkfromredmind.dataloaderimportDataloaderfromredmind.lossimportBinaryCrossEntropyLossimporttorch# Prepare dataxor=torch.tensor([[0,0],[0,1],[1,0],[1,1]],dtype=torch.float32)y=torch.tensor([0,1,1,0],dtype=torch.float32).reshape(1,4)# xor = torch.tensor([[0,1]], dtype=torch.float32)# y = torch.tensor([1], dtype=torch.float32).reshape(1,1)x_test=xor.T# Build NNn_weights_1=3# 3 neurons in the first layern_weights_2=1# 1 neuron in the second layer (output)nn=NeuralNetwork(layers=[Dense(n_weights_1,x_test.shape[0],seed=1),ReLU(),Dense(n_weights_2,n_weights_1,seed=1),Sigmoid()])# Load data in dataloader so we can loop itdata=Dataloader(x_test,y,batch_size=2)# training variableslearning_rate=1e-1epochs=600costs={}loss_fn=BinaryCrossEntropyLoss()optimizer=optim.GradientDescent(nn.layers_parameters(),learning_rate=learning_rate)# Manual trainforepochinrange(epochs):epoch_losses=[]forx,yindata:# forwardy_pred=nn.forward(x)# clear gradientsoptimizer.zero_grad()# calculate lossloss=loss_fn(y,y_pred)epoch_losses.append(loss.detach())loss.backward()# Gradient descent stepoptimizer.step()# Calculate total run costcosts[epoch]=torch.stack(epoch_losses).mean().item()accuracy=round(100-(costs[epoch]*100),3)print(f"epoch:{epoch+1}/{epochs}, cost:{round(costs[epoch],4)}, accuracy:{accuracy}%")Loss functionsYou can use different loss functions and even create your own, you just need to send the function as an argument to theTrainerasloss_functionlearning_rate=1e-1loss_fn=CategoricalCrossEntropyLoss()optimizer=optim.RMSprop(nn.layers_parameters(),learning_rate=learning_rate)# Initialize trainertrainer=Trainer(network=nn,loss_function=loss_fn,optimizer=optimizer)Defining custom loss functionIf you want to create your own loss function, you will need to inherit from the baseLosssuperclass and implement the__call__methodfromredmind.lossimportLossclassCustomLoss(Loss):def__call__(self,y_pred,y):...custom_loss=CustomLoss()...trainer=Trainer(network=nn,loss_function=custom_loss,optimizer=optimizer)OptimizersRedmind has support for different optimizers.Native supported optimizersGradientDescentMomentumRMSpropAdam (pending fix)Using a different OptimizerThe default optimizer is Gradient Descent, however you can change it.The optimizer object expects the NeuralNetwork as argument, so it can read the network layersimportredmind.optimizersasoptim...nn=NuralNetwork(...)adam=optimizer.Adam(nn.layers_parameters(),learning_rate=1e-2)trainer=Trainer(network=nn,loss_function=loss_fn,optimizer=optimizer)Creating your own optimizerYou can create your own optimizer, you just need to inherit from the Optimizer classfromredmind.optimizersimportOptimizer...classCustomOptimizer(Optimizer):def__call__(self)->None:forlayerinself.params:forparam_name,param_valueinlayer.items():direction=...# your learning algorithm# make sure the in place operation runs with no_gradwithtorch.no_grad():layer[param_name]-=directionnn=NeuralNetwork(...)myCustomOpt=CustomOptimizer(nn.layers_parameters(),learning_rate=1e-2)trainer=Trainer(network=nn,loss_function=loss_fn,optimizer=myCustomOpt)Save and Load ModelsYou can also save and load your trained models, this makes easy for you to
package, ship and use your models everywhere you want.Save modelfromredmind.utilsimportsave_model...nn=NeuralNetwork(...)# Create trainer objecttrainer=Trainer(network=nn,loss_function=loss_fn,optimizer=optimizer)# Traintrainer.train(X=x_test,Y=y,epochs=epochs,batch_size=64)# Save NN modelsave_model(nn,filename='bigNN.dill')Load modelfromredmind.utilsimportload_model# Load pretrained modelnn=load_model(filename='bigNN.dill')# predictnn.predict(x_test)Learning Rate DecayTheTrainerclass also supports learning_rate decay.fromredmind.functionsimportlr_decay...nn=NeuralNetwork(...)# Create trainer objecttrainer=Trainer(network=nn,learning_rate=0.01,lr_decay_function=lr_decay,decay_rate:0.1)# Traintrainer.train(X=x_test,Y=y,epochs=600,batch_size=1)PreprocessingRedmind also has a few handy preprocessing tools. These are tools to make your life a bit easier to handle dataNormalizer (pending fix)In case the features of your data have very high variance or are scaled in different way, normalizing makes it
fit between 0 and 1 (mostly). This is very useful to make your model train faster and avoid exploding gradientsUsage:importnumpyasnpfromredmind.normalizerimportNormalizer# column 1 Age# Column 2 Weightxtrain=np.array([[10,40],[11,35],[12,40],[13,41],[13,70],[15,60],[19,64],[15,60],[20,80],[40,100],[56,85]])# Initialize normalizer and fit the datanorm=Normalizer()norm.fit(xtrain)# scale xtrainxtrain=norm.scale(xtrain)xtest=np.array([[20,60],[21,75],[22,80],[23,59],[23,85],[25,77]])# no need to refit the normalizer for new data# You need to use the same scalexnorm=norm.scale(xtest)DataloaderThe dataloader is a useful tool to loop through the trainig examples and its labels.
It can also split your data in mini-batches very easily.Note: Make sure your data is entered as column vectorsfromredmind.dataloaderimportDataloaderxor=torch.tensor([[0,0],[0,1],[1,0],[1,1]])y_train=torch.tensor([0,1,1,0]).reshape(1,4)# we need to input data as column vectors to dataloaderx_train=xor.Tdata=Dataloader(X=X,Y=Y,batch_size=2)# then we can loop over the mini-batches# you can do forward and backpropagation like thisforx,yindata:print(x)print(y)#forward.....FeaturesClasses definition and constructionForward propagation fully workingBackward propagation workingTrain and predict fully workingAdd Optimization layersAdd mini batch Gradient descent (through Dataloader)Support for multiple optimizersLearning rate decayAdd early stoping supportSave and Load modelsBatch normalizationAdd convolutional layers