Package: gustavgrad
gustavgradAn autograd library built on NumPy, inspired byJoel Grus's livecoding.InstallationWith pippipinstallgustavgradHow to use the libraryTensoroperationsTheTensorclass is the cornerstone ofgustavgrad.
It behaves much like an ordinarynumpy.ndarray.Tensorscan be added together,>>>fromgustavgradimportTensor>>>x=Tensor([1,2,3])>>>x+xTensor(data=[246],requires_grad=False)... subracted from each other,>>>x-xTensor(data=[000],requires_grad=False)... their dot-product can calculated,>>>x*xTensor(data=[149],requires_grad=False)... and they can be multiplied with each other.>>>y=Tensor([[1],[2],[3]])>>>x@yTensor(data=[14],requires_grad=False)Tensoroperations also support broadcasting:>>>x*3Tensor(data=[369],requires_grad=False)>>>z=Tensor([[1,2,3],[4,5,6]])>>>x*zTensor(data=[[149][41018]],requires_grad=False)Automatic backpropagationBut aTensoris not just anndarray, they also keep track of their own gradient.>>>speed=Tensor(1,requires_grad=True)>>>time=Tensor(10,requires_grad=True)>>>distance=speed*time>>>distanceTensor(data=10,requires_grad=True)If aTensoris created as the result of aTensoroperation involving aTensorwithrequires_grad=True,
the resultingTensorwill be able to backpropagate it's own gradient to it's ancestor.>>>distance.backward()>>>speed.gradarray(10.)>>>time.gradarray(1.)By calling thebackwardmethod ondistancethe gradient ofspeedandtimeis automatically updated.
We can see that increasingspeedby 1 would result in an increase indistanceby 10, while an increase
intimeby 1 would only increasedistanceby 1.TheTensorclass supports backpropagation over arbitrary compositions ofTensoroperations.>>>t1=Tensor([[1,2,3],[4,5,6]],requires_grad=True)>>>t2=Tensor([[1],[2],[3]])>>>t3=t1@t2+1>>>t4=t3*7>>>t5=t4.sum()>>>t5.backward()>>>t1.gradarray([[7.,14.,21.],[7.,14.,21.]])TheModuleAPIgustavgradprovides some tools to simplify setting up and training machine learning models.
TheModuleAPI makes it easier to manage multiple relatedTensorsby registering them asParameters.
AParameteris just a randomly initializedTensor.fromgustavgrad.moduleimportModule,Parameterfromgustavgrad.functionimporttanhclassMultilayerPerceptron(Module):def__init__(self,input_size:int,output_size:int,hidden_size:int=100)->None:self.layer1=Parameter(input_size,hidden_size)self.bias1=Parameter(hidden_size)self.layer2=Parameter(hidden_size,output_size)self.bias2=Parameter(output_size)defpredict(self,x:Tensor)->Tensor:x=x@self.layer1+self.bias1x=tanh(x)x=x@self.layer2+self.bias2returnxBy subclassingModuleourMultilayerPerceptronclass automatically gets some helper methods
for managing itsParameters.
Let's create aMultilayerPerceptronthat tries to learn the XOR function.xor_input=Tensor([[0,0],[0,1],[1,0],[1,1]])xor_targets=Tensor([[0],[1],[1],[0]])xor_mlp=MultilayerPerceptron(input_size=2,output_size=1,hidden_size=4)We can use the model to make predictions on thexor_inputTensor.>>>predictions=xor_mlp.predict(xor_input)>>>predictionsTensor(data=[[-1.79888385][-1.07965756][0.34373135][1.63366069]],requires_grad=True)The predictions of the randomly initialized model aren't right, but we can improve the model by
calculating the gradient of it'sParametersin respect to a loss function.fromgustavgrad.lossimportSquaredErrorLossse_loss=SquaredErrorLoss()loss=se_loss.loss(xor_targets,predictions)loss.backward()lossis aTensor, so we can call itsbackwardmethod to do backpropagation through ourxor_mlp.
We can then adjust the weights of allParametersinxor_mlpusing gradient descent:fromgustavgrad.optimimportSGDoptim=SGD(lr=0.01)optim.step(xor_mlp)After updating the weights we can reset the gradients of all parameters and make new predictions:>>>xor_mlp.zero_grad()>>>predictions=xor_mlp.predict(xor_input)>>>predictionsTensor(data=[[-1.51682686][-0.78583272][0.55994602][1.67962174]],requires_grad=True)Seeexamples/xor.pyfor a full example of howgustavgradcan be used to learn the XOR function.
Theexamplesdirectory also contains some other basic examples of how the library can be used.