Package: pygrad
PyGradPyGrad is aNumPy-based pure Python mini automatic differentiation library. Adopting a define-by-run approach, PyGrad provides a clear, intuitive interface for constructing computational graphs, calculating gradients, and solving optimization problems. Building on top of these strengths as an autograd library, PyGrad also offers a minimal neural network API, inspired heavily byPyTorch.InstallationPyGrad is available on PyPi.pip install pygradTo build and develop from source, clone this repository viagit clone https://github.com/jaketae/pygrad.gitQuick StartFull documentation to be made available atPyGrad docs.Variable ClassPyGrad's main data class isVariable, through which computational graphs can be created.>>>frompygradimportVariable>>>a=Variable(5)>>>b=Variable(3)>>>a+bVariable(8)PyGrad can, of course, also deal with arrays and tensors.>>>m1=Variable([[1,2],[3,4]])>>>m2=Variable([[1,1],[1,1]])>>>m1+m2Variable([[23][45]])Since PyGrad uses NumPy as its backend, it supports broadcasting as well as other matrix operations including transpose, reshape, and matrix multiplication.>>>m1.TVariable([[13][24]])>>>m1.reshape(1,4)Variable([[1234]])Gradient ComputationUnder the hood, PyGrad creates computational graphs on the fly when operations are performed on aVariableinstance. To obtain gradient values, simply call thebackward()method on a leaf variable.>>>x=Variable(3)>>>y=x*x>>>y.backward()>>>x.gradVariable(6)By default, PyGrad deletes gradients for intermediary variables to efficiently utilize available memory. To keep the gradient values for all variables involved in the computation graph, setretain_grad=Truein thebackward()call. In the example below, we calculate, whereis an intermediary variable.>>>x=Variable(3)>>>y=x*x>>>z=y+y>>>z.backward(retain_grad=True)>>>z.gradVariable(1)>>>y.gradVariable(2)>>>x.gradVariable(12)Neural NetworksPyGrad offers an intuitive interface for building and training neural networks. Specifically, ANN-related components live in thepygrad.nnmodule. For those who are already familiar with PyTorch will immediately see that PyGrad's API is no different from that of PyTorch. We hope to add more flavor to PyGrad's API in the days to come.Network InitializationBelow is a simple example of how to declare a neural network in PyGrad.frompygradimportnnfrompygradimportfunctionsasFclassNeuralNet(nn.Module):def__init__(self,num_input,num_hidden,num_class,dropout):self.fc1=nn.Linear(num_input,num_hidden)self.fc2=nn.Linear(num_hidden,num_class)self.dropout=nn.Dropout(dropout)defforward(self,x):x=self.fc1(x)x=F.relu(x)x=self.fc2(x)x=self.dropout(x)returnxpygrad.nnincludes layers and theModuleclass through which neural networks can be initialized. Also noteworthy ispygrad.functions, which includes activation functions, trigonometric functions, as well as a host of other basic operations such as reshape and transpose.DataLoader ClassPyGrad'sDataLoaderclass allows basic batching and shuffling functionality to be applied topygrad.data.Datasetinstances. Here, we assume a customDatasetinstance, calledCatDogDataset.frompygrad.dataimportDataLoader,ratio_splitBATCH_SIZE=8dataset=CatDogDataset()train_ds,test_ds=ratio_split(dataset,0.8,0.2)train_loader=DataLoader(train_ds,BATCH_SIZE)test_loader=DataLoader(test_ds,BATCH_SIZE)DataLoaderinstances can be iterated as follows:fordata,labelsintrain_loader:# training logic hereOptimizersPyGrad offers a number of different optimizers. These includepygrad.nn.optim.SGDpygrad.nn.optim.AdaGradpygrad.nn.optim.AdaDeltapygrad.nn.optim.AdamTraining a model with optimizers can be done by instantiating annn.optim.Optimizerobject with some model's parameters. After callingbackward()on a loss value, simply callstep()on the optimizer to update the target model's parameters.# imports assumedoptimizer=nn.optim.Adam(model.params())fordata,labelsintrain_loader:pred=model(data)loss=F.softmax_cross_entropy(pred,labels)loss.backward()optimizer.step()optimizer.zero_grad()The optimizer will update the model's weights according to the gradient values of each parameter.CreditsPyGrad is heavily based uponDeZero, an educational library introduced inDeep Learning from Scratch 3. Much of PyGrad's initial code base was adapted from DeZero. The design language of PyGrad's neural network API was inspired by and borrowed from PyTorch.Chaineris also worthy of mention as well, as DeZero itself also adapted many features from Chainer. Last but not least, PyGrad would not have been made possible without NumPy. Our acknowledgement goes to all the developers who put their time and effort into developing the aforementioned libraries.LicenseReleased under theMIT License.