Package: nets
1. Table of Content1. Table of Content2. Overview2.1. About2.2. Requirements2.3. Installation3. Status4. Documentation5. Get Started5.1. Computational Graph5.2. Building a model6. Notebooks7. References2. Overview2.1. AboutNETSis a light-weight Deep LearningPythonpackage, made using only (mostly)numpy.
This project was first introduced as an assignment at theUniversity of Oslo, which is similar to the second
assignment fromStanford University.However, this project was pushed further to make itOOPwith an easier API.
In addition, the back-propagation and update rules where changed, using a customautogradsystem.NETSwas highly inspired fromPyTorchandTensorFlowpackages.But why ?NETSpackage hasNO CLAIMSto shadow already well build deep learning packages likePyTorchorTensorFlow. Instead, this package was made to understand how all of these libraries work and handle
forward / backward propagation by making one from scratch.
As I am going through thisdeepunderstanding, I found interesting to share
as much as possible my work, which I hope will help students or people who want to learn more about this subject.2.2. RequirementsAll packages withinNETSare made from scratch, using mainlynumpy. However, some additional
packages can offer a better experience if installed (saving checkpoints and models for example).numpyjson(Optional)time(Optional)pandas(Optional)scipy(Optional)sklearn(Optional)2.3. InstallationTo install this package fromPyPi$pipinstallnetsor from this repository$gitclonehttps://github.com/arthurdjn/nets$cdnets$pipinstall.3. StatusDevelopmentStatusFeatureAutograd Systemfinished[x] Tensor[x] ParameterOptimizationfinished[x] SGD[x] Adam[x] RMSpropLossin progress[x] MSE[x] Cross Entropy[ ] BCESolverfinished[x] Train[x] Eval[x] CheckpointsDatafinished[x] Dataset[x] Batch[x] IteratorDense Neural Networkfinished[x] Linear[x] SequentialConvolutional Neural Networkfinished[x] Conv2d[x] MaxPool2d[x] DropoutRecurrent Neural Networkin progress[x] RNN[ ] LSTM[ ] GRU4. DocumentationThe documentation and tutorials are in process and will be released soon.
You will find some tutorials and application on how to get started or build a similar package.5. Get StartedNETSarchitecture follows the one fromPyTorch.
It provides a basic neural network structure so you can create your own with numpy. You will need to
wrap your arrays in aTensorclass to keep track of the gradients, just like inPyTorch.5.1. Computational GraphNETSuses a forward & backward pass for gradient descent optimizations (NOTE: there are now other optimizers !).You can also uses the autograd system (recommended). It behaves similarly as Pytorch, except it is entirely done with NumPy.importnetst1=nets.Tensor([1,2,3],require_grad=True)t2=nets.Tensor([4,5,6])t3=t1+t2# t3 now requires gradientt3=t3.sum()# Compute the gradients for t1t3.backward()5.2. Building a modelA model is aModulesubclass, where biases, weights and parameters transformations are computed.
All modules have aforwardmethod, that MUST be overwritten.
This method will compute the forward propagation from an input tensor, and compute the transformation.
If using theautogradsystem, no back-propagation need to be added. However,
if you prefer to manually compute the gradients, you will need to override thebackwardmethod.YourModelshould inherits from theModuleclass and override
theforwardmethod.importnetsimportnets.nnasnnclassModel(nn.Module):"""Create your own model.The attributes should be your submodels used during the forward pass.You don't have to necessary affect the activation function as an attribute,unless you want to set a manual backward pass."""def__init__(self,input_dim,hidden_dim,output_dim):# Initializationsuper().__init__()# Don't forget to add this lineself.layer1=nn.Linear(input_dim,hidden_dim)self.layer2=nn.Linear(hidden_dim,hidden_dim)self.layer3=nn.Linear(hidden_dim,output_dim)defforward(self,inputs):# Forward passout1=nets.tanh(self.layer1(inputs))out2=nets.tanh(self.layer2(out1))returnself.layer3(out2)model=Model(10,100,2)# Let's check the architecturemodelOut:Model((layer1): Linear(input_dim=10, output_dim=100, bias=True)(layer2): Linear(input_dim=100, output_dim=100, bias=True)(layer3): Linear(input_dim=100, output_dim=2, bias=True))Again, this is really similar to whatPyTorchoffers.6. Notebooks0 -Getting Started with NETSThis tutorial highlights the main part and modules ofNETS.1 -Build an Autograd System with NumPyTo be released2 -Build a Feed Forward Neural Network with NumPyBuild a Fully Connected Neural Network for image classification.3 -Build a Convolutional Neural Network with NumPyBuild a Convolutional Neural Network for image classification.7. ReferencesHere is a list of tutorials and lectures/assignment that helped to developNETSPyTorch documentationPyTorch autograd tutorialJoel Grus autograd tutorialJoel Grus autograd live codingStanford University cs231n 2nd assignmentUniversity of Oslo in5400 1st assignment