Package: nn-without-frameworks
NN-Without-FrameworksThis project aims to implement different Neural Network configurationwithoutusing scientific frameworks likeTensorFloworPyTorch.Each network/config is implemented in 4 formats while trying to get the best of both worlds of TensorFlow (+ Keras) and PyTorch :In Python using NumPyIn Python without taking advantage of NumPyIn javaIn C++What is currently supported? (equally in all languages and formats)Layers:Fully ConnectedActivations:LinearReLULoss functions:MSECross EntropyWeight Initializers:Xavier Uniform (aka Glorot)He Normal (aka Kaiming Normal)bias initializer:Constant (zero)Optimizers:SGDSGD + MomentumRMSPropAdaGradAdamRegularizer:l1l2Normalization:BatchNorm1dExamplesEach directory contains atrain.*that performs tests of correctness and functionality according to its corresponding format and language. You can run it to get a sense of what is going on.### DQNSnippetDefine your networkPython:fromnn_without_frameworksimportnumpy_nnasnn# from nn_without_frameworks import pure_nn as nnclassMyNet(nn.Module):def__init__(self,input_dim,out_dim):super().__init__()self.input_dim=input_dimself.out_dim=out_dimself.hidden1=nn.layers.Dense(in_features=self.input_dim,out_features=100,activation=nn.acts.ReLU(),weight_initializer=nn.inits.HeNormal(nn.acts.ReLU()),bias_initializer=nn.inits.Constant(0.),regularizer_type="l2",lam=1e-3)self.output=nn.layers.Dense(in_features=100,out_features=self.out_dim,weight_initializer=nn.inits.XavierUniform(),bias_initializer=nn.inits.Constant(0.),regularizer_type="l2",lam=1e-3)defforward(self,x):x=self.hidden1(x)returnself.output(x)Java:importLayers.Dense;classMyNetextendsModule{intin_features=0,out_features=0;Densehidden1,output;publicMyNet(intin_features,intout_features){this.in_features=in_features;this.out_features=out_features;this.hidden1=newDense(this.in_features,100,"relu","he_normal","zeros",// bias initializer"l2",0.001F);this.layers.add(this.hidden1);// the Crucial and only different part to PyTorch's subclassingthis.output=newDense(100,out_features,"linear","xavier_uniform","zeros",// bias initializer"l2",0.001F);this.layers.add(this.output);// Crucial and different part to PyTorch's subclassing}publicfloat[][]forward(float[][]x){x=this.hidden1.forward(x);x=this.output.forward(x);returnx;}}C++:#include<iostream>#include<module.h>#include<layers.h>usingnamespacestd;classMyNet:publicModule{public:intin_features=0,out_features=0;Dense*hidden,*output;// Layers should be predifned especially, they should be pointersMyNet(intin_features,intout_features){this->in_features=in_features;this->out_features=out_features;this->hidden=newDense{this->in_features,100,"relu","he_normal","zeros",// bias initializer"l2",0.001};this->parameters.push_back(this->hidden);// same as javathis->output=newDense{100,this->out_features,"linear","xavier_uniform","zeros",// bias initializer"l2",0.001};this->parameters.push_back(this->output);sameasjava}float_batchforward(constfloat_batch&input){// float_batch =: vector<vector<float> >float_batchx=this->hidden->forward(input);x=this->output->forward(x);returnx;}};Train your networkPythonmy_net=MyNet(num_features,num_classes)ce_loss=nn.losses.CrossEntropyLoss()opt=nn.optims.SGD(my_net.parameters,lr=1.)forstepinrange(epoch):y=my_net(x)loss=ce_loss(y,t)my_net.backward(loss)opt.apply()Java:importLosses.*;importOptimizers.*;MyNetmy_net=newMyNet(num_features,num_classes);CrossEntropyLossceloss=newCrossEntropyLoss();SGDopt=newSGD(1.0F,my_net.layers);for(intepoch=0;epoch<num_epoch;epoch++){y=my_net.forward(x);Lossloss=celoss.apply(y,t);my_net.backward(loss);opt.apply();}C++:#include<losses.h>#include<optimizers.h>MyNetmy_net=MyNet{num_features,num_classes};CrossEntropyLossceloss;SGDopt(1,my_net.parameters);float_batchy;// float_batch =: vector<vector<float> >for(intstep=0;step<num_epoch;step++){y=my_net.forward(x);Lossloss=celoss.apply(y,t);my_net.backward(loss);opt.apply();}AcknowledgementCurrent code is inspired by the elegant and simple repositorySimple Neural Networksby@MorvanZhou.Mathematical foundation of different parts is based on slides ofCS W182 / 282AandCS231ncourses.ContributingThe current code is far from done and any fix, suggestion, pull request, issue, etc is highly appreciated and welcome. 🤗Current work focuses on discovering what is under the hood rather optimization involved in implementing ideas so, feel free to conduct sanity-checks behind the math and correctness of each part and/or if you come up with a better or optimized solution, please don't hesitate to bring up a PR. [thank you in advance. 😊]You can take a look attodo.md.