Package: crysx-nn
crysx_nnA simplistic and efficient pure-python neural network library from Phys Whiz with CPU and GPU support.Explore the docs Â»View DemoÂ·Report BugÂ·Request FeatureTable of ContentsAbout The ProjectBuilt WithGetting StartedPrerequisitesInstallationUsageFeaturesRoadmapContributingLicenseContactAcknowledgmentsCitationAbout The ProjectNeural networks are an integral part of machine learning.
The project provides an easy-to-use, yet efficient implementation that can be used in your projects or for teaching/learning purposes.The library is written in pure-python with some optimizations using numpy, opt_einsum, and numba when using CPU and cupy for CUDA support.The goal was to create a framework that is efficient yet easy to understand, so that everyone can see and learn about what goes inside a neural network. After all, the project did spawn from a semester project onCP_IV: Machine Learning courseat theUniversity of Jena, Germany.(back to top)Built WithCupy(Optional)NumPy(Required)numba(Required)opt_einsum(Required)matplotlib(Required)nnv(Required)pillow(Optional)zipfile(Optional)(back to top)Getting StartedTo get a local copy up and running follow these simple example steps.PrerequisitesYou need to havepython3installed along withpip.InstallationThere are many ways to installcrysx_nnInstall the release (stable) version fromPyPipipinstallcrysx_nnInstall the latest development version, by cloning the git repo and installing it.
This requires you to havegitinstalled.gitclonehttps://github.com/manassharma07/crysx_nn.gitcdcrysx_nn
pipinstall.Install the latest development version withoutgit.pipinstall--upgradehttps://github.com/manassharma07/crysx_nn/tarball/mainCheck if the installation was successful by running python shell and trying to import the packagepython3Python3.7.11(default,Jul272021,07:03:16)[Clang10.0.0]::Anaconda,Inc.ondarwinType"help","copyright","credits"or"license"formoreinformation.>>>importcrysx_nn>>>crysx_nn.__version__'0.1.0'>>>Finally download the example script (here) for simulating logic gates like AND, XOR, NAND, and OR,
and try running itpythonSimluating_logic_gates.py(back to top)UsageThe most important thing for using this library properly is to use 2D NumPy/Cupy arrays for defining the inputs and expected outputs (targets) for a network. 1D arrays for inputs and targets are not supported and will result in an error.For example, let us try to simulate the logic gate AND. The AND gate takes two input bits and returns a single input bit.
The bits can take a value of either 0 or 1. The AND gate returns 1 only if both the inputs are 1, otherwise it returns 0.The truth table of the AND gate is as followsx1x2output000010100111The four possible set of inputs areimportnumpyasnpinputs=np.array([[0.,0.,1.,1.],[0.,1.,0.,1.]]).T.astype('float32')print(inputs)print(inputs.dtype)Output:[[0.0.][0.1.][1.0.][1.1.]]float32Similarly, set the corresponding four possible outputs as a 2D numpy array# AND outputsoutputAND=np.array([0.,0.,0.,1.])# 1D arrayoutputAND=np.asarray([outputAND]).T# 2D arrayprint('AND outputs\n',outputAND)Output:ANDoutputs[[0.][0.][0.][1.]]Next, we need to set some parameters of our Neural networknInputs=2# No. of nodes in the input layerneurons_per_layer=[3,1]# Neurons per layer (excluding the input layer)activation_func_names=['Sigmoid','Sigmoid']nLayers=len(neurons_per_layer)eta=0.5# Learning ratenEpochs=10**4# For stochastic gradient descentbatchSize=4# No. of input samples to process at a time for optimizationLet us now create the neural network model using the above parametersfromcrysx_nnimportnetworkmodel=network.nn_model(nInputs=nInputs,neurons_per_layer=neurons_per_layer,activation_func_names=activation_func_names,batch_size=batchSize,device='CPU',init_method='Xavier')Note:device='CPU'indicates that the we are going to be running the computations on the CPU. If you have cupy installed and a compatible GPU, then you could also usedevice='GPU'. However, in that case the inputs and ouputs should also be cupy arrays instead of numpy arrays.
To check if the model was initialized correctly, you can check the model details asmodel.details()Output:----------------------------------------------------------------------------------
****Neural Network Model Details****
----------------------------------------------------------------------------------
Number of input nodes:  2
Number of layers (hidden+output):  2
Number of nodes in each layer (hidden & output):  [3, 1]
Activation function for each layer (hidden & output):   ['Sigmoid', 'Sigmoid']
Method used for weights and biases initialization:   Xavier
Batch Size:  4
Device:  CPU
Optimization method:  SGD
Learning rate:  0.5
----------------------------------------------------------------------------------For a better understanding of our network, let us visualize it.model.visualize()Output:Weights and biases are already initialized when you create the model using the'Xavier'method.
Weights and biases are lists of 2D and 1D NumPy arrays, respectively (1 Numpy array for each layer). In our case, we have 2 layers (1 hidden+ 1 output), therefore, the list of Weights and Biases will have 2 NumPy arrays each.
You can also initialize them again using some other method as follows:model.init_params(method='NormXavier')print('Initial Weights:\n',model.init_weights)print('Initial Biases:\n',model.init_biases)Output:Initial Weights:
 [array([[-0.66735507,  1.89981375],
       [ 1.58135381,  0.30297808],
       [-1.02875636,  0.39637066]]), array([[ 1.14150595,  0.86579216, -1.50878406]])]
Initial Biases:
 [array([0., 0., 0.]), array([0.])]Finally it is time to train our neural network. We will use mean squared error (MSE) loss function as the metric of performance. Currently, only stochastic gradient descent is supported.# Run optimizationmodel.optimize(inputs,outputAND,lr=0.5,nEpochs=nEpochs,loss_func_name='MSE',miniterEpoch=1,batchProgressBar=False,miniterBatch=100)The optimization saves the optimized weightsmodel.weightsand biasesmodel.biasesas well as the errormodel.errorsat each epoch in the attributes of thenn_modelobject.We can then plot the training loss at each epochimportmatplotlib.pyplotasplt# Plot the error vs epochsplt.plot(model.errors)plt.yscale('log')plt.show()Output:Finally, you can make predictions using the optimized network using the following codepredictions,error=model.predict(inputs,outputAND,loss_func_name='MSE')print('Input:\n',inputs)print('Prediction:\n',predictions)print('Expected Output:\n',outputAND)print('New Average Error with optimized weights:\n',error)OUTPUT:Input:
 [[0. 0.]
 [0. 1.]
 [1. 0.]
 [1. 1.]]
Prediction:
 [[6.51744058e-05]
 [8.16272136e-03]
 [5.94678339e-03]
 [9.90297134e-01]]
Expected Output:
 [[0.]
 [0.]
 [0.]
 [1.]]
New Average Error with optimized weights:
 4.903602650103742e-05For more examples, please refer to theExamples SectionCrysX-NN (crysx_nn) also provides CUDA support by using cupy versions of all the features ike activation functions, loss functions, neural network calculations, etc.
Note: For small networks the Cupy versions may actually be slower than CPU versions. But the benefit becomes evident as you go beyond 1.5 Million parameters.(back to top)FeaturesEfficient implementations of activation functions and their gradientsSigmoid, Sigmoid_gradReLU, ReLU_gradSoftmax, Softmax_gradSoftplus, Sofplus_gradTanh, Tanh_gradTanh_offset, Tanh_offset_gradIdentity, Identity_gradEfficient implementations of loss functions and their gradientsMean squared errorMean absolute errorBinary cross entropyCategorical cross entropySeveral methods for weights initialization'random1','random2','random3''Xavier''NormXavier''He'Neural network optimization usingStochastic Gradient DescentSupport for batched inputs, i.e., supplying a matrix of inputs where the columns correspond to features and rows to the samplesSupport for GPU through Cupypip install cupy-cuda102(Tested with CUDA 10.2)JIT compiled functions when possible for efficiencyObject oriented(back to top)RoadmapMore activation functionsLeakyReLU, Swish, Mish, etc.More loss functionsHinge loss, and othersOptimization algorithms apart from Stochastic Gradient Descent, like ADAM, RMSprop, etc.Implement regularizersBatch normalization and layer normalizationDropoutEarly stoppingSome metric functions, although there is no harm in usingsklearnfor thatSee theopen issuesfor a full list of proposed features (and known issues).(back to top)ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)ContactManas Sharma -@manassharma07-feedback@bragitoff.comProject Link:https://github.com/manassharma07/crysx_nnProject Documentation:https://bragitoff.comBlog:https://bragitoff.com(back to top)AcknowledgmentsLecture notes byProf. Dr. Bernd BrÃ¼gmannOndÅ™ej  Zelenkafor useful discussions.Neural Networks and Deep LearningbyMichael NielsenA high-bias, low-variance introduction to Machine Learning for physicistsby Mehta et al., 2019Deep Learningby Goodfellow et al., 2016(back to top)CitationIf you use this library and would like to cite it, you can use:M. Sharma, "CrysX-NN: Neural Network libray", 2021. [Online]. Available: https://github.com/manassharma07/crysx_nn. [Accessed: DD- Month- 20YY].or:@Misc{,
  author = {Manas Sharma},
  title  = {CrysX-NN: Neural Network libray},
  month  = december,
  year   = {2021},
  note   = {Online; accessed <today>},
  url    = {https://github.com/manassharma07/crysx_nn},
}(back to top)