Package: neuralforge
Autograd Framework From ScratchNeuralForge is a unit-tested and documented educational framework. Similar to PyTorch, but with moreclear code.The autograd from scratch engine is intensor_operations.py. I got a lot of inspiration from Andrej Karpathy's micrograd videos.The deep learning model layers are innn/layers.py.Check out theimplemented basic operations:AdditionSubtractionMultiplicationDivisionMatrix multiplicationExponentiationLogSquare RootTheimplemented statistics:SumMeanMaxVarianceAnd theimplemented tensor operations:ReshapeTransposeConcatenateStackMaskedFillSlice1. Project Structureneuralforge/: Framework with python files.neuralforge/tensor_operations.py:  File with theTensorclass and all of the tensorOperations.neuralforge/utils.py: File with operations and helper functions.neuralforge/nn/: Submodule of the framework. Contains full layers and optimizers.neuralforge/nn/nn.py: Most deep learning layers, andnn.Moduleclass.neuralforge/nn/optim.py: File with optimizers.data/: Folder to store training data. Currently holdsshakespeare.txt.test/: Folder with unit tests. Containstest_framework.py.setup.py: Setup file for the framework.2. Running it YourselfSimple Autograd Example:importneuralforgeasforge# Instantiate Tensors:x=forge.randn((8,4,5))w=forge.randn((8,5,4),requires_grad=True)b=forge.randint((4),requires_grad=True)# Make calculations:out=x@wout+=b# Compute gradients on whole graph:out.backward()# Get gradients from specific Tensors:print(w.grad)print(b.grad)Complex Autograd Example (Transformer):importneuralforgeasforgeimportneuralforge.nnasnn# Implement Transformer class inheriting from forge.nn.Module:classTransformer(nn.Module):def__init__(self,vocab_size:int,hidden_size:int,n_timesteps:int,n_heads:int,p:float):super().__init__()# Instantiate Transformer's Layers:self.embed=nn.Embedding(vocab_size,hidden_size)self.pos_embed=nn.PositionalEmbedding(n_timesteps,hidden_size)self.b1=nn.Block(hidden_size,hidden_size,n_heads,n_timesteps,dropout_prob=p)self.b2=nn.Block(hidden_size,hidden_size,n_heads,n_timesteps,dropout_prob=p)self.ln=nn.LayerNorm(hidden_size)self.linear=nn.Linear(hidden_size,vocab_size)defforward(self,x):z=self.embed(x)+self.pos_embed(x)z=self.b1(z)z=self.b2(z)z=self.ln(z)z=self.linear(z)returnz# Get tiny Shakespeare test data:text=load_text_data(f'{PATH}/data/shakespeare.txt')# Create Transformer instance:model=Transformer(vocab_size,hidden_size,n_timesteps,n_heads,dropout_p)# Define loss function and optimizer:loss_func=nn.CrossEntropyLoss()optimizer=optim.Adam(model.parameters(),lr=0.01,reg=0)# Training Loop:for_inrange(n_iters):x,y=get_batch(test_data,n_timesteps,batch_size)z=model.forward(x)# Get loss:loss=loss_func(z,y)# Backpropagate the loss using forge.tensor's backward() method:loss.backward()# Update the weights:optimizer.step()# Reset the gradients to zero after each training step:optimizer.zero_grad()Note:You can install the framework locally with:pip install neuralforgeRequirementsThe required packages are listed inrequirements.txt.The requirements can be installed on a virtual environment with the command:pip install -r requirements.txtNote:The framework is built around numpy, so there is no CUDA availability.Build a Custom ModelTo create a custom model class, you can use the exact same syntax as you would in PyTorch, inheriting from nn.Module.You may chose amongthe following layers:nn.Embedding(first layer, turns input indexes into vectors)nn.PositionalEmbedding(second layer, adds position information to every timestep of the input)nn.Linear(simple fully-connected layer)nn.MultiHeadSelfAttention(core of the transformer, calculates weighted sum of inputs)nn.Block(full transformer block - Contains MHSA, Linear and LayerNorm layers)nn.CrossEntropyLoss(last layer, returns probabilities for next generated character)Andthe following functions:nn.Dropout(can be added to apply dropout)nn.LayerNorm(normalizes the tensors)nn.Softmax(scales the values between 0 and 1)nn.Tanh(scales the values between -1 and 1)nn.Relu(zeroes all negative values)3. ResultsThe models implemented intest_framework.pyall converged tonear-zero losses.This framework is not as fast or as optimized as PyTorch, but I tried making it more interpretable.Hope you enjoy!