Package: aeronet-learn
AeroNetAeroNet is a simple neural network library. It is written entirely using python, numpy, and scipy. This is an educational neural network library made with the hope of demystifying some of these modern deep learning libraries. Many modern deep learning libraries do not fundamentally differ from AeroNet all that much (minus automatic differentiation). Most of the code in modern libraries is just dedicated to efficiency and optimization of the algorithms used. Feel free to play around with AeroNet and see what deep learning projects you can make.Installing AeroNetAeroNet is now hosted on PyPi for your convenience. The only dependencies are numpy, scipy, and dill which are all installed automatically when AeroNet is installed. Python 3.10 is recommended for using AeroNet.pipinstallaeronet-learnRunning From The Source CodeCreating A Virtual EnvironmentYou may want to run AeroNet directly from the source code. Or maybe you want to directly run the training examples hosted on github. This can be done by cloning the code fromgithub. You can then create a virtual enviroment with the following code.python -m venv venvNeuralNetworkLibrary
venvNeuralNetworkLibrary/scripts/activate.bat
pip install -r requirements.txtYou will then be able to run training examples directly. As seen in the next section.Testing Out A NetworkTest your network against one of these datasets...# move into the training examples directorycdtraining_examples# then execute one of the scripts below...# Simple xor example with dense layerspythonxor.py# Mnist with dense layerspythonmnist.py# Convolutional neural network implementation for mnistpythonmnist_conv.py# Fully convolutional network implementation for mnistpythonmnist_fcn.py# Convolutional neural network implementation for mnist with max poolingpythonmnist_maxpooling.pyAeroNet Example Usage# Import all neural network classes.fromnnimport*# Network layers are initialized as a list of objectsnetwork_layers=[Dense(28*28,70),Sigmoid(),Dense(70,35),Sigmoid(),Dense(35,10),Softmax()]# Create a network objectnetwork=Network(network_layers,TrainingSet(input_train,output_train,input_test,output_test,post_processing),loss_function,loss_function_prime,epochs=10,batch_size=1)# Train the networknetwork.train()# Generate a prediction from the network given a single input arrayprediction_array=network.predict(input_array)# Save the network to a filesave_network(network,"network_filename.pkl")# Load the network from a filenetwork=load_network("network_filename.pkl")TrainingSet ExplainedThe TrainingSet object is simply an object that stores your training and validation set data for your network. As well as a post processing function that is applied to your network predictions and training set data before comparisons are made.TrainingSet(input_train,output_train,input_test,output_test,post_processing)input_train = training set inputsoutput_train = training set outputsinput_test = test / valdation set inputsoutput_test = test / valdation set outputspost_processing is a function applied to your network prediction and training set data before they are compared. For example with the xor problem your network makes predictions based on floats. So you may want to round to the nearest float before comparing the prediction to the desired output. So you could pass np.rint as the post processing function to round to the nearest int. This allows you to get accurate test accuracy output during training. The same applies to things like mnist. You may have a softmax output with percentage predictions based on 10 indices (0 through 9). You may want your prediction to be the indice with the highest percentage. Therefore, you could pass the np.argmax function as the post processing function. Hopefully you can see why this is useful.Supported Neural Layers# LayersDense(num_input_neurons,num_output_neurons)# For weight manipulationConvolutional((input_depth,input_height,input_width),kernel_size,num_kernels,stride=(int,int),padding=((int,int),(int,int)))Reshape()# Modifies the shape of the numpy arrays passed between layersFlatten()# Flattens a numpy array into a 2D matrix with a single columnDropout(probability)# Randomly drops layer outputs based on a probability to prevent overfitting. A probability of 0.25 would drop 25% of connections.MaxPooling2D((input_depth,input_height,input_width),kernel_size,stride=(int,int),padding=((int,int),(int,int)))# Activation FunctionsSigmoid()Tanh()Relu()LeakyRelu()# Leaky Relu not validatedSoftmax()# For output percentage predictionsSupported OptimizersSGD()# Stochastic Gradient DescentMomentumSGD()# Stochastic Gradient Descent with MomentumSupported InitializersUniform(low=-1,high=1)Normal(mean=0,std=1)Zero()# Zero initialized array for biasesXavier()The Network Classnetwork = Network(
    layers,
    TrainingSet(x_train, y_train, x_test, y_test, post_processing),
    loss=loss_function,
    loss_prime=loss_function_prime,
    epochs=10,
    batch_size=1,
    layer_properties=LayerProperties(learning_rate, weight_initializer, bias_initializer, optimizer),
    data_augmentation=[function1, function2, function3],
    percent_error_threshold=None
    verbose=true
)The data_augmentation parameter is a list of pure functions. These functions must take in a single numpy array and return a numpy array. To use functions that take multiple parameters you can pass in a lambda function that calls the multi-parameter function. It can be used for numpy array rotation, scaling, noise, and translation for datasets like mnist (that are uniform and centered). These functions are applied sequentially to every training sample, for every epoch, before they are passed into the network. If no data_augmentation list is supplied then no data_augmentation is performed. You can always choose to augment the dataset before network training begins.The percent_error_threshold parameters on the network class can be used to stop training when the network reaches a certain error percentage. For example, if you passedpercent_error_threshold = 0.05then the network would exit the training loop if the error percentage fell below 5%. This can be used to prevent overtraining or shorten training time if you are only aiming to get below a certain error rate.Network MetadataThe network class does track some metadata to help with debugging or generate training statistics.total_training_time (float).The total training time for the network in minutes.per_epoch_errors (list of floats)A list of the errors per epoch. This is only for the last time the train() method was called.Layer PropertiesLearning rates can be set at both a network level (every layer) or at individual layers themselves. This is done through the use of a layer properties class. Each layer with trainable parameters has a default learning rate, weight / bias initializer, and optimizer. So even if you input no layer properties for the layer (or network) it will be populated with some defaults.# This example would set these specific layer properties for the first and second dense layerlayer1_properties=LayerProperties(learning_rate=0.05,weight_initializer=Uniform(),bias_initializer=Uniform(),optimizer=SGD())layer2_properties=LayerProperties(learning_rate=0.03,weight_initializer=Uniform(),bias_initializer=Zero(),optimizer=SGD())network_layers=[Dense(28*28,70,layer_properties=layer1_properties),Sigmoid(),Dense(70,35,layer_properties=layer2_properties),Sigmoid(),Dense(35,10),Softmax()]# Optionally you can set the layer properties for every layer in the network.# This is done by setting layer properties on the network class itself.all_layer_properties=LayerProperties(learning_rate=0.05,weight_initializer=Uniform(),bias_initializer=Uniform(),optimizer=SGD())network=Network(network_layers,TrainingSet(input_train,output_train,input_test,output_test),loss_function,loss_function_prime,epochs=10,batch_size=1,layer_properties=all_layer_properties)# You can also choose to overwrite only some properties at both the network and layer level.# For example the following would only change the learning rate (for all layers) but leave all other defaults the same.all_layer_properties=LayerProperties(learning_rate=0.05)network=Network(network_layers,TrainingSet(input_train,output_train,input_test,output_test),loss_function,loss_function_prime,epochs=10,batch_size=1,layer_properties=all_layer_properties)Kernel, Stride, and Padding Notation:warning: Striding and Padding were only recently implemented. I cannot confirm that they work in all cases yet. So use at your own risk.Sometimes you might see notation likearray[x][y]. However, this is kind of a confusing syntax that people (including myself) write. Do not confuse the arbitrary variablexwith the width. It does not represent the "x" of a coordinate system; in reality it represents the height. If you view the array syntax from a coordinate perspective it is reallyarray[height][width]. So when you are inputting parameters for strides and padding it is really with the syntax(height, width).KernelThe kernel supports 3 different syntax listed below.kernel_size = 3         # Uses 3 for the height and width
kernel_size = (3)       # Uses 3 for the height and width
kernel_size = (3,3)StridingStriding only has two dimensions, that is striding in the "height" dimension and striding in the "width" dimension. Striding follows the syntax(height_stride, width_stride). So a stride of (2, 3) would move the kernel 2 positions down and 3 positions to the right. Syntax example listed below.stride = (3, 3)PaddingThere are four different allowed padding formats.(pad_all_4_sides)padding = 1or...padding = (1)(pad_height, pad_width)padding = (1, 1)((pad_top_height,pad_bottom_height), (pad_left_width,pad_right_width))Alternatively you could think of the format as...((top, bottom), (left, right))padding = ((1,1), (2,2))CUDA SupportThis library no longer supports CUDA. It has been removed because it was too hard to maintain with more complex layer implementations.TODOPackage And Host The LibraryImplement Avg PoolingImplement Adaptive Avg PoolingImplement Batch NormalizationImplement Layer Input Size Calculations